A method for processing consistency of big data
By introducing a consistency coordination server in the big data system, checking and correcting data inconsistency between storage servers, and using trust to determine the most reliable hash value, the problem of data inconsistency in the big data system is solved and the system stability is improved sex.
Patent Information
- Application Number
- CN202410375769.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-03-29
- Publication Date
- 2024-11-05
- Estimated Expiration
- 2044-03-29
AI Technical Summary
In big data systems, due to hardware or software errors, data is inconsistent among multiple storage servers. How to ensure the consistency of data is a core issue.
A big data consistency processing method is adopted to receive data operation requests through the central server, send them to the consistency coordination server for consistency checks, determine the storage server set, calculate the hash value of the data record, and determine the most reliable hash value based on the trust level. , correct inconsistent data records to ensure data consistency.
Timely discover and accurately process inconsistent data in big data systems to ensure data consistency, thereby improving system stability.
Smart Images

Figure CN118296019B_ABST
Abstract
Description
[Technical field]
[0001] The present invention belongs to the field of big data processing, and in particular relates to a big data consistency processing method. [Background technology]
[0002] In the prior art, massive data is usually stored in multiple storage servers. Specifically, each storage server has a database for storing data, and each copy of data is stored in multiple databases at the same time. In other words, each data record has multiple copies at the same time, so it is necessary to ensure the consistency of the data record in multiple copies. This is one of the core requirements of big data management and also the requirement to meet the most basic ACID characteristics of the database.
[0003] In the field of big data, data security is no longer guaranteed by hardware, but by software means, by writing data to multiple databases at the same time. However, after writing data records to multiple databases at the same time, data errors may occur due to hardware or software errors, resulting in inconsistent data. In this case, how to ensure data consistency is a problem that needs to be solved in this field. [Summary of the invention]
[0004] In order to solve the above problems in the prior art, the present invention provides a big data consistency processing method.
[0005] The technical solution adopted by the present invention is specifically as follows:
[0006] A method for processing consistency of big data includes the following steps:
[0007] Step 100: The central server receives a data operation request from an external application and determines one or more storage servers to execute the data operation request;
[0008] Step 200: The central server sends the relevant information of the data operation request to the consistency coordination server;
[0009] Step 300: the consistency coordination server starts a consistency check on some data records after a predetermined condition is met;
[0010] Step 400: If the consistency coordination server determines to perform a consistency check on the data record R, the consistency coordination server determines all storage servers storing the data record R as the storage server set C corresponding to the data record R. R ; The storage server set C R There are n storage servers S1, S2, ... S n ;
[0011] Step 500: The consistency coordination server sends a message to the storage server set C R Each storage server in sends a consistency check message for the data record R;
[0012] Step 600: The storage server that receives the consistency check message locks the data record R, calculates the hash value of the data record R, and returns the calculation result to the consistency coordination server;
[0013] Step 700: The consistency coordination server determines whether all n hash values corresponding to the data record R are consistent. If they are consistent, the consistency check for the data record R ends; if they are inconsistent, the most reliable hash value H of the data record R is determined from the n hash values. R ;
[0014] Step 800: The consistency coordination server calculates the H R The data record R in the storage server of the storage server is corrected, and the inconsistent data record R in other storage servers is corrected; after the correction is completed, the consistency coordination server notifies all storage servers to unlock the data record R;
[0015] Wherein, the step 700 specifically includes:
[0016] Step 710: Determine whether there are k equal hash values among the n hash values, and satisfy k>n / 2; if so, these k equal hash values are the most reliable hash value H R ; If not, proceed to the next steps;
[0017] Step 720: Determine each storage server S i The original trust Z i , 1≤i≤n;
[0018] Step 730: For storage server S i , determine the storage server S involved i The number of times L that the hash value it computed was not selected as the most reliable hash value in the most recent m consistency checks i ; Where m is a predetermined number of times;
[0019] Step 740: Based on the original trust level Z of the storage server i and the number L i , computing storage server S i The trust level W i ;
[0020] Step 750: For any hash value H among the n hash values i , determine the i The corresponding storage server set SH i ; The storage server set SH i is the storage server set C R A subset of SH i The hash value calculated by any storage server in step 600 is equal to H i ;
[0021] Step 760: For any hash value H i , calculate its corresponding SH i The sum of the trust of all storage servers in H i The trust level of the hash value with the highest trust level is taken as the most reliable hash value H R .
[0022] Furthermore, the relevant information of the data operation request includes a data record identifier and a data operation type involved in the data operation request.
[0023] Furthermore, the predetermined condition includes a predetermined time interval, that is, a consistency check is started every predetermined time interval.
[0024] Furthermore, the predetermined condition includes: the data operation history of the partial data record meets the corresponding condition.
[0025] Furthermore, the data record R is prohibited from accepting any write operation from the central server during the locking period, but can be read.
[0026] Further, the system administrator sets an original trust level for the storage server based on the hardware reliability and / or software reliability of the storage server.
[0027] Furthermore, in step 740, W i =Z i -aL i ; where a is a predefined constant.
[0028] Furthermore, in step 740, W i =Z i -A i L i Among them, A i It is based on the storage server S i The value is determined by the specific circumstances.
[0029] Furthermore, A i =Z i / m.
[0030] Further, the step 800 specifically includes: based on H R The corresponding storage server set SH R , the consistency coordination server randomly selects SH R A storage server is selected, a consistency read command is sent to it, data record R is read, and then a consistency correction command is sent to the storage server that needs to be corrected. The consistency correction command includes data record R and H. R The storage server that receives the consistency correction command corrects the locally stored data record R according to the data record R in the command, and R Verify the correctness of the correction results.
[0031] The beneficial effects of the present invention are: timely discovering and accurately processing inconsistent data in a big data system, ensuring data consistency in the big data system, and thus improving the stability of the entire system.
Brief Description of the Drawings
[0032] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of the present application, but do not constitute an improper limitation of the present invention. In the drawings:
[0033] Figure 1 It is a basic architecture diagram of the big data processing system involved in the present invention. [Specific implementation method]
[0034] The present invention will be described in detail below in conjunction with the accompanying drawings and specific embodiments, wherein the illustrative embodiments and descriptions are only used to explain the present invention but are not intended to limit the present invention.
[0035] The big data consistency processing method of the present invention is applied to a system for storing and processing big data. Figure 1 , which shows the basic architecture of the system, which includes multiple storage servers 1, a central server 2 and a consistency coordination server 3, and each server is interconnected and communicates with each other through a network.
[0036] The storage server is used to store data. As a big data system, it usually stores massive amounts of data. In order to ensure the efficiency and security of data storage, the system includes multiple storage servers that can work in parallel. In practical applications, each storage server can store a portion of the data records of the entire system, and a data record can be stored in multiple (or even all) storage servers at the same time, thereby naturally realizing the disaster recovery function of the data record. Even if one of the storage servers is damaged, the data record can also be read from other storage servers. In specific implementation, each storage server can be set up with a database to store the corresponding data. The databases of different storage servers are usually databases of the same type, but of course they can also be databases of different types. The present invention does not limit this.
[0037] The central server is the interface between the system and external applications, and is used to receive data operation requests from external applications, such as adding data records, deleting data records, querying data records, etc. Generally speaking, when an external application needs to operate data, it sends the data operation request to the central server through the network, and the central server initiates the corresponding data operation to the storage server according to the specific content of the data operation request. For example, if the data operation request is to add a data record, the central server needs to determine which storage servers the data record should be stored in, and then send the request to add the data record to the determined storage server, thereby realizing the distributed storage of data records on multiple storage servers. The method for processing data operation requests by the central server is an existing prior art in the field and will not be repeated here.
[0038] The consistency coordination server is the core server used to execute the big data consistency processing method in the present invention. As mentioned above, a data record can be stored on multiple storage servers at the same time, that is, there are multiple copies of the data record at the same time. Due to hardware or software errors, these copies may be inconsistent, which may lead to a series of subsequent data errors. Therefore, the present invention executes the big data consistency processing method through the consistency coordination server to discover inconsistent data records and coordinate the consistency of the data.
[0039] Based on the above system architecture, the big data consistency processing method of the present invention is described in detail below.
[0040] Step 100: The central server receives a data operation request from an external application and determines one or more storage servers to execute the data operation request.
[0041] As mentioned above, the data operation request involves operations such as adding, deleting, modifying, and querying data records in the system. The central server determines one or more storage servers for executing the data operation request based on the data operation request, and then sends the data operation request to the determined storage server. The storage server executes the data operation request and returns the execution result to the central server, which then returns the execution result to the external application.
[0042] For example, if the data operation request involves modifying the data record, the central server needs to determine all storage servers corresponding to the data record and let these storage servers synchronously perform the corresponding modification operation. For another example, if the data operation request is to read a specified data record, the central server only needs to determine a storage server corresponding to the data record and read the corresponding data record through the storage server. The specific implementation method of the above process is the prior art of the big data processing system in this field, which is well known to those skilled in the art and is not the focus of the present invention, so it will not be repeated here.
[0043] Step 200: The central server sends relevant information of the data operation request to the consistency coordination server.
[0044] Specifically, the central server determines the relevant information required by the consistency coordination server by analyzing the data operation request, and the relevant information includes the data record identifier and data operation type (such as adding, deleting, modifying, querying, etc.) involved in the data operation request, but does not include the specific content of the data record. The consistency coordination server stores the received relevant information for subsequent consistency coordination tasks.
[0045] Step 300: The consistency coordination server starts a consistency check on some data records after a predetermined condition is met.
[0046] The predetermined condition may be pre-set by the system administrator according to the management policy. Specifically, the predetermined condition may include a predetermined time interval, that is, a consistency check is started every predetermined time interval. The predetermined condition may also include: the data operation history of the partial data record meets the corresponding conditions, for example, the number of modification operations performed on the data record after the last consistency check is greater than a predetermined threshold. In other words, if the number of modification operations performed on a data record after the consistency check is greater than a predetermined threshold, the data record needs to be checked for consistency again, so the consistency coordination server starts a consistency check on the data record this time. For another example, the number of various operations on the data record after the last consistency check can be comprehensively considered, the weighted sum of the various operation numbers can be calculated, and whether the weighted sum exceeds the predetermined threshold is used as the predetermined condition.
[0047] Step 400: If the consistency coordination server determines to perform a consistency check on the data record R, the consistency coordination server determines all storage servers storing the data record R as the storage server set C corresponding to the data record R. R .
[0048] Specifically, when the consistency coordination server starts to perform consistency check on a data record R, the consistency coordination server first determines all storage servers storing the data record R as the corresponding storage server set C by querying. R The consistency coordination server can determine the storage server set C through the relevant information collected in step 200. R , or directly determine the storage server set C by querying the central server R Alternatively, if the big data system used has a corresponding global index server or global catalog server, the consistency coordination server can also determine the storage server set C by querying these servers. R In summary, those skilled in the art can determine how to determine the storage server set C according to the specific big data system. R , the present invention is not limited to this.
[0049] Step 500: The consistency coordination server sends a message to the storage server set C R Each storage server in sends a consistency check message for the data record R.
[0050] Specifically, assuming that the storage server set C R There are n storage servers S1, S2, ... S n , that is, C R ={S1, S2, ..., S n}, the consistency coordination server sends a i (1≤i≤n) sends a consistency check message, the consistency check message is used to instruct the storage server S i , start the consistency check of the data record R. Of course, if the consistency coordination server needs to check the storage server S i The consistency check of multiple data records on the storage server S i The consistency check message may also include an indication to start a consistency check on a plurality of data records.
[0051] Step 600: The storage server that receives the consistency check message locks the data record R, calculates the hash value of the data record R, and returns the calculation result to the consistency coordination server.
[0052] Specifically, for the storage server set C R Any storage server S i After receiving the consistency check message for the data record R, it locks the data record R. Here, "locking" means that the data record R is prohibited from accepting any write operation (including modification and deletion) from the central server during the locking period, but the data record R can be read.
[0053] After locking the data record R, the storage server S i Calculate the hash value H of the data record R i , that is, H i = hash(R). The specific hash function may be any hash function in the art, such as MD5, SHA-1, etc., and the present invention does not limit this.
[0054] Each storage server S i (1≤i≤n) the calculated H i Return to the consistency coordination server, so that the consistency coordination server can obtain n hash values H1, H2, ..., H of the data record R n .
[0055] Step 700: The consistency coordination server determines whether all hash values corresponding to the data record R are consistent. If they are consistent, the consistency check for the data record R ends; if they are inconsistent, the most reliable hash value H of the data record R is determined from all hash values. R .
[0056] Specifically, in step 600, the consistency coordination server receives n hash values H1, H2, ..., H of the data record R from n storage servers. n If the n hash values are all equal, it means that the data records R stored in the n storage servers are the same, that is, the data records R in the system are consistent, and the consistency check of the data record R is completed.
[0057] If these n hash values are not completely equal, it means that the data records R stored in these n storage servers are inconsistent. At this time, the consistency coordination server needs to determine the most reliable hash value from these n hash values. Therefore, the present invention proposes a method for determining the most reliable hash value based on the trustworthiness of the storage server, which specifically includes the following steps.
[0058] Step 710: Determine whether there are k equal hash values among the n hash values, and satisfy k>n / 2; if so, these k equal hash values are the most reliable hash value H R ; If not, proceed to the next steps;
[0059] Specifically, if more than half of the n hash values are equal, then this hash value can be considered the most reliable hash value.
[0060] Step 720: Determine each storage server S i The original trust Z i , 1≤i≤n.
[0061] Specifically, each storage server has an original trust, which is preset by the system administrator. The system administrator can set an original trust for the storage server based on experience, based on the hardware reliability and / or software reliability of the storage server. That is, if the hardware and / or software reliability of a storage server itself is higher, its original trust should be higher, and the specific value is set by the system administrator based on experience.
[0062] Step 730: For storage server S i , determine the storage server S involved i The number of times L that the hash value it computed was not selected as the most reliable hash value in the most recent m consistency checks i ; where m is a predetermined value.
[0063] Specifically, for each storage server S i , it has participated in multiple consistency checks in the past, and the data records checked can be the current data record R or other data records. In short, the most recent m consistency checks it participated in are selected, and each consistency check will calculate a hash value based on step 600. This hash value may be selected as the most reliable hash value (in step 700), or it may not be the most reliable hash value. Therefore, the number of times L that the hash values calculated in these m times are not selected as the most reliable hash value can be counted. i .
[0064] Step 740: Based on the original trust level Z of the storage server i and the number L i , computing storage server S i The trust level W i .
[0065] According to a preferred embodiment of the present invention, W i =Z i -aL i ; Where a is a predefined constant, that is, each time an unreliable hash value is calculated, the trust of the storage server will decrease by a value, and the trust is not less than zero.
[0066] According to another preferred embodiment of the present invention, W i =Z i -A i L i Among them, A i It is based on the storage server S i The value determined by the specific situation, preferably, A i =Z i / m.
[0067] After determining n storage servers S i After the trust levels are determined, the most reliable hash value can be determined based on these n trust levels.
[0068] Step 750: For any hash value H among the n hash values i , determine the i The corresponding storage server set SH i ; The storage server set SH i is the storage server set C R A subset of SH i The hash value calculated by any storage server in step 600 is equal to H i .
[0069] Specifically, although there are n hash values, some of them are equal. Based on the result of step 710, the n hash values include at least two different specific values. Step 750 is actually to obtain all storage server sets that calculate any one of the specific values. Therefore, if the n hash values include k different specific values, then k different storage server sets SH are actually obtained. i , that is, if H i =H j , then SH i =SH j Otherwise, there must be SH iNot equal to SH j .
[0070] Step 760: For any hash value H i , calculate its corresponding SH i The sum of the trust of all storage servers in H i The trust level of the hash value with the highest trust level is taken as the most reliable hash value H R .
[0071] At this point, the consistency coordination server has obtained the most reliable hash value H from n incompletely equal hash values. R , as the hash value of the data record R. At the same time, the consistency coordination server can also calculate the H R All storage servers of the storage server store data records R which will serve as the basis for consistency coordination and replace inconsistent data records R in other storage servers.
[0072] Step 800: The consistency coordination server calculates the H R The consistency coordination server detects the data record R in the storage server of the storage server, and corrects the inconsistent data record R in other storage servers. After the correction is completed, the consistency coordination server notifies all storage servers to unlock the data record R.
[0073] Specifically, based on H R The corresponding storage server set SH R , S.H. R The data records R stored in the storage server in SH are considered reliable, while R The data records R stored outside the SH are inconsistent and unreliable, so R Any storage server reads the data record R, and then corrects other inconsistent data records R based on the data record R.
[0074] In the specific implementation, the consistency coordination server can randomly select SH R A storage server is selected, a consistency read command is sent to it, data record R is read, and then a consistency correction command is sent to the storage server that needs to be corrected. The consistency correction command includes data record R and H. R The storage server that receives the consistency correction command can correct the locally stored data record R according to the data record R in the command, and R Verify the correctness of the correction results.
[0075] Through the above steps, the consistency check and processing of the big data system are completed. The present invention sets a consistency coordination server in the system to execute the above steps, which can timely discover and accurately process inconsistent data in the big data system, ensure data consistency in the big data system, and thus improve the stability of the entire system.
[0076] The above description is only a preferred embodiment of the present invention, so all equivalent changes or modifications made according to the structure, characteristics and principles described in the scope of the patent application of the present invention are included in the scope of the patent application of the present invention.
Claims
1. A method for processing consistency of big data, characterized in that: The following steps are involved: Step 100: The central server receives a data operation request from an external application and determines one or more storage servers to execute the data operation request; Step 200: The central server sends the relevant information of the data operation request to the consistency coordination server; Step 300: the consistency coordination server starts a consistency check on some data records after a predetermined condition is met; Step 400: If the consistency coordination server determines to perform a consistency check on the data record R, the consistency coordination server determines all storage servers storing the data record R as the storage server set C corresponding to the data record R. R ; The storage server set C R There are n storage servers S1, S2, ... S n ; Step 500: The consistency coordination server sends a message to the storage server set C R Each storage server in sends a consistency check message for the data record R; Step 600: The storage server that receives the consistency check message locks the data record R, calculates the hash value of the data record R, and returns the calculation result to the consistency coordination server; Step 700: The consistency coordination server determines whether all n hash values corresponding to the data record R are consistent. If they are consistent, the consistency check for the data record R ends; if they are inconsistent, the most reliable hash value H of the data record R is determined from the n hash values. R ; Step 800: The consistency coordination server calculates the H R The data record R in the storage server is corrected, and the inconsistent data record R in other storage servers is corrected; After the correction is completed, the consistency coordination server notifies all storage servers to unlock the data record R; Wherein, the step 700 specifically includes: Step 710: Determine whether there are k equal hash values among the n hash values, and satisfy k>n / 2; if so, these k equal hash values are the most reliable hash value H R ; If not, proceed to the next steps; Step 720: Determine each storage server S i The original trust Z i , 1≤i≤n; Step 730: For storage server S i , determine the storage server S involved i The number of times L that the hash value it computed was not selected as the most reliable hash value in the most recent m consistency checks i ; Where m is a predetermined number of times; Step 740: Based on the original trust level Z of the storage server i and the number L i , computing storage server S i The trust level W i ; Step 750: For any hash value H among the n hash values i , determine the i The corresponding storage server set SH i ; The storage server set SH i is the storage server set C R A subset of SH i The hash value calculated by any storage server in step 600 is equal to H i ; Step 760: For any hash value H i , calculate its corresponding SH i The sum of the trust of all storage servers in H i The trust level of the hash value with the highest trust level is taken as the most reliable hash value H R .
2. The method according to claim 1, characterized in that The relevant information of the data operation request includes the data record identifier and the data operation type involved in the data operation request.
3. The method according to any one of claims 1-2, characterized in that: The predetermined condition includes a predetermined time interval, that is, a consistency check is started every predetermined time interval.
4. The method according to any one of claims 1 to 2, characterized in that: The predetermined condition includes: the data operation history of the partial data record meets the corresponding condition.
5. The method according to any one of claims 1 to 2, characterized in that: The data record R is prohibited from accepting any write operation from the central server during the locking period, but can be read.
6. The method according to any one of claims 1-2, characterized in that: The system administrator sets an original trust level for the storage server based on its hardware reliability and / or software reliability.
7. The method according to claim 1, characterized in that In step 740, W i =Z i -aL i ; where a is a predefined constant.
8. The method according to claim 1, characterized in that: In step 740, W i =Z i -A i L i ; Among them, A i It is based on the storage server S i The value is determined by the specific circumstances.
9. The method according to claim 8, characterized in that And i =Z i / m.
10. The method according to claim 1, characterized in that The step 800 specifically includes: based on H R The corresponding storage server set SH R , the consistency coordination server randomly selects SH R A storage server is selected, a consistency read command is sent to it, data record R is read, and then a consistency correction command is sent to the storage server that needs to be corrected. The consistency correction command includes data record R and H. R The storage server that receives the consistency correction command corrects the locally stored data record R according to the data record R in the command, and R Verify the correctness of the correction results.
Citation Information
Patent Citations
Data consistency checking method, apparatus and device and storage medium
CN108573016A
Distributed system data final consistency method based on message queue
CN112579620A
Cooperative edge storage data privacy protection system and method based on block chain
CN115913677A
Distributed cache and data storage consistency processing system and method
CN116383227A
Method for verifying deterministic deletion of data in cloud storage
CN116451270A
