Method, device, electronic equipment and medium for data processing and node deployment

Abstract

The present disclosure relates to a data processing and node deployment method, device, electronic equipment and medium. The method for data processing of a master node of a service cluster comprises: receiving a data submission request carrying to-be-stored data; segmenting and encoding the to-be-stored data to obtain target data containing a preset number of original data segments and check data segments, and generating a corresponding target log; storing the target log and the data segment in the target data that matches the master node according to a predefined node storage role, and sending a log replication instruction to a slave node in the service cluster; wherein, for each slave node, the log replication instruction carries information of the target log and the data segment in the target data that matches the current slave node; and generating a response result for the data submission request according to the execution feedback result of the slave node on the log replication instruction.

Description

Technical Field

[0001] This disclosure relates to the field of storage technology, and in particular to a method, apparatus, electronic device, and medium for data processing and node deployment. Background Technology

[0002] With the continuous development and updates of various applications, the amount of user data, maintenance data, and operational data is increasing exponentially, making storage capacity and reliability key indicators for evaluating storage systems. To improve the reliability, availability, and access efficiency of storage systems, distributed storage systems have emerged. Distributed storage systems employ a scalable system architecture, utilizing multiple storage servers to share the storage load and location servers to locate stored information. This not only improves the reliability, availability, and access efficiency of the storage system but also facilitates expansion.

[0003] However, current storage systems, based on log consistency-related consensus algorithms / protocols (such as Paxos and Raft), can achieve consistency in replica storage at the multi-replica management level. However, their application scenarios are limited, only supporting the situation where data is stored consistently across all nodes in the storage system, resulting in high data redundancy. Summary of the Invention

[0004] To address, or at least partially address, the aforementioned technical problems, embodiments of this disclosure provide a method, apparatus, electronic device, and medium for data processing and node deployment.

[0005] In a first aspect, embodiments of this disclosure provide a data processing method applied to a master node of a service cluster. The method includes: receiving a data submission request carrying data to be stored; segmenting and encoding the data to be stored to obtain target data containing a preset number of original data segments and checksum data segments, and generating a corresponding target log; storing the target log and data segments from the target data that match the master node according to a predefined node storage role, and sending a log replication instruction to slave nodes in the service cluster; wherein, for each slave node, the log replication instruction carries information about the target log and data segments from the target data that match the current slave node; and generating a response result for the data submission request based on the slave node's execution feedback result of the log replication instruction.

[0006] According to embodiments of this disclosure, the process of storing the target log and the data segment matching the master node according to a predefined node storage role, and sending log replication instructions to the slave nodes in the service cluster, includes: determining, according to the predefined node storage role, the data segment matching the master node and the data segment matching each slave node in the target data; storing the target log in a log file storage area; storing the data segment matching the master node in the master node's storage engine; and sending log replication instructions to the slave nodes in the service cluster, wherein the log replication instructions instruct each slave node to store the target log and its corresponding matching data segment.

[0007] According to embodiments of this disclosure, the service cluster includes T nodes for storage, where T is a positive integer. The segmentation and encoding of the data to be stored to obtain target data containing a preset number of original data segments and check data segments, and the generation of corresponding target logs, includes: dividing the data to be stored into K original data segments according to a preset number of segments K, where K is a positive integer; performing erasure coding on the K original data segments according to a preset number of checks M, to obtain M check data segments, where M is a natural number; wherein the K original data segments and the M check data segments constitute the target data, and the number of nodes T satisfies: T = K + M; assigning a corresponding data identifier to the target data; and appending a target log entry related to the data identifier of the target data to the end of the log sequence of the master node.

[0008] According to embodiments of this disclosure, the data segments matching the master node are stored in key-value pairs, the data identifier of the target data is stored as the key in the key-value pair, and the data segments in the target data that match the master node are stored as the corresponding values ​​in the key-value pair.

[0009] According to an embodiment of this disclosure, generating a response result for the data submission request based on the execution feedback result of the slave node on the log replication instruction includes: determining whether to submit the target log to the state machine based on whether the execution feedback result of the slave node meets the preset submission consensus conditions; and generating a response result for the data submission request based on the submission status of the target log.

[0010] According to embodiments of this disclosure, the number of original data segments is a preset K, and the number of verification data segments is a preset M, where K is a positive integer and M is a natural number; the consensus submission conditions include: when at least K′+F nodes among the master node and the slave node successfully store the target log, the target log is submitted to the state machine; where K′ represents the number of nodes corresponding to the original data segments, F is a preset integer value used to characterize fault tolerance reliability, and F satisfies the following condition: 0≤F≤M′, M′ represents the number of nodes corresponding to the verification data segments, the value of K′ is equal to the value of K, and the value of M′ is equal to the value of M.

[0011] According to an embodiment of this disclosure, in the above-mentioned service cluster, the conditions for generating the master node are as follows: during the current term, if the number of votes received by the target candidate node from the other candidate nodes in the above-mentioned service cluster exceeds Max(T / 2+1, K′+F), the target node is designated as the master node for the current term. Here, T is the total number of nodes used for storage in the service cluster, T=K′+M′, K′ represents the number of nodes corresponding to the original data segment, M′ represents the number of nodes corresponding to the verification data segment, and F is a preset integer value used to characterize fault tolerance reliability. F satisfies the following condition: 0≤F≤M′.

[0012] Secondly, embodiments of this disclosure provide a data processing method applied to slave nodes of a service cluster. For each slave node, the method includes: receiving a log replication instruction sent by a master node in the service cluster, the log replication instruction carrying: information of a target log, and a data segment in the target data matching the current slave node; wherein the target data includes a preset number of original data segments and checksum data segments obtained after segmentation and encoding of the data to be stored; storing the data segment in the target data matching the current slave node and the target log; monitoring a first storage state of the data segment matching the current slave node and a second storage state of the target log; and generating an execution feedback result for feeding back to the master node of the service cluster based on the first storage state and the second storage state.

[0013] According to embodiments of this disclosure, based on the first storage state and the second storage state, an execution feedback result for feeding back to the master node of the service cluster is generated, including: generating an execution feedback result indicating successful storage of the target log when both the first storage state and the second storage state are successful; and generating an execution feedback result indicating failed storage of the target log when at least one of the first storage state and the second storage state is a storage failure.

[0014] According to embodiments of this disclosure, the data segments matching the slave node are stored in key-value pairs, the data identifier of the target data is stored as the key in the key-value pair, and the data segments in the target data that match the slave node are stored as the corresponding values ​​in the key-value pair.

[0015] Thirdly, embodiments of this disclosure provide a method for deploying nodes in a service cluster. The method includes: receiving definition information that defines the storage of nodes in the service cluster, wherein the definition information is used to limit the storage mapping relationship between the nodes and data segment numbers; configuring node storage roles for each node according to the definition information; and generating master nodes and slave nodes in the service cluster according to the conditions for generating master nodes, each master node and slave node having a corresponding node storage role. The total number of data segment numbers is the sum of a preset number of segments and a preset number of checks, wherein the preset number of segments is used to divide the data to be stored to obtain original data segments, and the preset number of checks is used to limit the number of check data segments obtained after encoding the original data segments.

[0016] Fourthly, embodiments of this disclosure provide a data processing apparatus. The apparatus includes a master node of a service cluster, the master node comprising a request receiving module, a data processing module, a log storage and log replication module, and a response result generation module. The request receiving module receives a data submission request carrying data to be stored. The data processing module segments and encodes the data to be stored to obtain target data containing a preset number of original data segments and checksum data segments, and generates a corresponding target log. The log storage and log replication module stores the target log and the data segments in the target data that match the master node according to predefined node storage roles, and sends log replication instructions to the slave nodes in the service cluster; wherein, for each slave node, the log replication instruction carries information about the target log and the data segments in the target data that match the current slave node. The response result generation module generates a response result for the data submission request based on the execution feedback result of the slave node's response to the log replication instruction.

[0017] Fifthly, embodiments of this disclosure provide a data processing apparatus. The apparatus includes a slave node of a service cluster, the slave node comprising: an instruction receiving module, a storage module, a status monitoring module, and a feedback result generation module. The instruction receiving module receives a log replication instruction sent by the master node in the service cluster. The log replication instruction carries: information about a target log, and a data segment in the target data that matches the current slave node; wherein the target data includes a preset number of original data segments and checksum data segments obtained after segmentation and encoding of the data to be stored. The storage module stores the data segment in the target data that matches the current slave node and the target log. The status monitoring module monitors a first storage state of the data segment matching the current slave node and a second storage state of the target log. The feedback result generation module generates an execution feedback result to be fed back to the master node of the service cluster based on the first storage state and the second storage state.

[0018] Sixthly, embodiments of this disclosure provide an apparatus for deploying nodes in a service cluster. The apparatus includes: an information receiving module, a configuration module, and a master selection module. The information receiving module receives definition information defining the storage of nodes in the service cluster. The definition information defines the storage mapping relationship between the nodes and data segment numbers. The total number of data segment numbers is the sum of a preset number of segments and a preset number of checks. The preset number of segments is used to divide the data to be stored into original data segments, and the preset number of checks is used to limit the number of check data segments obtained after encoding the original data segments. The configuration module configures node storage roles for each node according to the definition information. The master selection module generates master nodes and slave nodes in the service cluster according to the conditions for generating master nodes. Each master node and slave node has a corresponding node storage role.

[0019] In a seventh aspect, embodiments of this disclosure provide an electronic device. The electronic device includes a processor, a communication interface, a memory, and a communication bus, wherein the processor, communication interface, and memory communicate with each other via the communication bus; the memory stores computer programs; and the processor, when executing the program stored in the memory, implements any of the data processing methods or node deployment methods described above.

[0020] Eighthly, embodiments of this disclosure provide a computer-readable storage medium. The computer-readable storage medium stores a computer program that, when executed by a processor, implements any of the data processing methods or node deployment methods described above.

[0021] The technical solutions provided in the embodiments of this disclosure have at least some or all of the following advantages:

[0022] Each node in the service cluster has a predefined node storage role. The master node segments and encodes the data to be stored to obtain target data containing a preset number of original data segments and verification data segments, and generates corresponding target logs. The master node can store matching data segments according to the predefined node storage roles. Moreover, when the master node sends log replication instructions to the slave nodes, it can allocate the matching data segments to each slave node for storage according to the predefined node storage roles. At the same time, the target logs stored by each slave node are consistent with those of the master node, realizing the organic combination of log replication consistency and error correction code (EC) storage. This enables the consensus algorithm to support EC storage. The data stored by each node can be different, expanding the application scenarios of log replication consistency and EC storage. It has broad application prospects in the field of distributed storage, effectively reducing data redundancy and improving storage utilization. Attached Figure Description

[0023] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure.

[0024] To more clearly illustrate the technical solutions in the embodiments of this disclosure or the prior art, the accompanying drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, those skilled in the art can obtain other drawings based on these drawings without creative effort.

[0025] Figure 1 The system architecture of the data processing method and apparatus applicable to embodiments of this disclosure is illustrated schematically;

[0026] Figure 2 A flowchart illustrating a method for data processing applied to a master node according to an embodiment of the present disclosure is shown schematically.

[0027] Figure 3 The detailed implementation process of operation S202 according to an embodiment of the present disclosure and an example of the composition of the obtained target data are illustrated schematically.

[0028] Figure 4 A detailed implementation flowchart of operation S203 according to an embodiment of the present disclosure is illustrated schematically;

[0029] Figure 5 A schematic diagram illustrating a detailed implementation process of operation S401 according to an embodiment of the present disclosure is provided.

[0030] Figure 6A detailed implementation flowchart of operation S204 according to an embodiment of the present disclosure is illustrated schematically;

[0031] Figure 7 A flowchart illustrating a method for data processing applied to a slave node according to an embodiment of the present disclosure is shown schematically.

[0032] Figure 8 An example of a process for data processing by a service cluster according to an embodiment of the present disclosure is illustrated schematically;

[0033] Figure 9 A flowchart illustrating a method for deploying nodes in a service cluster according to an embodiment of the present disclosure is shown schematically.

[0034] Figure 10 A structural block diagram of a data processing apparatus according to an embodiment of the present disclosure is shown schematically.

[0035] Figure 11 A structural block diagram of a data processing apparatus according to another embodiment of the present disclosure is shown schematically;

[0036] Figure 12 A structural block diagram of an apparatus for deploying nodes of a service cluster according to an embodiment of the present disclosure is schematically shown; and

[0037] Figure 13 A schematic block diagram of an electronic device provided in an embodiment of the present disclosure is shown. Detailed Implementation

[0038] The embodiments of this disclosure provide a method, apparatus, electronic device, and medium for data processing and node deployment, which realizes the organic combination of log replication consistency and error correction code (EC) storage, enabling consensus algorithms to support EC storage. The data stored on each node can be different, expanding the application scenarios of log replication consistency and EC storage. It has broad application prospects in the field of distributed storage and can effectively reduce data redundancy.

[0039] In embodiments of this disclosure, when the above data processing method is applied to the master node of a service cluster, the method includes: receiving a data submission request carrying data to be stored; segmenting and encoding the data to be stored to obtain target data containing a preset number of original data segments and check data segments, and generating a corresponding target log; storing the target log and the data segments in the target data that match the master node according to a predefined node storage role, and sending a log replication instruction to the slave nodes in the service cluster; wherein, for each slave node, the log replication instruction carries information about the target log and the data segments in the target data that match the current slave node; and generating a response result for the data submission request based on the execution feedback result of the slave node on the log replication instruction.

[0040] In embodiments of this disclosure, when the above data processing method is applied to a slave node of a service cluster, for each slave node, the method includes: receiving a log replication instruction sent by the master node in the service cluster, the log replication instruction carrying: information of a target log, and a data segment in the target data that matches the current slave node; wherein the target data includes a preset number of original data segments and check data segments obtained after the data to be stored has been segmented and encoded; storing the data segment in the target data that matches the current slave node and the target log; monitoring a first storage state of the data segment that matches the current slave node and a second storage state of the target log; and generating an execution feedback result for feeding back to the master node of the service cluster based on the first storage state and the second storage state.

[0041] To make the objectives, technical solutions, and advantages of the embodiments of this disclosure clearer, the technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this disclosure. Based on the embodiments of this disclosure, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this disclosure.

[0042] Figure 1 The system architecture of the data processing method and apparatus applicable to embodiments of this disclosure is illustrated schematically.

[0043] Reference Figure 1 As shown, the system architecture 100 of the data processing method and apparatus applicable to the embodiments of this disclosure includes: an electronic device 110 with data storage requirements and a service cluster 120. The service cluster 120 is connected to the electronic device 110 through a network. The service cluster 120 includes T service nodes for providing storage services to the electronic device 110, where T is a positive integer, i.e., T≥1.

[0044] exist Figure 1 Taking T=6 as an example, the service nodes (hereinafter referred to as nodes) in the service cluster 120 that provide data storage services for electronic device 110 are: node N1, node N2, node N3, node N4, node N5 and node N6.

[0045] The nodes N1 to N6 mentioned above have dual identities: identity A, which is used to execute the log replication consistency consensus algorithm / protocol, and identity B, which is used to store error correction codes. Here, identity B is described as the node storage role.

[0046] The aforementioned identity A includes a master node and a slave node. When T≥2, the master node and the slave node are different nodes, and the service cluster 120 can be a distributed storage architecture. When T=1, the corresponding service cluster has only one service node, which is a special case where the master node and the slave node are the same node.

[0047] Reference Figure 1 As shown, node N1 is the master node, and the other nodes N2 to N6 are slave nodes as examples.

[0048] Reference Figure 1 As shown, the node storage role corresponding to identity B is used to characterize the storage mapping relationship between each node used for storage and the data segment sequence number in the service cluster. For example, when deploying each node in service cluster 120, the node storage role C1 of node N1 is predefined to store the first data segment (part1) in the target data, the node storage role C2 of node N2 is used to store the second data segment (part2) in the target data, the node storage role C3 of node N3 is used to store the third data segment (part3) in the target data, the node storage role C4 of node N4 is used to store the fourth data segment (part4) in the target data, the node storage role C5 of node N5 is used to store the fifth data segment (part5) in the target data, and the node storage role C6 of node N6 is used to store the sixth data segment (part6) in the target data.

[0049] A network is the medium that provides a communication link between the aforementioned electronic device 110 and service cluster 120. A network can include various connection types, such as wired or wireless communication links or fiber optic cables, etc.

[0050] The nodes N1 to N6 in the aforementioned service cluster 120 can be servers that provide data storage services, including traditional database servers, cloud servers, and so on.

[0051] Taking a terminal device as an example of an electronic device 110 with data storage needs, the terminal device is equipped with various communication client applications, including but not limited to: shopping applications, web browser applications, search applications, instant messaging tools, email clients, social media platform software, etc. (for example only). The terminal device includes, but is not limited to, smartphones, tablets, laptops, desktop computers, smartwatches, self-driving cars, intelligent robots, etc. Users can use the terminal device to interact with the service cluster 120 via the network to receive or send messages, etc. The service cluster 120 provides data storage service support for data processing performed by users using applications or websites browsed on the terminal device.

[0052] It should be noted that the data processing method applied to the master node of the service cluster provided in this embodiment can generally be executed by the master node in the service cluster 120, and the data processing method applied to the slave node of the service cluster can generally be executed by the slave node in the service cluster 120. Accordingly, the data processing device corresponds to a device that includes a master node or a slave node.

[0053] For example, the master node in the service cluster 120 receives a data storage request from the electronic device 110, executes the corresponding data processing method, analyzes and processes the received data storage request, and feeds back the processing result to the electronic device 110. The slave node in the service cluster 120 receives a log replication instruction from the master node, executes the corresponding data processing method, performs log replication operation, and sends the generated execution feedback result to the master node.

[0054] Despite Figure 1 The example of the system architecture 100 consisting of terminal device 110 and service cluster 120 illustrates the scenarios in which this disclosure can be applied. The data processing methods and apparatus provided in the embodiments of this disclosure are not limited to the scenarios described above. In other application scenarios, in addition to terminal devices, devices with data storage needs can also be application servers or other electronic devices with storage service needs.

[0055] It should be understood that Figure 1 The number of electronic devices, service clusters, and service nodes shown is merely illustrative. Depending on implementation needs, there can be any number of electronic devices, service clusters, and service nodes.

[0056] The first exemplary embodiment of this disclosure provides a data processing method, which is applied to the master node of a service cluster.

[0057] Figure 2 A flowchart illustrating a method for data processing applied to a master node according to an embodiment of the present disclosure is shown schematically.

[0058] According to embodiments of this disclosure, referring to Figure 1 As shown, the service cluster consists of T nodes. When T ≥ 2, the master node and the slave node are different nodes, and this service cluster 120 can be a distributed storage architecture. When T = 1, the corresponding service cluster has only one service node, which is a special case where the master node and the slave node are the same node.

[0059] Reference Figure 2 As shown, the data processing method of this embodiment includes the following operations: S201, S202, S203, and S204. Operations S201 to S204 are performed by the master node in the service cluster (e.g., in...). Figure 1 The example uses node N1 as the master node, and the following description will use node N1 as the execution point. The master and slave nodes in the service cluster constitute a storage system.

[0060] In operation S201, a data submission request carrying data to be stored is received.

[0061] In an exemplary implementation scenario, the master node N1 may receive a data submission request for user data from an electronic device (such as a terminal device or application server) that has data storage needs. The data submission request carries data to be stored, which includes various types of data for storage, such as, but not limited to, user login data, user permission data, data after processing user requests, user browsing history, etc.

[0062] In operation S202, the data to be stored is segmented and encoded to obtain target data containing a preset number of original data segments and check data segments, and a corresponding target log is generated.

[0063] The generated target log is related to the target data mentioned above. For example, it can be a data identifier that records the target data. For example, if the user identifier corresponding to the data submission request (e.g., a Put request) is UserKey_1, and the data identifier of the target data is Bid_1, then the target log would be something like: "Put(UserKey_1, Bid_1)".

[0064] The aforementioned preset number is set based on the total number T of master and slave nodes in the storage system. K raw data segments and M parity data segments can be preset, requiring the condition: K + M = T. For example, refer to... Figure 1 As shown, there is one master node used for storage in service cluster 120. Figure 1 The example in the text is node N1), and there are 5 slave nodes. Figure 1In the example N2 to N6, there are a total of 6 nodes (one example of T), so K+M=6 needs to be satisfied. For example, taking K=4 and M=2 as the preset number, the number of nodes used to store K original data segments K′=4, and the number of nodes used to store M check data segments M′=2.

[0065] In operation S203, according to the predefined node storage role, the target log and the data segment in the target data that matches the master node are stored, and a log replication instruction is sent to the slave nodes in the service cluster; wherein, for each slave node, the log replication instruction carries the information of the target log and the data segment in the target data that matches the current slave node.

[0066] Based on predefined node storage roles, the data segments referenced by the target log have different storage mapping relationships on different nodes. Therefore, when the master node reads the target log, it can locate the corresponding data segment (original data segment or checksum data segment) on each node using the target data's data identifier (e.g., Bid_1) and the node storage role. For example, when master node N1 reads the target log "Put(UserKey_1, Bid_1)", it can obtain the corresponding matching data segments for each of the slave nodes N2 to N6 based on the target data's identifier "Bid_1" and the predefined node storage roles of each slave node. It then issues log replication instructions to each slave node N2 to N6 accordingly. That is, for each slave node, the log replication instruction carries the information of the target log and the data segment from the target data that matches the current slave node.

[0067] From the perspective of the service cluster as a whole, it can perform consistent replication operations on the target logs on the master and slave nodes according to the predefined node storage roles. During the consistent replication operation of the logs, the master and slave nodes store data segments that match their respective node storage roles.

[0068] In operation S204, a response to the above data submission request is generated based on the feedback result of the slave node's execution of the above log replication command.

[0069] Since the above log replication command carries information about the target log and a data segment that matches each slave node, the execution of the above log replication command by each slave node includes two aspects: copying the target log and storing the data segment that matches each slave node.

[0070] Based on the above operations S201-S204, each node in the service cluster has a predefined node storage role. After the master node segments and encodes the data to be stored, it obtains target data containing a preset number of original data segments and verification data segments, and generates corresponding target logs. The master node can store matching data segments according to the predefined node storage roles. Moreover, when the master node sends log replication instructions to the slave nodes, it can allocate the matching data segments to each slave node for storage according to the predefined node storage roles. At the same time, the target logs stored by each slave node are consistent with those of the master node, realizing the organic combination of log replication consistency and error correction code (EC) storage. This enables the consensus algorithm to support EC storage, and the data stored by each node can be different, expanding the application scenarios of log replication consistency and EC storage. It has broad application prospects in the field of distributed storage, effectively reducing data redundancy and improving storage utilization.

[0071] The following is combined Figure 3 The implementation process of operation S202 will be described in detail below.

[0072] Figure 3 The detailed implementation process of operation S202 according to an embodiment of the present disclosure and an example of the composition of the obtained target data are illustrated schematically.

[0073] In one embodiment, the service cluster includes T nodes for storage, where T is a positive integer. In operation S202, the data to be stored is segmented and encoded to obtain target data containing a preset number of original data segments and checksum data segments, and a corresponding target log is generated. This includes the following sub-operations: S2021, S2022, S2023, and S2024.

[0074] In sub-operation S2021, the data to be stored is divided into K original data segments according to the preset number of segments K, where K is a positive integer.

[0075] exist Figure 3 The example uses K=4. Generally, the preset number of segments K can be set to 2 or a value greater than 2, which corresponds to dividing the data to be stored into at least two original data segments; in some special implementation cases, the preset number of segments K can also be set to 1, in which case the execution logic is that the entire data segment is a single original data segment.

[0076] In sub-operation S2022, according to the preset number of checksums M, erasure coding (EC) is performed on the above K original data segments to obtain M checksum data segments, where M is a natural number; the above K original data segments and the above M checksum data segments constitute the target data, and the number of nodes T satisfies: T = K + M.

[0077] exist Figure 3 In this example, M=2 is used. Generally, the preset number of checks can be set to any integer greater than or equal to 1. Corresponding to scenarios where there are check data segments, the storage system corresponding to the service cluster provided in this embodiment can support target data composition modes that include both original data segments and check data segments, or it can support target data composition modes that only contain original data segments. Therefore, the preset number of checks can also be set to 0. When the preset number of checks K=0, it corresponds to entering the target data composition mode where the target data only contains original data segments.

[0078] From a practical application perspective, the data processing method provided in this disclosure is designed for massive amounts of data to be stored. Therefore, the preset numbers K and M will be set according to the size of the service cluster, the availability of the number of nodes, and the resource allocation.

[0079] For example, refer to Figure 3 As shown, the 4M (megabytes, 1024B (bits) equals 1KB, 1024KB equals 1M) of data to be stored 310 is segmented into four raw data segments, each containing 1M of data. Erasure coding (EC) is performed on these four raw data segments. For example, the RS algorithm can be used for EC encoding. By performing RS encoding on the four 1M raw data segments, two 1M parity data segments are obtained. The data segments (containing data content) corresponding to the data segment numbers part1, part2, part3, part4, part5, and part6 are: raw data segment 321, raw data segment 322, raw data segment 323, raw data segment 324, parity data segment 325, and parity data segment 326, respectively. These raw data segments 321, 322, 323, 324, 325, and 326 constitute the target data 320.

[0080] In sub-operation S2023, a corresponding data identifier is assigned to the aforementioned target data.

[0081] For example, refer to Figure 3 As shown, the target data 320 is assigned the corresponding data identifier Bid_1.

[0082] In sub-operation S2024, a target log entry related to the data identifier of the target data is appended to the end of the log sequence of the aforementioned master node.

[0083] Reference Figure 3As shown, the log sequence of the master node includes: log 1, log 2, log 3, ..., log X (the number of X is not limited, it is just an example). After the last log X, a target log related to the target data 320 is appended. For example, the target log is "Put(UserKey_1, Bid_1)".

[0084] Figure 4 A detailed implementation flowchart of operation S203 according to an embodiment of the present disclosure is illustrated schematically.

[0085] According to embodiments of this disclosure, referring to Figure 4 As shown, in the above operation S203, the target log and the data segment matching the master node are stored according to the predefined node storage role, and a log replication instruction is sent to the slave node in the service cluster, including the following operations: S401, S402 and S403.

[0086] In operation S401, based on the predefined node storage roles, the data segments matching the master node and the data segments matching each slave node are determined in the target data.

[0087] Figure 5 A schematic diagram illustrating a detailed implementation process of operation S401 according to an embodiment of the present disclosure is provided.

[0088] As described above regarding the system architecture 100, in the predefined node storage roles, nodes N1, N2, N3, N4, N5, and N6 are respectively assigned the data segment numbers applied to storage as part 1, part 2, part 3, part 4, part 5, and part 6. Based on the predefined node storage roles and the composition of the target data obtained in operation S202, the data segment matching the master node N1 is: original data segment 321; the data segment matching the slave node N2 is: original data segment 322; the data segment matching the slave node N3 is: original data segment 323; the data segment matching the slave node N4 is: original data segment 324; the data segment matching the slave node N5 is: checksum data segment 325; and the data segment matching the slave node N6 is: checksum data segment 326. Figure 5 As shown.

[0089] It should be noted that the part1 to part6 of the predefined node storage roles used for storage by each node are data segment numbers. The content of the specific data segment is not fixed. After receiving the data to be stored, the data to be stored is segmented and encoded before a data segment containing the specific storage content is generated.

[0090] For example, the data to be stored can be of different sizes, such as 4M and 6M respectively. After segmenting and encoding the data to be stored, it can be divided into a preset number of data segments, such as 4 original data segments and 2 check data segments. For 4M of data to be stored, we can get 4 original data segments of 1M and 2 check data segments of 1M; for 6M of data to be stored, we can get 4 original data segments of 1.5M and 2 check data segments of 1.5M.

[0091] In operation S402, the target log is stored in the log file storage area, and the data segment matching the master node is stored in the master node's storage engine.

[0092] For example, in one embodiment, the master node N1 stores the target log "Put(UserKey_1, Bid_1)" in the log file area (an example of a log file storage area) and stores the data segment matching the master node N1, the original data segment 321, in the master node's Blob Storage (an example of a storage engine).

[0093] According to an embodiment of this disclosure, the data segment matching the master node (e.g., the master node is node N1) is stored in the form of key-value pairs. The data identifier (e.g., Bid_1) of the target data (e.g., target data 320) is stored as the key in the key-value pair, and the data segment in the target data that matches the master node (e.g., the original data segment 321 that matches the master node N1) is stored as the value in the key-value pair.

[0094] In operation S403, a log replication instruction is sent to the slave nodes in the aforementioned service cluster. The log replication instruction is used to instruct each slave node to store the target log and its corresponding matching data segment.

[0095] The master node sends log replication commands to the slave nodes in the storage system of the service cluster.

[0096] Meanwhile, during the master node's term, the master node will periodically send heartbeat messages to all slave nodes in the service cluster's storage system according to preset rules to indicate its master node identity. In specific implementation scenarios, heartbeat messages and log replication instructions are executed sequentially according to their respective occurrence order; there may also be implementation scenarios where a single heartbeat message carries a log replication instruction simultaneously.

[0097] Figure 6 A detailed implementation flowchart of operation S204 according to an embodiment of the present disclosure is illustrated schematically.

[0098] According to embodiments of this disclosure, referring to Figure 6As shown, in the above operation S204, a response result for the above data submission request is generated based on the execution feedback result of the slave node on the above log replication instruction, including the following operations: S601 and S602.

[0099] In operation S601, based on whether the execution feedback results of the slave node meet the preset submission consensus conditions, it is determined whether to submit the target log to the state machine.

[0100] According to embodiments of this disclosure, the number of the original data segments is a preset K, and the number of the verification data segments is a preset M, where K is a positive integer and M is a natural number.

[0101] The above consensus submission conditions include: when at least K′+F nodes among the master node and the slave node successfully store the target log, the target log is submitted to the state machine; where K′ represents the number of nodes corresponding to the original data segment, F is a preset integer value used to characterize fault tolerance reliability, and F satisfies the following conditions: 0≤F≤M′, M′ represents the number of nodes corresponding to the verification data segment, the value of K′ is equal to the value of K, and the value of M′ is equal to the value of M.

[0102] For example, if K = K′ = 4, M = M′ = 2, and F = 1, then the consensus submission condition is: when at least 5 (K′ + F = 4 + 1 = 5) nodes among the master and slave nodes successfully store the target log, the target log is submitted to the state machine.

[0103] In some embodiments, the master node receives execution feedback results from the slave nodes. During the process of storing the target log in operation S202, the master node generates a storage result indicating successful storage. In one embodiment, the master node sends a log replication instruction to the slave nodes only when the storage is successful. Therefore, when at least 4 (K′+F-1=4+1-1) execution feedback results indicating successful storage of the target log are received from the slave nodes, the master node submits the target log to the state machine.

[0104] In other embodiments, the master node performs the write / storage operation of the target log. Regardless of success or failure, it sends a log replication instruction to the slave nodes. In the event of a write / storage failure by the master node, as long as the total number of slave nodes and master nodes is not less than K′+F, that is, in this case, the master node receives at least 5 (K′+F=4+1=5) execution feedback results from slave nodes indicating successful storage of the target log, the master node submits the target log to the state machine.

[0105] In one implementation scenario, when the execution feedback result of a slave node does not meet the preset commit consensus conditions, such as when the master node receives execution feedback results from only 3 slave nodes indicating that the target log has been successfully stored, the master node will reissue the log replication instruction to the slave nodes that have not yet provided execution feedback results or whose execution feedback results indicate that the target log has failed to be stored. If the execution feedback result of the slave node meets the commit consensus conditions within the preset response period, the target log will be committed to the state machine; otherwise, the target log will not be committed, and the target log will be in a commit state of failure.

[0106] When the execution feedback from the slave node meets the preset commit consensus conditions, the target log is committed to the state machine. After the master node commits / acknowledges the target log to the state machine, it broadcasts the consensus result to all slave nodes. This consensus result is a log sequence containing the target log (e.g., a sequence of logs containing the target log). Figure 3 The example log sequence is shown in the image. All nodes (including master and slave nodes) use their state machines (e.g., state machines containing user-related metadata) to execute (apply) the log sequence based on the committed log sequence containing the target log. If the log sequence is executed successfully, a commit state is generated indicating that the target log has been successfully committed; otherwise, a commit state is generated indicating that the target log has failed to be committed.

[0107] In operation S602, based on the submission status of the target log, a response result is generated for the data submission request.

[0108] When the target log shows a successful commit status, a response result indicating successful data commit is generated; when the target log shows a failed commit status, a response result indicating failed data commit is generated.

[0109] Based on the feedback results of the log replication commands executed by multiple slave nodes, the master node can obtain the consensus result of all nodes in the storage system of the service cluster (e.g., the 6 nodes in the example). Based on the consensus result, it determines whether to commit / acknowledge the target log to the state machine. If the target log is committed / acknowledged to the state machine, each node (including the master node and slave nodes) will execute (apply) according to the committed log sequence to generate the commit status of the target log, and then generate the response result for the data commit request based on the above commit status.

[0110] Based on the above consensus conditions, the master node will commit / acknowledge the target log to the state machine only when at least K′+F nodes among the master and slave nodes have successfully stored the target log. Since F satisfies 0≤F≤M′, when F is set to 0, the number of nodes that need to reach consensus to commit the target log to the state machine is at least K′. Based on the combination of error correction coding and log replication consistency, it can effectively ensure that even if the log data of any M nodes is lost / erroneous, all nodes can obtain the correct stored data based on the data of the remaining K′ nodes after encoding with error correction codes and performing log replication, and ensure the consistency of log replication. When F is set to any value of 0<F≤M′, based on the same principle, it can ensure the correctness and consistency of the stored data of each node and the efficient use of storage space, reducing data redundancy.

[0111] According to the embodiments of this disclosure, in the above-mentioned service cluster, the conditions for generating the master node are as follows: during the current term, if the number of votes received by the target candidate node from the other candidate nodes in the above-mentioned service cluster exceeds Max(T / 2+1, K′+F), the target node is designated as the master node for the current term. Here, T is the total number of nodes used for storage in the service cluster, T=K′+M′, K′ represents the number of nodes corresponding to the original data segment, M′ represents the number of nodes corresponding to the verification data segment, and F is a preset integer value used to characterize fault tolerance reliability. F satisfies the following condition: 0≤F≤M′.

[0112] By setting the master node election condition to Max(T / 2+1, K′+F), and selecting the larger of T / 2+1 and K′+F as the minimum voting value for master election, this setting is based on a comprehensive consideration of the following two aspects and has corresponding beneficial effects: Firstly, it ensures that only one node is elected as the master node at any given time, thus ensuring that the elected master node is unique and preventing multiple master nodes from being elected simultaneously. Secondly, it ensures that the elected master node has all committed log information and also guarantees that the elected master node has the possibility of fulfilling the consensus conditions. This differs from the majority vote requirement in the conventional Raft consensus algorithm / protocol.

[0113] The following example illustrates this. For instance, when T = 13, where K = K′ = 3, M = M′ = 10, and F = 1, if only T / 2 + 1 = 7 is set as the leader election condition, although a unique leader node can be elected, the consensus condition is K′ + F = 4. Therefore, at least four nodes must successfully store the target log and commit / acknowledge it to the state machine. If the four nodes that met the consensus condition in the previous commit are not among the seven nodes that meet the current commit consensus condition, the committed data already returned to the user will be lost. Therefore, this embodiment sets the leader election condition to the larger of T / 2 + 1 and K′ + F as the minimum voting value for leader election. In this case, 7 is used as the minimum voting value, ensuring the uniqueness of the leader node when one exists, and also ensuring that the elected leader node has all committed log information.

[0114] It is understandable that during the process of voting to elect a master node in the service cluster, there may be a situation where no master node can be elected within a certain election period. In this case, the timeout mechanism will trigger the node to conduct a new round of voting.

[0115] A second exemplary embodiment of this disclosure provides a data processing method applied to slave nodes of a service cluster, wherein each slave node can execute the data processing method of this embodiment. The overall information flow of data processing in the service cluster, as well as the interaction process between the master and slave nodes, can be understood by combining the first and second embodiments.

[0116] Figure 7 A flowchart illustrating a method for data processing applied to a slave node according to an embodiment of the present disclosure is shown schematically.

[0117] Reference Figure 7 As shown, the data processing method performed by each slave node provided in this embodiment of the disclosure includes the following operations: S701, S702, S703 and S704.

[0118] During operation S701, a log replication instruction is received from the master node in the aforementioned service cluster. The log replication instruction carries information about the target log and a data segment in the target data that matches the current slave node.

[0119] The target data mentioned above includes a preset number of original data segments and verification data segments obtained after the data to be stored has been segmented and encoded.

[0120] In one embodiment, the master node of the service cluster performs the above operations S201 and S202, and then performs operation S203, storing the target log and the data segment in the target data that matches the master node according to the predefined node storage role, and sending a log replication instruction to the slave node in the service cluster; correspondingly, the slave node performs operation S701, receiving the log replication instruction sent by the master node.

[0121] In operation S702, the data segment matching the current slave node and the target log from the target data are stored.

[0122] When the current slave node receives a log replication instruction, it stores the target log and the data segment that matches it; for example, referring to... Figure 3 and Figure 5 As shown, for each slave node: N2, N3, N4, N5, N6, the corresponding matching data segments are: original data segment 322, original data segment 323, original data segment 324, check data segment 325, and check data segment 326, respectively.

[0123] According to an embodiment of this disclosure, the data segment matching the slave node is stored in the form of key-value pairs, the data identifier of the target data (e.g., Bid_1) is stored as the key in the key-value pair, and the data segment in the target data that matches the slave node is stored as the value in the key-value pair.

[0124] In one embodiment, the target log is stored in the log file area of ​​the slave node (an example of a log file storage area), and the data segments matching each slave node are stored in the Blob Storage corresponding to each slave node (an example of a storage engine).

[0125] During operation S703, monitor the first storage state of the data segment that matches the current slave node and the second storage state of the target log.

[0126] In operation S704, based on the first storage state and the second storage state, an execution feedback result is generated for feeding back to the master node of the service cluster.

[0127] After the slave node executes operation S701, it executes operations S702-S703, and then operation S704, generating an execution feedback result to be fed back to the master node of the aforementioned service cluster based on the first storage state and the second storage state. For example, the slave node may further include a step of sending the execution feedback result to the master node, and correspondingly, the master node receives the execution feedback result fed back by the slave node (the communication process between the slave node and the master node). Then, the master node executes operation S204, generating a response result for the aforementioned data submission request based on the execution feedback result of the slave node on the aforementioned log replication instruction.

[0128] According to embodiments of this disclosure, based on the first storage state and the second storage state, an execution feedback result for feeding back to the master node of the service cluster is generated, including: generating an execution feedback result indicating successful storage of the target log when both the first storage state and the second storage state are successful; and generating an execution feedback result indicating failed storage of the target log when at least one of the first storage state and the second storage state is a storage failure.

[0129] Based on the above operations S701 to S704, each slave node can store different data segments from the same target data, and can achieve log replication consistency based on consensus algorithms / protocols. This achieves an organic combination of EC-encoded storage and log replication consistency, improving the storage space utilization of the service cluster and reducing data redundancy while ensuring the correctness of data storage.

[0130] In this embodiment, similar to the first embodiment, the condition for generating the master node in the aforementioned service cluster is as follows: During the current term, if a target candidate node in the service cluster receives more than Max(T / 2+1, K′+F) votes from other candidate nodes, the target node is designated as the master node for the current term. Here, T is the total number of nodes used for storage in the service cluster, T = K′+M′, K′ represents the number of nodes corresponding to the original data segment, M′ represents the number of nodes corresponding to the verification data segment, and F is a preset integer value used to characterize fault tolerance reliability. F satisfies the condition: 0 ≤ F ≤ M′. A detailed description can be found in the first embodiment, and will not be repeated here.

[0131] Figure 8 An example of a process for data processing by a service cluster according to an embodiment of the present disclosure is illustrated schematically.

[0132] Assuming that a master node and slave nodes have already been generated in the service cluster, Figure 8 In this example, the service cluster consists of one master node and five slave nodes. Figure 8The example uses only one slave node as a detailed example. The other slave nodes are generally similar in logic, and the similarities have been omitted. Only the differences between the corresponding stored matching data segments are shown.

[0133] Reference Figure 8 As shown, in operation step a (corresponding to operation S201 described in the first embodiment above), the master node receives a data submission request from the electronic device carrying 4M of data to be stored. The data submission request is a data submission request for user 1, and the identity identifier of user 1 is: UserKey_1.

[0134] Then, the master node executes operation step b (corresponding to operation S202 described in the first embodiment above), segments and encodes the above 4M data to be stored to obtain target data containing four 1M original data segments and two 1M check data segments, and generates the corresponding target log: "Put(UserKey_1, Bid_1)". In this embodiment, the process of segmenting and encoding the data to be stored is carried out in the cache, and the obtained target data and target log are first placed in the cache.

[0135] Next, the master node executes operation steps c1, c2, and d (corresponding to operation S203 described in the first embodiment above). The execution order and execution status of operation steps c1, c2, and d are not limited; in some embodiments, d may be executed after steps c1 and c2 are completed; in other embodiments, d may be executed simultaneously during the execution of c1 and c2; in still other embodiments, d may be executed first, followed by c1 and c2; additionally, d may be executed only if both c1 and c2 are successful, or d may be executed regardless of the results of c1 and c2. The execution order of c1 and c2 is not limited.

[0136] In operation step c1, according to the predefined node storage role, the data segment from the target data that matches the master node is stored, for example... Figure 8 The diagram uses six rectangles to represent the target data, and rectangles filled in the target data to represent the data segments that match the master node. It also demonstrates how to store the part1 (first segment) data segment that matches the master node into the master node's storage engine, Blob Storage.

[0137] In operation step c2, the target log mentioned above is stored, for example... Figure 8 The example in the example stores the target log "Put(UserKey_1, Bid_1)" into the log (log storage area) of the master node.

[0138] In operation step d, log replication commands are sent to slave nodes 1 through 5. For slave node 1, the log replication command sent carries the target log information "Put(UserKey_1, Bid_1)" and the data segment matching slave node 1: part2 (second segment). Similarly, for slave nodes 2, 3, 4, and 5, the log replication commands sent carry the target log information "Put(UserKey_1, Bid_1)" and the corresponding data segments matching slave nodes 2, 3, 4, and 5: part3 (third segment), part4 (fourth segment), part5 (fifth segment), and part6 (sixth segment), respectively. Figure 8 As shown.

[0139] Accordingly, log replication instructions are received from nodes 1 to 5 (corresponding to operation S701 described in the second embodiment).

[0140] Taking slave node 1 as an example, after slave node 1 receives information carrying the target log and a data segment (part2 data segment) matching slave node 1, it executes operation steps d1 and d2 (corresponding to operation S702 described in the second embodiment). The execution order of d1 and d2 is not limited.

[0141] In operation step d1, the data segment matching the current slave node 1 from the above target data, part2, is stored, for example, in... Figure 8 The example shows how to store the part2 data segment into the Blob Storage engine on node 1.

[0142] In operation step d2, the target log is stored, for example in... Figure 8 The example in the example stores the target log "Put(UserKey_1, Bid_1)" into the log (log storage area) of slave node 1.

[0143] Next, each slave node 1 to 5 monitors the first storage state of the data segment matching the current slave node and the second storage state of the target log, and generates an execution feedback result (corresponding to operations S703 and S704 in the second embodiment) for feeding back to the master node of the service cluster based on the first and second storage states. For simplicity, this is not explicitly stated here. Figure 8 The illustration is shown below.

[0144] Finally, each slave node 1 through 5 sends its execution feedback results to the master node. Figure 8 The example in the text uses operation step e to illustrate how node 1 sends its own execution feedback results to the master node.

[0145] After receiving the execution feedback result from the slave node, the master node generates a response result for the data commit request based on the slave node's execution feedback result of the aforementioned log replication command. The specific implementation process can be referred to the aforementioned description of operations S601 and S602, which is explained here... Figure 8 The example illustrates how, when the consensus conditions are met and the master node commits / confirms the target log to the state machine, each node ( Figure 8 The example illustrates the process by which the master node and slave node 1 will execute (apply) according to the committed log sequence. The example of the log sequence execution process of the master node is f1, and the example of the log sequence execution process of the slave node 1 is f2. The execution timing of f1 and f2 is different.

[0146] After each node executes the log sequence, a commit status for the target log is generated. The master node can then generate a response to the data commit request based on this commit status. Figure 8 The submission response result output by the state machine is used for description.

[0147] Similarly, for multiple data submission requests, each data submission request can be processed based on the data processing method provided in this disclosure embodiment. For example, regarding the data submission request for user 2, user 2's identity identifier is: UserKey_2, and the corresponding generated target log is in the form of... Figure 8 The example in the example is "Put(UserKey_2, Bid_2)".

[0148] Based on the same technical concept, a third exemplary embodiment of this disclosure provides a method for deploying nodes in a service cluster.

[0149] Figure 9 A flowchart illustrating a method for deploying nodes in a service cluster according to an embodiment of the present disclosure is shown schematically.

[0150] Reference Figure 9 As shown, the method for deploying nodes in a service cluster provided in this embodiment includes the following operations: S901, S902 and S903.

[0151] In operation S901, the system receives definition information for the storage definition of nodes in the service cluster, wherein the definition information is used to limit the storage mapping relationship between the nodes and data segment numbers.

[0152] The total number of data segment numbers (e.g., numbers 1 to 6 in part1 to 6 in the first embodiment) is the sum of the preset number of segments (e.g., K in the first embodiment, K=4) and the preset number of checks (e.g., M in the first embodiment, M=2). The preset number of segments is used to divide the data to be stored to obtain the original data segments, and the preset number of checks is used to limit the number of check data segments obtained after encoding the original data segments.

[0153] When operating S902, configure the node storage role for each node according to the above definition information.

[0154] The aforementioned node storage roles are used to characterize the storage mapping relationship between each node in the service cluster and the data segment sequence number.

[0155] In operation S903, based on the conditions for the generation of the master node, master nodes and slave nodes are generated in the above service cluster, and each of the master nodes and slave nodes has a corresponding node storage role.

[0156] Based on the above operations S901 to S903, dual identities are deployed for the nodes used for storage in the service cluster (e.g., nodes described as nodes in the storage system): node storage role (storage identity based on EC coding) and master-slave identity for log replication consistency execution. This achieves an organic combination of log replication consistency and error correction code (EC) storage, enabling the consensus algorithm to support EC storage. The data stored by each node can be different, expanding the application scenarios of log replication consistency and EC storage. It has broad application prospects in the field of distributed storage, effectively reducing data redundancy and improving storage utilization.

[0157] In this embodiment, similar to the first embodiment, the condition for generating the master node in the aforementioned service cluster is as follows: During the current term, if a target candidate node in the service cluster receives more than Max(T / 2+1, K′+F) votes from other candidate nodes, the target node is designated as the master node for the current term. Here, T is the total number of nodes used for storage in the service cluster, T = K′+M′, K′ represents the number of nodes corresponding to the original data segment, M′ represents the number of nodes corresponding to the verification data segment, and F is a preset integer value used to characterize fault tolerance reliability. F satisfies the condition: 0 ≤ F ≤ M′. A detailed description can be found in the first embodiment, and will not be repeated here.

[0158] A fourth exemplary embodiment of this disclosure provides an apparatus for data processing.

[0159] Figure 10 A structural block diagram of a data processing apparatus according to an embodiment of the present disclosure is shown schematically.

[0160] Reference Figure 10 As shown, the data processing apparatus 1000 provided in this embodiment includes: a master node of a service cluster, wherein the master node includes: a request receiving module 1001, a data processing module 1002, a log storage and log replication module 1003, and a response result generation module 1004.

[0161] The aforementioned request receiving module 1001 is used to receive data submission requests carrying data to be stored.

[0162] The data processing module 1002 is used to segment and encode the data to be stored to obtain target data containing a preset number of original data segments and check data segments, and to generate the corresponding target log.

[0163] The log storage and log replication module 1003 is used to store the target log and the data segment in the target data that matches the master node according to the predefined node storage role, and send log replication instructions to the slave nodes in the service cluster; wherein, for each slave node, the log replication instruction carries the information of the target log and the data segment in the target data that matches the current slave node.

[0164] The response result generation module 1004 is used to generate a response result for the data submission request based on the execution feedback result of the slave node on the above log replication instruction.

[0165] The fifth exemplary embodiment of this disclosure provides an apparatus for data processing.

[0166] Figure 11 A structural block diagram of a data processing apparatus according to another embodiment of the present disclosure is shown schematically;

[0167] Reference Figure 11 As shown, the data processing apparatus 1100 provided in this embodiment includes: a slave node of a service cluster, wherein the slave node includes: an instruction receiving module 1101, a storage module 1102, a status monitoring module 1103, and a feedback result generation module 1104.

[0168] The aforementioned instruction receiving module 1101 is used to receive the log replication instruction sent by the master node in the aforementioned service cluster. The log replication instruction carries: information of the target log, and data segments in the target data that match the current slave node; wherein the target data includes a preset number of original data segments and check data segments obtained after the data to be stored has been segmented and encoded.

[0169] The aforementioned storage module 1102 is used to store the data segment in the target data that matches the current slave node and the aforementioned target log.

[0170] The aforementioned status monitoring module 1103 is used to monitor the first storage status of the data segment matched with the current slave node and the second storage status of the target log.

[0171] The aforementioned feedback result generation module 1104 is used to generate an execution feedback result for feeding back to the master node of the aforementioned service cluster based on the aforementioned first storage state and the aforementioned second storage state.

[0172] A sixth exemplary embodiment of this disclosure provides an apparatus for deploying nodes of a service cluster.

[0173] Figure 12 A structural block diagram of an apparatus for deploying nodes of a service cluster according to an embodiment of the present disclosure is shown schematically.

[0174] Reference Figure 12 As shown, the device 1200 for deploying service cluster nodes provided in this embodiment includes: an information receiving module 1201, a configuration module 1202, and a master selection module 1203.

[0175] The aforementioned information receiving module 1201 is used to receive definition information for storing nodes in the service cluster. The definition information is used to limit the storage mapping relationship between the nodes and data segment numbers. The total number of data segment numbers is the sum of the preset number of segments and the preset number of checks. The preset number of segments is used to divide the data to be stored to obtain the original data segments. The preset number of checks is used to limit the number of check data segments obtained after encoding the original data segments.

[0176] The configuration module 1202 is used to configure node storage roles for each node according to the above definition information.

[0177] The aforementioned master selection module 1203 is used to generate master nodes and slave nodes in the aforementioned service cluster according to the conditions for generating master nodes. Each of the aforementioned master nodes and slave nodes has a corresponding node storage role.

[0178] Any multiple modules mentioned in the aforementioned devices 1000, 1100, and 1200 (e.g., request receiving module 1001, data processing module 1002, log storage and log replication module 1003, and response result generation module 1004 in device 1000; instruction receiving module 1101, storage module 1102, status monitoring module 1103, and feedback result generation module 1104 in device 1100; information receiving module 1201, configuration module 1202, and master selection module 1203 in device 1200) can be combined into one module, or any one of these modules can be split into multiple modules. Alternatively, at least some of the functions of one or more of these modules can be combined with at least some of the functions of other modules and implemented in one module. At least one of the modules mentioned in the aforementioned devices 1000, 1100, and 1200 can be at least partially implemented as hardware circuitry, such as a Field Programmable Gate Array (FPGA), a Programmable Logic Array (PLA), a System-on-a-Chip, a System-on-a-Substrate, a System-on-Package, an Application-Specific Integrated Circuit (ASIC), or any other reasonable means of integrating or packaging circuitry, or implemented in software, hardware, or firmware, or in any suitable combination of any of these three methods. Alternatively, at least one of the modules mentioned in the aforementioned devices 1000, 1100, and 1200 can be at least partially implemented as a computer program module, which, when run, can perform corresponding functions.

[0179] The seventh exemplary embodiment of this disclosure provides an electronic device.

[0180] Figure 13 A schematic block diagram of an electronic device provided in an embodiment of the present disclosure is shown.

[0181] Reference Figure 13 As shown, the electronic device 1300 provided in this embodiment includes a processor 1301, a communication interface 1302, a memory 1303, and a communication bus 1304. The processor 1301, the communication interface 1302, and the memory 1303 communicate with each other through the communication bus 1304. The memory 1303 is used to store computer programs. When the processor 1301 executes the program stored in the memory, it implements the data processing method or node deployment method described above.

[0182] An eighth exemplary embodiment of this disclosure also provides a computer-readable storage medium. The computer-readable storage medium stores a computer program that, when executed by a processor, implements the data processing method or node deployment method described above.

[0183] The computer-readable storage medium may be included in the device / apparatus described in the above embodiments; or it may exist independently and not assembled into the device / apparatus. The computer-readable storage medium carries one or more programs that, when executed, implement the method according to the embodiments of this disclosure.

[0184] According to embodiments of this disclosure, the computer-readable storage medium can be a non-volatile computer-readable storage medium, such as including, but not limited to: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this disclosure, the computer-readable storage medium can be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0185] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0186] The above description is merely a specific embodiment of this disclosure, enabling those skilled in the art to understand or implement it. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this disclosure. Therefore, this disclosure is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features claimed herein.

Claims

1. A method of data processing, characterized by, A method applied to a master node of a service cluster, the method comprising: receiving a data submission request carrying to-be-stored data; segmenting and encoding the to-be-stored data to obtain target data comprising a preset number of original data segments and check data segments, and generating a corresponding target log; storing the target log and data segments matching the master node in the target data according to a predefined node storage role, and sending a log replication instruction to a slave node in the service cluster; wherein, for each slave node, the log replication instruction carries information of the target log and data segments matching the current slave node in the target data; wherein the data segments matching the master node and the data segments matching the slave nodes are different data segments in the target data; the node storage role is used to indicate a storage mapping relationship between each node for storage in the service cluster and a data segment number, and different nodes in the storage mapping relationship are used to allocate the target data and correspond to different data segments; and According to whether a feedback result of execution of the log replication instruction by the slave node satisfies a preset commit consensus condition, a response result for the data commit request is generated; the commit consensus condition comprises: when at least one of the master node and the slave node +F nodes successfully store the target log, the target log is committed; wherein, represents a number of nodes corresponding to the original data segment, F is a preset integer value, used to represent fault tolerance reliability, and F satisfies the following condition: 0 , represents a number of nodes corresponding to the check data segment.

2. The method of claim 1, wherein, storing the target log and data segments matching the master node according to the predefined node storage role, and sending a log replication instruction to a slave node in the service cluster, comprising: determining data segments matching the master node and data segments matching each slave node in the target data according to the predefined node storage role; storing the target log in a log file storage area and storing the data segments matching the master node to a storage engine of the master node; and sending a log replication instruction to a slave node in the service cluster, the log replication instruction being used to instruct each slave node to store the target log and store the respective matching data segments.

3. The method of claim 1, wherein, The service cluster comprises T nodes for storage, T being a positive integer; segmenting and encoding the to-be-stored data to obtain target data comprising a preset number of original data segments and check data segments, and generating a corresponding target log, comprising: dividing the to-be-stored data into K original data segments according to a preset segment number K, K being a positive integer; performing erasure code encoding calculation on the K original data segments according to a preset check number M to obtain M check data segments, M being a natural number; wherein the K original data segments and the M check data segments constitute target data, and the number of nodes T satisfies: T = K + M; allocating a corresponding data identifier to the target data; and appending a target log related to the data identifier of the target data at the end of a log sequence of the master node.

4. The method of claim 3, wherein, The data segments matching the master node are stored in the form of key-value pairs, the data identifier of the target data is stored as a key in the key-value pair, and the data segments matching the master node in the target data are stored as a value in the key-value pair.

5. The method of claim 1, wherein, The response result for the data submission request is generated according to whether the execution feedback result of the slave node on the log replication instruction satisfies a preset submission consensus condition, comprising: determine whether to submit the target log to a state machine according to whether execution feedback of the slave node meets a preset submission consensus condition; and generate a response result for the data submission request according to a submission state of the target log.

6. The method of claim 5, wherein, The number of the original data segments is a preset K, and the number of the check data segments is a preset M, K is a positive integer, and M is a natural number; the value of K is equal to the value of K, the value of M is equal to the value of M.

7. The method of claim 1, wherein, In the service cluster, the generation condition of the master node is that in the current term, the target candidate node in the service cluster receives the number of votes from the remaining candidate nodes exceeding Max(T / 2+1, +F), the target candidate node is taken as the master node in the current term, wherein T is the total number of nodes in the service cluster for storage, T= + , represents the number of nodes corresponding to the original data segment, represents the number of nodes corresponding to the check data segment, F is a preset integer value, used to represent fault tolerance reliability, F satisfies the following condition: 0≤F≤ .

8. A method of data processing, characterized by, The method is applied to a slave node of a service cluster, and for each slave node, the method comprises: receiving a log replication instruction sent by a master node in the service cluster, the log replication instruction carrying information of a target log and data segments in target data that match a current slave node; the target data comprising a preset number of original data segments and check data segments obtained after segmenting and encoding processing of to-be-stored data; the data segments stored by the master node and matching each slave node are different data segments in the target data; storing the data segments in the target data that match the current slave node and the target log; monitoring a first storage state of the data segments that match the current slave node and a second storage state of the target log; According to the first storage state and the second storage state, an execution feedback result is generated for feedback to a master node of the service cluster, so that the master node generates a response result for a data commit request according to whether the execution feedback result meets a preset commit consensus condition; the commit consensus condition includes that at least +F nodes successfully store the target log, and the target log is committed; wherein, represents the number of nodes corresponding to the original data segment, F is a preset integer value, and is used to represent fault tolerance reliability, F meets the following condition: 0≤F≤ , represents the number of nodes corresponding to the check data segment.

9. The method of claim 8, wherein, generating an execution feedback result for feeding back to the master node of the service cluster according to the first storage state and the second storage state, comprising: generating an execution feedback result that the target log is stored successfully in a case where the first storage state and the second storage state are both storage successes; generating an execution feedback result that the target log is stored unsuccessfully in a case where at least one of the first storage state and the second storage state is a storage failure.

10. The method of claim 8, wherein, The data segments matching the slave node are stored in the form of key-value pairs, a data identifier of the target data is stored as a key in the key-value pair, and the data segments in the target data that match the slave node are stored as values in the key-value pair.

11. A method of node deployment for a service cluster, characterized in that, The method comprises: receiving definition information for defining storage of nodes in a service cluster, wherein the definition information is used to define a storage mapping relationship between the nodes and data segment serial numbers; the total number of the data segment serial numbers is a sum of a preset segment number and a preset check number, the preset segment number is used to divide to-be-stored data to obtain original data segments, and the preset check number is used to define a number of check data segments obtained after encoding of the original data segments; the original data segments and the check data segments constitute target data; configuring a node storage role for each node according to the definition information; different nodes in the storage mapping relationship are used to allocate the target data and correspond to different data segments; and According to a generation condition of a master node, a master node and a slave node are generated in the service cluster, and each of the master node and the slave node has a corresponding node storage role; wherein the data segment corresponding to the node storage role of the master node and the data segment corresponding to the node storage role of the slave node are different data segments in the target data; wherein the master node is configured to generate a response result for a data submission request according to whether an execution feedback result of a log replication instruction by the slave node satisfies a preset submission consensus condition; the submission consensus condition comprises: when at least +F nodes successfully store the target log, the target log is submitted; wherein, represents the number of nodes corresponding to the original data segment, F is a preset integer value, used to represent fault tolerance reliability, and F satisfies the following condition: 0≤F≤ , represents the number of nodes corresponding to the check data segment.

12. An apparatus for data processing, characterized by The device comprises a master node of a service cluster, and the master node comprises: a request receiving module configured to receive a data submission request carrying to-be-stored data; a data processing module configured to segment and encode the to-be-stored data to obtain target data comprising a preset number of original data segments and check data segments, and generate a corresponding target log; and a feedback sending module configured to generate an execution feedback result for feeding back to the master node of the service cluster according to a first storage state of the data segments matching the current slave node and a second storage state of the target log. The log storage and log replication module is configured to store data segments in the target data and the target log that match the master node according to a predefined node storage role, and send log replication instructions to slave nodes in the service cluster; wherein, for each slave node, the log replication instructions carry information of the target log and data segments in the target data that match the current slave node; wherein, the data segments that match the master node and the data segments that match the slave nodes are different data segments in the target data; the node storage role is used to indicate a storage mapping relationship between each node for storage in the service cluster and a data segment number, and different nodes in the storage mapping relationship are used to allocate the target data and correspond to different data segments; and The response result generation module is configured to generate a response result for the data commit request according to whether a feedback result of execution of the log replication instruction by the node satisfies a preset commit consensus condition; the commit consensus condition comprises: when at least one of the master node and the slave node successfully stores a target log, the target log is committed; wherein, F nodes successfully store the target log, the target log is committed; wherein, represents a number of nodes corresponding to the original data segment, F is a preset integer value, and is used to represent fault tolerance reliability. F satisfies the following condition: 0 , represents a number of nodes corresponding to the check data segment.

13. An apparatus for data processing, characterized by The device comprises a slave node of a service cluster, and the slave node comprises: An instruction receiving module is configured to receive log replication instructions sent by a master node in the service cluster, wherein the log replication instructions carry information of a target log and data segments in target data that match a current slave node; wherein the target data comprises a preset number of original data segments and check data segments obtained after segmenting and encoding processing of to-be-stored data; wherein the data segments stored by the master node and the data segments that match each slave node are different data segments in the target data. A storage module is configured to store the data segments in the target data that match the current slave node and the target log. A state monitoring module is configured to monitor a first storage state of the data segments that match the current slave node and a second storage state of the target log. The feedback result generation module is configured to generate an execution feedback result for feeding back to a master node of the service cluster according to the first storage state and the second storage state, so that the master node generates a response result for a data submission request according to whether the execution feedback result meets a preset submission consensus condition; the submission consensus condition comprises that when at least +F nodes successfully store the target log, the target log is submitted; wherein, represents the number of nodes corresponding to the original data segment, F is a preset integer value, used to represent fault tolerance reliability, and F meets the following condition: 0≤F≤ , represents the number of nodes corresponding to the check data segment.

14. An apparatus for deploying nodes of a service cluster, characterized by: The device comprises: An information receiving module is configured to receive definition information for defining storage of nodes in a service cluster, wherein the definition information is used to define a storage mapping relationship between the nodes and data segment numbers; the total number of the data segment numbers is a sum of a preset segment number and a preset check number, the preset segment number is used to divide to-be-stored data to obtain original data segments, and the preset check number is used to define a number of check data segments obtained after encoding of the original data segments; the original data segments and the check data segments constitute target data. A configuration module is configured to configure node storage roles for the nodes according to the definition information; different nodes in the storage mapping relationship are used to allocate the target data and correspond to different data segments; and The master selection module is configured to generate a master node and a slave node in the service cluster according to a generation condition of the master node, and the master node and the slave node each have a corresponding node storage role; wherein the data segment corresponding to the node storage role of the master node and the data segment corresponding to the node storage role of the slave node are different data segments in the target data; wherein the master node is configured to generate a response result for a data submission request according to whether the execution feedback result of the slave node on the log replication instruction satisfies a preset submission consensus condition; the submission consensus condition includes: when at least +F nodes successfully store the target log, the target log is submitted; wherein, represents the number of nodes corresponding to the original data segment, F is a preset integer value, used to represent fault tolerance reliability, and F satisfies the following condition: 0≤F≤ , represents the number of nodes corresponding to the check data segment.

15. An electronic device, comprising: The device comprises a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory complete communication with each other through the communication bus. The memory is used to store a computer program. The processor is used to execute the program stored on the memory, and implement the method in any one of claims 1-11.

16. A computer readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement the method in any one of claims 1-11. The computer program is executed by the processor to implement the method in any one of claims 1-11.

Images