Data processing method and device, electronic equipment and storage medium

Abstract

Embodiments of the present application provide a data processing method and device, electronic equipment and storage medium, the method comprises: in the case of receiving the data modification instruction sent by the client, analyzing the original data carried in the data modification instruction; find the target consensus group from the preset consensus group set, wherein the target master block and the target slave block in the target consensus group locally store the original data; wherein, through the preset consensus algorithm, the master block and the slave block are elected from the preset multiple block groups respectively, and multiple consensus groups are obtained, and the consensus group set is composed of multiple consensus groups; the data modification instruction is sent to the target master block, so that the target master block executes the data modification instruction to modify the original data, and forwards the data modification instruction to the target slave block, so that the target slave block executes the data modification instruction to modify the original data. Thus, the consensus algorithm is used to ensure data consistency.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of Internet, and in particular to a data processing method and device, electronic equipment and storage medium. BACKGROUND

[0002] A distributed file storage system is usually composed of three parts: a client, a proxy server and a storage end. The client is responsible for receiving all data operation instructions of a user, the proxy server is responsible for authentication and forwarding all data operation instructions of the user to the corresponding storage end, and the storage end is usually composed of N storage servers, which store the data of the user.

[0003] Each storage server is physically divided into multiple blocks, each block has a unique identifier (block id). A group is usually composed of blocks on different storage servers, and the data stored in the blocks in the group is completely consistent, thereby achieving data backup. For example, a group is composed of three blocks, and the three blocks are located on different storage servers, and the data of the three blocks is completely consistent.

[0004] Currently, in the scenario of modifying data in a distributed file storage system, when a certain data is modified, if the data of a block in a group is successfully modified, for example, data a is modified to a', it is considered that the data modification instruction is executed successfully, but in fact, the data of the remaining blocks in the group may fail to be modified, so the data consistency problem cannot be guaranteed. SUMMARY

[0005] The purpose of the embodiment of the present application is to provide a data processing method and device, electronic equipment and storage medium to achieve the beneficial effect of guaranteeing data consistency. The specific technical solutions are as follows:

[0006] In a first aspect of the embodiment of the present application, a data processing method is first provided, the method comprising:

[0007] In the case of receiving a data modification instruction sent by a client, the original data carried in the data modification instruction is parsed;

[0008] A target consensus group is searched from a preset consensus group set, wherein the target master block and the target slave block in the target consensus group both store the original data locally;

[0009] Wherein, the master block and the slave block are elected from a plurality of block groups by a preset consensus algorithm, and a plurality of consensus groups are obtained, and the consensus group set is composed of the plurality of consensus groups;

[0010] sending the data modification instruction to the target master block to make the target master block execute the data modification instruction to modify the original data, and forwarding the data modification instruction to the target slave block to make the target slave block execute the data modification instruction to modify the original data.

[0011] In an optional implementation, the searching for the target consensus group from the preset consensus group set comprises:

[0012] analyzing target data carried in the data modification instruction, wherein the target data comprises data after modification of the original data;

[0013] obtaining a data modification amount threshold set for a data modification scenario, and determining whether a data amount of the target data is less than or equal to the data modification amount threshold;

[0014] If the data amount of the target data is less than or equal to the data modification amount threshold, searching for the target consensus group from the preset consensus group set.

[0015] In an optional implementation, the searching for the target consensus group from the preset consensus group set comprises:

[0016] analyzing target data carried in the data modification instruction, wherein the target data comprises data after modification of the original data;

[0017] obtaining a data level set for a data modification scenario, and determining whether a level of the target data is greater than or equal to the data level;

[0018] If the level of the target data is greater than or equal to the data level, searching for the target consensus group from the preset consensus group set.

[0019] In an optional implementation, the method further comprises:

[0020] receiving data modification success messages returned by the target master block and the target slave block, and counting a number of the data modification success messages;

[0021] determining whether the number of the data modification success messages is greater than or equal to a preset message number threshold, the preset message number threshold being less than or equal to an element number in the target consensus group;

[0022] If the number of the data modification success messages is greater than or equal to the preset message number threshold, returning a message of successful execution of the data modification instruction to a client;

[0023] If the number of the data modification success messages is less than the preset message number threshold, returning a message of failed execution of the data modification instruction to the client.

[0024] In an optional implementation, the method further includes:

[0025] If the data amount of the target data is greater than the data modification amount threshold, searching for a target block group from the plurality of block groups, wherein each block in the target block group locally stores the original data;

[0026] selecting a target block from the blocks included in the target block group, and reading the original data from the target block;

[0027] modifying the original data according to the target data, and writing the modified original data into the blocks included in the target block group.

[0028] In an optional implementation, the method further includes:

[0029] If the level of the target data is less than the data level, searching for a target block group from the plurality of block groups, wherein each block in the target block group stores the original data;

[0030] selecting a target block from the blocks included in the target block group, and reading the original data from the target block;

[0031] modifying the original data according to the target data, and writing the modified original data into the blocks included in the target block group.

[0032] In an optional implementation, the selecting a target block from the blocks included in the target block group includes:

[0033] obtaining a block selection threshold set for a data modification scenario, and determining whether the data amount of the target data is less than or equal to the block selection threshold;

[0034] If the data amount of the target data is less than or equal to the block selection threshold, randomly selecting a target block from the blocks included in the target block group according to a preset random selection rule.

[0035] In an optional implementation, the selecting a target block from the blocks included in the target block group further includes:

[0036] If the data amount of the target data is greater than the block selection threshold, determining storage servers on which each block in the target block group is located, and obtaining performance parameters of the storage servers;

[0037] randomly selecting a target block from the blocks in the target block group according to the performance parameters of the storage servers.

[0038] In an optional implementation, the performance parameter comprises at least one of the following: memory size, network condition, CPU performance;

[0039] The performance parameter of the storage server comprises at least one of the following: memory size, network condition, CPU performance.

[0040] The performance parameter of the storage server comprises at least one of the following: memory size, network condition, CPU performance.

[0041] The performance parameter of the storage server comprises at least one of the following: memory size, network condition, CPU performance.

[0042] In a second aspect of the embodiments of the present application, a data processing apparatus is further provided, and the apparatus comprises:

[0043] The data analysis module is configured to analyze original data carried in a data modification instruction sent by a client.

[0044] The consensus group searching module is configured to search for a target consensus group from a preset consensus group set, wherein the target master block and the target slave block in the target consensus group both store the original data locally.

[0045] The consensus group set is composed of a plurality of consensus groups, wherein the master blocks and the slave blocks are elected from a plurality of block groups respectively by using a preset consensus algorithm.

[0046] The data processing module is configured to send the data modification instruction to the target master block, so that the target master block executes the data modification instruction to modify the original data, and forwards the data modification instruction to the target slave block, so that the target slave block executes the data modification instruction to modify the original data.

[0047] In a third aspect of the embodiments of the present application, an electronic device is further provided, comprising a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory complete mutual communication through the communication bus.

[0048] The memory is configured to store a computer program.

[0049] The processor is configured to execute the program stored on the memory, and implement the data processing method in any of the first aspect.

[0050] In a fourth aspect of the embodiments of the present application, a storage medium is further provided, and the storage medium stores instructions, which, when executed on a computer, cause the computer to execute the data processing method in any of the first aspect.

[0051] In a fifth aspect of the embodiments of the present application, a computer program product containing instructions, which, when executed on a computer, cause the computer to perform the data processing method of any one of the first aspect is further provided.

[0052] The technical scheme provided by the embodiments of the present application, in the case of receiving the data modification instruction sent by the client, the original data carried in the data modification instruction is parsed, and a target consensus group is searched from a preset consensus group set, wherein the target master block and the target slave block in the target consensus group both store the original data locally, wherein the master block and the slave block are elected from a plurality of block groups by a preset consensus algorithm, a plurality of consensus groups are obtained, the consensus group set is composed of the plurality of consensus groups, the data modification instruction is sent to the target master block, so that the target master block executes the data modification instruction to modify the original data, and the data modification instruction is forwarded to the target slave block, so that the target slave block executes the data modification instruction to modify the original data. By the preset consensus algorithm, the master block and the slave block are elected from the plurality of block groups, the plurality of consensus groups are obtained, the consensus group set is composed of the plurality of consensus groups, the target consensus group is searched from the preset consensus group set, the data modification instruction is sent to the target master block in the target consensus group, so that the target master block executes the data modification instruction to modify the original data, and the target master block forwards the data modification instruction to the target slave block in the target consensus group, so that the target slave block executes the data modification instruction to modify the original data, so that the consensus algorithm is used to ensure data consistency. BRIEF DESCRIPTION OF DRAWINGS

[0053] The accompanying drawings, which are incorporated into and form part of the specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the application.

[0054] In order to more clearly illustrate the technical scheme in the embodiments of the present application or the prior art, the drawings needed in the embodiment or prior art description will be briefly introduced below, and obviously, other drawings can also be obtained by those skilled in the art without creative labor.

[0055] Figure 1 An implementation flowchart of a data processing method shown in the embodiments of the present application;

[0056] Figure 2 An application scenario diagram shown in the embodiments of the present application;

[0057] Figure 3 An implementation flowchart of another data processing method shown in the embodiments of the present application;

[0058] Figure 4An implementation flowchart of another data processing method shown in the embodiment of the present application is shown in the figure.

[0059] Figure 5 An implementation flowchart of another data processing method shown in the embodiment of the present application is shown in the figure.

[0060] Figure 6 An implementation flowchart of another data processing method shown in the embodiment of the present application is shown in the figure. DETAILED DESCRIPTION

[0061] To make the objectives, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described below in connection with the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application but not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the protection scope of the present application.

[0062] As shown in the figure, an implementation flowchart of a data processing method provided by the embodiment of the present application is executed by a proxy server and can specifically include the following steps: Figure 1 S101, in the case of receiving a data modification instruction sent by a client, parsing original data carried in the data modification instruction.

[0063] In the embodiment of the present application, a user can submit all data operation instructions on a client. The data operation instructions here can include data modification, storage, viewing and other instructions. In the data modification scenario, the user can submit a data modification instruction on the client, so that the client can receive the data modification instruction submitted by the user and send the data modification instruction to a proxy server.

[0064] For the proxy server, the data modification instruction sent by the client can be received, and in the case of receiving the data modification instruction sent by the client, the original data carried in the data modification instruction can be parsed. In addition, the data modification instruction carries target data, which is data after modification of the original data.

[0065] For example, as shown in the figure, for the proxy server, the data modification instruction sent by the client can be received, and in the case of receiving the data modification instruction sent by the client, the original data a carried in the data modification instruction can be parsed. In addition, the data modification instruction carries target data a', which is data after modification of the original data a.

[0066] Figure 2

[0067] ​​S102, find a target consensus group from the preset consensus group set, wherein the target master block and the target follower in the target consensus group locally store the original data.

[0068] In the embodiment of the present application, a plurality of block groups are pre-divided in the storage end (the storage end is usually composed of N storage servers), each block group includes a plurality of blocks, and the blocks included in each block group are located in different storage servers. For example, each block group includes three blocks: block 1, block 2, and block 3, and the blocks included in each block group are located in different storage servers, that is, block 1 is located in storage server 1, block 2 is located in storage server 2, and block 3 is located in storage server 3.

[0069] For the plurality of block groups, a master block (leader) and a follower are elected from the plurality of block groups by a preset consensus algorithm in the embodiment of the present application, so that a plurality of consensus groups are obtained, and the consensus group set is composed of the plurality of consensus groups. For example, for the block group one including three blocks: block 1, block 2, and block 3, a master block (leader) and two followers are elected from them by a consensus algorithm, and then services can be provided externally.

[0070] It should be noted that the consensus algorithm can be a raft algorithm, and can also be other consensus algorithms, and the embodiment of the present application does not limit this. Thus, a master block and a follower can be elected from the preset plurality of block groups by a preset consensus algorithm, and a plurality of consensus groups are obtained, and the consensus group set is composed of the plurality of consensus groups.

[0071] Based on this, in the embodiment of the present application, a target consensus group can be found from the preset consensus group set, wherein the target master block and the target follower in the target consensus group locally store the original data, which means that the target consensus group storing the original data is found from the preset consensus group set. For example, it is assumed that the consensus group set includes consensus group 1, consensus group 2, and consensus group 3, and consensus group 1 is found therefrom, and a master block (leader) and two followers in consensus group 1 locally store the original data a.

[0072] The target data in the data modification instruction is generally carried, and the target data in the data modification instruction is parsed in the embodiment of the present application, wherein the target data includes data modified based on the original data. For example, the original data a, and the data a' modified based on the original data a is the target data. Based on the data size of the target data, it is determined whether to perform the above step S102, that is, whether to select a data processing mode with better I / O performance.

[0073] Specifically, the embodiment of the present application acquires a data modification amount threshold set for a data modification scenario, and judges whether the data amount of the target data is less than or equal to the data modification amount threshold (for deciding whether to select a data processing mode with better I / O performance), and if the data amount of the target data is less than or equal to the data modification amount threshold, searches for a target consensus group from a preset consensus group set.

[0074] For example, a data modification amount threshold is preset in the embodiment of the present application, assuming 2KB, and the data processing mode with better I / O performance can be selected if the data amount of the data is less than or equal to this value, based on which, it is judged whether the data amount of the target data (for example, a') is less than or equal to 2KB, and if the data amount of the target data is less than or equal to 2KB, a target consensus group is searched from a preset consensus group set.

[0075] In addition, in view of the attribute that data generally has priority, it is considered that the higher the priority, the more important the data, at this time, the I / O performance requirement is higher, based on which, the embodiment of the present application can decide whether to execute the above step S102 based on the importance of the data, that is, whether to select a data processing mode with better I / O performance.

[0076] Specifically, the embodiment of the present application acquires a data level set for a data modification scenario (for deciding whether to select a data processing mode with better I / O performance), and judges whether the level of the target data is greater than or equal to the data level, and if the level of the target data is greater than or equal to the data level, a target consensus group is searched from a preset consensus group set.

[0077] For example, a data level is preset in the embodiment of the present application, assuming level 2 (the higher the level, the more important the data), which means that the data processing mode with better I / O performance can be selected if the level of the data is greater than or equal to this data level, based on which, the level of the target data (for example, a') (that is, the data level is level 3), from which it can be determined that the level of the target data is greater than level 2, and a target consensus group is searched from a preset consensus group set.

[0078] S103, the data modification instruction is sent to the target master block to make the target master block execute the data modification instruction to modify the original data, and the data modification instruction is forwarded to the target slave block to make the target slave block execute the data modification instruction to modify the original data.

[0079] In the embodiment of the present application, the data modification instruction can be sent (directly) to the target master block (that is, leader), so that the target master block (that is, leader) can receive the data modification instruction, and then can execute the data modification instruction to modify the original data, so that the target master block (that is, leader) completes the modification of the original data.

[0080] In addition, the target master block (i.e., leader) can forward the data modification instruction to the target slave block (i.e., follower), so that the target slave block (i.e., follower) can execute the data modification instruction to modify the original data, so that the target slave block (i.e., follower) completes the modification of the original data, so that the data consistency is ensured through the consensus algorithm.

[0081] For the target slave block (i.e., follower), if the execution of the data modification instruction is successful, a data modification success message can be returned to the target master block (i.e., leader), and the target master block (i.e., leader) can replace the target slave block (i.e., follower) to return the data modification success message. For the target master block (i.e., leader), if the execution of the data modification instruction is successful, a data modification success message can also be returned.

[0082] Based on this, the embodiment of the application can receive the data modification success messages returned by the target master block and the target slave block, and count the number of data modification success messages. It is determined whether the number of data modification success messages is greater than or equal to a preset message quantity threshold. The preset message quantity threshold is less than or equal to the number of elements in the target consensus group. If the number of data modification success messages is greater than or equal to the preset message quantity threshold, a message that the execution of the data modification instruction is successful is returned to the client. If the number of data modification success messages is less than the preset message quantity threshold, a message that the execution of the data modification instruction fails is returned to the client.

[0083] For example, the embodiment of the application receives the data modification success messages returned by the target master block (i.e., leader) and the target slave block (i.e., follower), counts the number of data modification success messages, and determines whether the number of data modification success messages is greater than or equal to a preset message quantity threshold (2). Here, the target consensus group includes one target master block (i.e., leader) and two target slave blocks (i.e., follower), which means that the number of elements in the target consensus group is 3, and the preset message quantity threshold is less than the number of elements in the target consensus group.

[0084] If the number of data modification success messages is greater than or equal to the preset message quantity threshold (2), it means that most of the elements in the target consensus group successfully execute the data modification instruction. At this time, it can be determined that the execution of the data modification instruction is successful, and a message that the execution of the data modification instruction is successful is returned to the client. If the number of data modification success messages is less than the preset message quantity threshold (2), it means that a very small part of the elements or none of the elements in the target consensus group successfully execute the data modification instruction. At this time, it can be determined that the execution of the data modification instruction fails, and a message that the execution of the data modification instruction fails is returned to the client.

[0085] By the description of the technical solutions provided by the embodiments of the application, in the case of receiving the data modification instruction sent by the client, the original data carried in the data modification instruction is analyzed, and a target consensus group is searched from the preset consensus group set, wherein the target master block and the target slave block in the target consensus group both locally store the original data, wherein the master block and the slave block are elected from the preset plurality of block groups by the preset consensus algorithm, a plurality of consensus groups are obtained, the consensus group set is formed by the plurality of consensus groups, the data modification instruction is sent to the target master block, so that the target master block executes the data modification instruction to modify the original data, and the data modification instruction is forwarded to the target slave block, so that the target slave block executes the data modification instruction to modify the original data.

[0086] By the preset consensus algorithm, the master block and the slave block are elected from the preset plurality of block groups, a plurality of consensus groups are obtained, the consensus group set is formed by the plurality of consensus groups, and subsequently, the target consensus group is searched from the preset consensus group set. The data modification instruction is sent to the target master block in the target consensus group, so that the target master block executes the data modification instruction to modify the original data. The target master block forwards the data modification instruction to the target slave block in the target consensus group, so that the target slave block executes the data modification instruction to modify the original data. In this way, the consensus algorithm is used to ensure data consistency.

[0087] As shown in FIG. 1, it is an implementation flow diagram of another data processing method provided by the embodiments of the application, which is executed by a proxy server and can specifically include the following steps: Figure 3

[0088] S301, if the data amount of the target data is greater than the data modification amount threshold, a target block group is searched from a plurality of block groups, wherein the blocks in the target block group both locally store the original data.

[0089] In the embodiments of the application, for the data amount of the target data, if the data amount of the target data is greater than the data modification amount threshold, it means that the data amount is large. At this time, a data processing mode with general I / O performance can be selected to relieve the storage end pressure and prevent the storage end from collapsing. Therefore, a target block group can be searched from a plurality of block groups, wherein the blocks in the target block group both locally store the original data.

[0090] For example, in the embodiments of the application, a data modification amount threshold is preset, which is assumed to be 2KB. If the data amount of the data is greater than this value, a data processing mode with general I / O performance can be selected. Based on this, a target block group can be searched from a plurality of block groups, wherein the blocks in the target block group both locally store the original data a.

[0091] S302, a target block is selected from the blocks included in the target block group, and the original data is read from the target block.

[0092] ​For the target block group, generally including a plurality of blocks, the embodiment of the present application selects a target block from the blocks included in the target block group, and reads the original data from the target block.

[0093] For example, for the target block group, including block 1, block 2, block 3, the embodiment of the present application selects block 1 as the target block from the blocks included in the target block group, and reads the original data a from the target block.

[0094] S303, modifying the original data according to the target data, and writing the modified original data into the blocks included in the target block group.

[0095] In the embodiment of the present application, the original data can be modified in the cache according to the target data, and the modified original data is (again) written into the blocks included in the target block group, so that the entire data processing flow undergoes one data reading and writing, and the I / O performance is generally, but the data consistency can be guaranteed.

[0096] For example, in the embodiment of the present application, the original data a is modified in the memory according to the target data a', so that the original data a is modified to the target data a', and then the modified original data (i.e. a') can be written into block 1, block 2 and block 3 again, so that the consistency of the data stored in all blocks in the target block group can be ensured, and the data consistency problem can be avoided.

[0097] As shown in Figure 4 Another embodiment of the data processing method provided by the embodiment of the present application is shown in the flowchart, which is executed by the proxy server, and can specifically include the following steps:

[0098] S401, if the level of the target data is less than the data level, searching for a target block group from a plurality of block groups, wherein the blocks in the target block group all store the original data.

[0099] In the embodiment of the present application, for the level of the target data, if the level of the target data is less than the data level, it means that the priority of the target data is not high, and the importance is general, at this time, the data processing mode with general I / O performance can be selected to relieve the storage end pressure and prevent the storage end from collapsing, so a target block group can be searched from a plurality of block groups, wherein the blocks in the target block group all store the original data.

[0100] For example, in the embodiment of the present application, a data level is pre-set, assuming that it is level 2 (the higher the level, the more important the data), which means that the level of the data is less than the data level, and the data processing mode with general I / O performance can be selected, based on which a target block group can be searched from a plurality of block groups, wherein the blocks in the target block group all store the original data a.

[0101] S402, selecting a target block from the blocks included in the target block group, and reading the original data from the target block.

[0102] For a target block group, generally including a plurality of blocks, an embodiment of the present application selects a target block from the blocks included in the target block group, and reads the original data from the target block.

[0103] For example, for a target block group including block 1, block 2, and block 3, an embodiment of the present application selects block 1 as the target block from the blocks included in the target block group, and reads the original data a from the target block.

[0104] S403, modifying the original data according to the target data, and writing the modified original data into the blocks included in the target block group.

[0105] In an embodiment of the present application, the original data can be modified according to the target data in the cache, and the modified original data can be written (again) into the blocks included in the target block group, so that the entire data processing flow undergoes one reading and writing of data, and thus the I / O performance is generally, but the data consistency can be guaranteed.

[0106] For example, in an embodiment of the present application, the original data a is modified according to the target data a', so that the original data a is modified to the target data a', and then the modified original data (i.e. a') can be written again into block 1, block 2, and block 3, so that the consistency of the data stored in all the blocks in the target block group can be ensured, and the data consistency problem can be avoided.

[0107] In addition, in an embodiment of the present application, the target block can be selected from the blocks included in the target block group by the following manner: obtaining a block selection threshold set for a data modification scenario, and determining whether the data amount of the target data is less than or equal to the block selection threshold, if the data amount of the target data is less than or equal to the block selection threshold, a target block is randomly selected from the blocks included in the target block group according to a preset random selection rule.

[0108] For example, an embodiment of the present application sets a block selection threshold for a data modification scenario in advance, assuming 0.5KB, and determines whether the data amount of the target data is less than or equal to 0.5KB, if the data amount of the target data is less than or equal to 0.5KB, it means that the modification amount of the data is small, and it is not important to select which block to read the original data, at this time, a target block can be randomly selected from the blocks included in the target block group according to a preset random selection rule.

[0109] In addition, if the data amount of the target data is greater than the block selection threshold, it indicates that the modification amount of the data is large, and a more optimal block needs to be selected to read the original data, therefore, the storage server where each block in the target block group is located is determined, and the performance parameters of the storage server are obtained; and the target block is selected from the blocks in the target block group according to the performance parameters of the storage server.

[0110] For example, if the data amount of the target data is greater than 0.5 KB, it indicates that the modification amount of the data is large, at this time, a more optimal block can be selected to read the original data, therefore, the storage server where each block in the target block group is located is determined, and the performance parameters of the storage server are obtained; and the target block is selected from the blocks in the target block group according to the performance parameters of the storage server.

[0111] Specifically, in the embodiment of the present application, the performance parameters at least include one of the following: memory size, network condition, and CPU performance, the election parameters corresponding to the memory size, network condition, and CPU performance of the storage server are obtained, the sum of the election parameters is obtained, or the weighted sum of the election parameters is obtained, and the target block is selected from the blocks in the target block group according to the sum or the weighted sum.

[0112] For the memory size, it can be generally understood as the capacity of the content, for example, 8 GB memory or 16 GB memory, for the network condition, it can be generally understood as network delay, and for the CPU performance, it can be generally understood as CPU model and parameters, etc., which are not limited in the embodiment of the present application.

[0113] For example, in the embodiment of the present application, the election parameters corresponding to the memory size, network condition, and CPU performance of the storage server (i.e., the storage server 1 where the block 1 is located, the storage server 2 where the block 2 is located, and the storage server 3 where the block 3 is located) are obtained, and here the election parameters can be scores, which means that different memory sizes have different scores, and similarly, different network conditions have different scores, and different CPU performances have different scores.

[0114] Based on this, the sum of the election parameters corresponding to the memory size, network condition, and CPU performance of the storage server 1 is obtained, or the weighted sum of the election parameters is obtained; the sum of the election parameters corresponding to the memory size, network condition, and CPU performance of the storage server 2 is obtained, or the weighted sum of the election parameters is obtained; and the sum of the election parameters corresponding to the memory size, network condition, and CPU performance of the storage server 3 is obtained, or the weighted sum of the election parameters is obtained.

[0115] For storage server 1, there exists a corresponding sum of election parameters or a weighted sum of election parameters. For storage server 2, there exists a corresponding sum of election parameters or a weighted sum of election parameters. For storage server 3, there exists a corresponding sum of election parameters or a weighted sum of election parameters. Based on these sums or weighted sums of election parameters, a target block is selected from the blocks in the target block group. Specifically, the maximum value is selected from these sums or weighted sums of election parameters, and then the block corresponding to this maximum value is selected as the target block from the blocks in the target block group.

[0116] Corresponding to the above method embodiments, this invention also provides a data processing apparatus, such as... Figure 5 As shown, the device may include: a data parsing module 510, a consensus group lookup module 520, and a data processing module 530.

[0117] The data parsing module 510 is used to parse the original data carried in the data modification instruction sent by the client when it receives the data modification instruction;

[0118] The consensus group lookup module 520 is used to look up a target consensus group from a preset consensus group set, wherein the target master block and the target slave block in the target consensus group both store the original data locally;

[0119] Among them, a master block and slave blocks are elected from multiple preset block groups through a preset consensus algorithm to obtain multiple consensus groups, and the consensus group set is composed of multiple consensus groups;

[0120] The data processing module 530 is used to send the data modification instruction to the target master block so that the target master block executes the data modification instruction to modify the original data, and forwards the data modification instruction to the target slave block so that the target slave block executes the data modification instruction to modify the original data.

[0121] This invention also provides an electronic device, such as... Figure 6 As shown, it includes a processor 61, a communication interface 62, a memory 63, and a communication bus 64, wherein the processor 61, the communication interface 62, and the memory 63 communicate with each other through the communication bus 64.

[0122] Memory 63 is used to store computer programs;

[0123] When processor 61 executes the program stored in memory 63, it performs the following steps:

[0124] In a case that the data modification instruction sent by the client is received, original data carried in the data modification instruction is parsed; a target consensus group is searched from a preset consensus group set, wherein the target master block and the target slave block in the target consensus group both locally store the original data; wherein a master block and a slave block are respectively elected from a plurality of block groups by a preset consensus algorithm to obtain a plurality of consensus groups, and the plurality of consensus groups form the consensus group set; the data modification instruction is sent to the target master block, so that the target master block executes the data modification instruction to modify the original data, and the data modification instruction is forwarded to the target slave block, so that the target slave block executes the data modification instruction to modify the original data.

[0125] The communication bus mentioned in the above electronic device can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For convenience of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or only one type of bus.

[0126] The communication interface is used for communication between the above electronic device and other devices.

[0127] The memory can include a Random Access Memory (RAM) and can also include a non-volatile memory, such as at least one disk memory. Optionally, the memory can also be at least one storage device located away from the aforementioned processor.

[0128] The processor mentioned above can be a general-purpose processor, including a Central Processing Unit (CPU), a Network Processor (NP), etc.; can also be a Digital Signal Processing (DSP), an Application Specific Integrated Circuit (ASIC), a Field-Programmable Gate Array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components.

[0129] In yet another embodiment of the present disclosure, there is provided a storage medium having instructions stored therein, which when executed on a computer, cause the computer to perform the data processing method of any one of the above embodiments.

[0130] In yet another embodiment of the present disclosure, there is provided a computer program product having instructions stored therein, which when executed on a computer, cause the computer to perform the data processing method of any one of the above embodiments.

[0131] In the above embodiments, the implementation can be wholly or partially in software, hardware, firmware, or any combination thereof. When implemented in software, the implementation can be in the form of a computer program product. The computer program product includes one or more computer instructions. When loaded and executed by a computer, the computer program instructions cause the computer to perform the processes or functions described in the embodiments of the present disclosure. The computer can be a general purpose computer, a special purpose computer, a computer network, or other programmable apparatus. The computer instructions can be stored in a storage medium or transmitted from one storage medium to another storage medium, for example, from a website site, a computer, a server, or a data center to another website site, a computer, a server, or a data center through a wired (such as a coaxial cable, an optical fiber, a digital subscriber line (DSL)) or wireless (such as infrared, wireless, microwave, etc.) manner. The storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server, data center, etc. that includes one or more available media sets. The available medium can be a magnetic medium (such as a floppy disk, a hard disk, a magnetic tape), an optical medium (such as a DVD), or a semiconductor medium (such as a solid state disk (SSD)), etc.

[0132] It should be noted that, in this document, the terms such as first and second are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply that there is any such actual relationship or order between these entities or operations. Moreover, the terms "include", "contain" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements not only includes those elements, but also includes other elements not explicitly listed or inherent to such a process, method, article or device. Without more limitations, the element defined by the statement "including a" does not exclude the presence of other identical elements in the process, method, article or device including the element.

[0133] The various embodiments in the specification are described in a related manner, and the same or similar parts between the various embodiments can be referred to each other. Each embodiment focuses on the difference from other embodiments. In particular, for the system embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the part of the method embodiments.

[0134] The above only describes the preferred embodiments of the present application, and is not intended to limit the protection scope of the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A data processing method, characterized in that, The method includes: Upon receiving a data modification instruction from the client, the original data carried in the data modification instruction is parsed. The target consensus group is searched from the preset consensus group set, wherein the target master block and the target slave block in the target consensus group both store the original data locally; Among them, a master block and slave blocks are elected from multiple preset block groups through a preset consensus algorithm to obtain multiple consensus groups, and the consensus group set is composed of multiple consensus groups; The step of searching for a target consensus group from a preset consensus group set includes: parsing the target data carried in the data modification instruction, wherein the target data includes data modified from the original data; obtaining a data modification amount threshold set for the data modification scenario, and determining whether the data amount of the target data is less than or equal to the data modification amount threshold; if the data amount of the target data is less than or equal to the data modification amount threshold, then searching for a target consensus group from the preset consensus group set. The data modification instruction is sent to the target master block so that the target master block executes the data modification instruction to modify the original data, and forwards the data modification instruction to the target slave block so that the target slave block executes the data modification instruction to modify the original data.

2. The method according to claim 1, characterized in that, The step of searching for the target consensus group from the preset consensus group set includes: The target data carried in the data modification instruction is parsed, wherein the target data includes the data modified from the original data; Obtain the data level set for the data modification scenario, and determine whether the level of the target data is greater than or equal to the data level; If the level of the target data is greater than or equal to the data level, then the target consensus group is searched from the preset consensus group set.

3. The method according to any one of claims 1 to 2, characterized in that, The method further includes: Receive data modification success messages returned by the target master block and the target slave block, and count the number of data modification success messages; Determine whether the number of data modification success messages is greater than or equal to a preset message number threshold, wherein the preset message number threshold is less than or equal to the number of elements in the target consensus group; If the number of data modification success messages is greater than or equal to the preset message number threshold, a message indicating that the data modification instruction was successfully executed is returned to the client. If the number of successful data modification messages is less than the preset message quantity threshold, a message indicating that the data modification instruction failed to execute is returned to the client.

4. The method according to claim 1, characterized in that, The method further includes: If the amount of the target data is greater than the data modification threshold, the target block group is searched from multiple block groups, wherein the original data is stored locally in each block of the target block group; Select a target block from the blocks included in the target block group, and read the original data from the target block; Modify the original data according to the target data, and write the modified original data into the blocks included in the target block group.

5. The method according to claim 2, characterized in that, The method further includes: If the level of the target data is lower than the data level, the target block group is searched from multiple block groups, wherein the blocks in the target block group all store the original data; Select a target block from the blocks included in the target block group, and read the original data from the target block; Modify the original data according to the target data, and write the modified original data into the blocks included in the target block group.

6. The method according to claim 4 or 5, characterized in that, Selecting a target block from the blocks included in the target block group includes: Obtain the block selection threshold set for the data modification scenario, and determine whether the data volume of the target data is less than or equal to the block selection threshold; If the amount of the target data is less than or equal to the block selection threshold, then a target block is randomly selected from the blocks included in the target block group according to a preset random selection rule.

7. The method according to claim 6, characterized in that, The step of selecting a target block from the blocks included in the target block group further includes: If the amount of the target data is greater than the block selection threshold, determine the storage server where each block in the target block group is located, and obtain the performance parameters of the storage server. Based on the performance parameters of the storage server, a target block is randomly selected from the blocks of the target block group.

8. The method according to claim 7, characterized in that, The performance parameters include at least one of the following: memory size, network status, and CPU performance; The step of randomly selecting a target block from the blocks of the target block group according to the performance parameters of the storage server includes: Obtain the election parameters corresponding to the memory size, network status, and CPU performance of the storage server; Obtain the sum of the election parameters, or obtain the weighted sum of the election parameters, and select a target block from the blocks of the target block group based on the sum of the election parameters or the weighted sum.

9. A data processing apparatus, characterized in that, The device includes: The data parsing module is used to parse the original data carried in the data modification instruction sent by the client when it receives the data modification instruction; The consensus group lookup module is used to find the target consensus group from a preset consensus group set, wherein the target master block and the target slave block in the target consensus group both store the original data locally; Among them, a master block and slave blocks are elected from multiple preset block groups through a preset consensus algorithm to obtain multiple consensus groups, and the consensus group set is composed of multiple consensus groups; The step of searching for a target consensus group from a preset consensus group set includes: parsing the target data carried in the data modification instruction, wherein the target data includes data modified from the original data; obtaining a data modification amount threshold set for the data modification scenario, and determining whether the data amount of the target data is less than or equal to the data modification amount threshold; if the data amount of the target data is less than or equal to the data modification amount threshold, then searching for a target consensus group from the preset consensus group set. The data processing module is used to send the data modification instruction to the target master block, so that the target master block executes the data modification instruction to modify the original data, and forwards the data modification instruction to the target slave block, so that the target slave block executes the data modification instruction to modify the original data.

10. An electronic device, characterized in that, It includes a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus; Memory, used to store computer programs; A processor, when executing a program stored in memory, implements the steps of the method described in any one of claims 1-8.

11. A storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method as described in any one of claims 1-8.

Images