Data storage method, device, electronic device and storage medium
By confirming that at least n data blocks are pre-stored successfully at the erasure end, the storage instructions are sent to the data node, the problem of successful misjudgment of data storage in the prior art is solved, the data storage efficiency is improved and the consistency of data block status is ensured.
Patent Information
- Application Number
- CN202210475295.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-04-29
- Publication Date
- 2024-11-12
- Estimated Expiration
- 2042-04-29
AI Technical Summary
In the prior art, due to network abnormalities and other reasons, the erasure end may misjudgment the number of data nodes that have successfully stored data is less than the number of data nodes that have been successfully stored, resulting in misjudgment as data storage failure in case of successful storage, affecting the data. Redundancy and correctness.
After confirming that at least n data blocks are pre-stored successfully, storage instructions are sent to each data node to complete the storage of the data blocks, and when each data node completes storage, the data storage is confirmed to be successful to ensure that the data block status is consistent.
It solves the problem of successful misjudgment of data storage, improves the efficiency of data storage, avoids data block residue and redundancy, and ensures the consistency of data block status on data nodes.
Smart Images

Figure CN114860167B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of data storage, and in particular to a data storage method, device, electronic equipment and storage medium. Background Art
[0002] When storing massive amounts of data, in order to improve data security and high availability, the erasure end uses erasure technology to process the original data after receiving it, obtains multiple data blocks, and stores the multiple data blocks in various data nodes of the distributed cluster. When the user needs to read the data, the data block is obtained from each data node, and the erasure calculation is performed to obtain the original data.
[0003] In the prior art, after each data node receives a data block, it stores the data block. The erasure end determines the number of data nodes that successfully store the data block, and determines whether the data is successfully stored based on the number of data nodes that successfully store the data block.
[0004] However, due to network anomalies and other reasons, the number of data nodes that successfully store the data block determined by the erasure end may be smaller than the number of data nodes that actually successfully store the data block, resulting in a misjudgment of data storage failure when the data is actually stored successfully. Summary of the invention
[0005] The objectives of the present invention include, for example, providing a data storage method, which can solve the problem of misjudging actual data storage success as data storage failure in the prior art by sending a storage instruction to the data node that successfully stored the data block after confirming that at least n data blocks have been pre-stored successfully, and confirming that the data storage is successful when each first data node completes the storage of its own data block, thereby improving the efficiency of data storage.
[0006] In order to achieve the above purpose, the technical solution adopted by the embodiment of the present invention is as follows:
[0007] In a first aspect, an embodiment of the present invention provides a data storage method, which is applied to an erasure end, wherein the erasure end respectively establishes communication connections with N data nodes, and the erasure end pre-stores N data blocks, wherein the N data blocks include n original data blocks, and n is less than N; the method comprises:
[0008] Sending the N data blocks to the N data nodes for pre-storage respectively, one data block corresponds to one data node;
[0009] When at least n first data nodes successfully pre-store the data block, a storage instruction is sent to each of the first data nodes, the storage instruction is used to instruct the first data node to complete the storage of its own data block, wherein the first data node is a data node that successfully pre-stores the data block among the N data nodes;
[0010] When each of the first data nodes completes storage of its own data blocks, it is confirmed that the data storage is successful.
[0011] In a possible implementation manner, a communication connection is established between any two of the data nodes; each of the data blocks includes a state identifier and a version number, and the version number is used to represent the sequence number of the current data storage operation;
[0012] The first data node completes storage of its own data block, including:
[0013] The first data node determines in real time within a preset time whether the storage instruction is received;
[0014] If yes, the first data node changes the state flag of its own data block from stateless to storage state to complete the storage of its own data block, and feeds back a response message to the erasure end;
[0015] If not, the first data node obtains the first version number and the first status identifier of each first data block, and processes its own data block according to each first version number and each first status identifier to complete the storage of its own data block, wherein one first data block corresponds to one data block pre-stored by the first data node.
[0016] In a possible implementation manner, the first data node processes its own data block according to each of the first version numbers and each of the first state identifiers to complete storage of its own data block, including:
[0017] The first data node determines that the first state identifier is a target version number of any one of the first data blocks in the storage state;
[0018] If the version number of the target data block is the same as the target version number, the first data node changes the state identifier of the target data block from stateless to storage state to complete the storage of the target data block, wherein the target data block is the data block of the first data node itself.
[0019] In a possible implementation manner, after the step of respectively sending the N data blocks to the N data nodes for pre-storage, the method further includes:
[0020] When p of the first data nodes successfully pre-store data blocks, a rollback instruction is sent to each of the data nodes, and the rollback instruction is used to instruct the data node to roll back its own data block to the correction and erasure end, and to delete its own data block when rollback fails, wherein p is less than n.
[0021] In a possible implementation manner, a communication connection is established between any two of the data nodes;
[0022] The data node deletes its own data block when the rollback fails, including:
[0023] When the data node fails to roll back its own data block, obtaining a second state identifier of a data block corresponding to each data node;
[0024] If each of the second state identifiers is in a stateless state, the data node deletes its own data block.
[0025] In a possible implementation manner, after the step of sending a storage instruction to each of the first data nodes, the method further includes:
[0026] Within the preset time, determining whether a response message returned by at least one of the first data nodes is received;
[0027] If yes, then determining that each of the first data nodes has completed storage of its own data blocks;
[0028] If not, it is confirmed that the data storage has failed.
[0029] In a possible implementation manner, the erasure correction end is deployed in an electronic device, and the electronic device is further deployed with at least one slave erasure correction end;
[0030] Before the step of determining whether a response message returned by at least one first data node is received within the preset time, the method further includes:
[0031] If an exception occurs on the erasure end, any one of the slave erasure ends is used as the erasure end, so that the erasure end receives each of the response messages.
[0032] In a second aspect, an embodiment of the present invention further provides a data storage device, which is applied to an erasure end, wherein the erasure end establishes communication connections with N data nodes respectively, and the erasure end pre-stores N data blocks, wherein the N data blocks include n original data blocks, and n is less than N; the device comprises:
[0033] A pre-storage module, used to send the N data blocks to the N data nodes for pre-storage respectively, one data block corresponding to one data node;
[0034] A storage module, configured to send a storage instruction to each of the first data nodes when at least n first data nodes successfully pre-store the data block, wherein the storage instruction is used to instruct the first data node to complete storage of its own data block, wherein the first data node is a data node that successfully pre-stores the data block among the N data nodes;
[0035] The confirmation module is used to confirm that the data storage is successful when each of the first data nodes completes the storage of its own data blocks.
[0036] In a third aspect, an embodiment of the present invention further provides an electronic device, the electronic device comprising:
[0037] one or more processors;
[0038] The memory is used to store one or more programs. When the one or more programs are executed by the one or more processors, the one or more processors implement the above-mentioned data storage method.
[0039] In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium having a computer program stored thereon, and the computer program implements the above-mentioned data storage method when executed by a processor.
[0040] Compared with the prior art, the embodiments of the present invention provide a data storage method, device, electronic device and storage medium. First, N data blocks are sent to N data nodes for pre-storage respectively, and one data block corresponds to one data node; when at least n first data nodes successfully pre-store the data blocks, a storage instruction is sent to each first data node, and the storage instruction is used to instruct the first data node to complete the storage of its own data block; when each first data node completes the storage of its own data block, the data storage is confirmed to be successful. By first sending the data block to the data node for pre-storage, and then confirming that at least n data blocks have been pre-stored successfully, and then sending the storage instruction to the data node that successfully stored the data block, so that the data node stores the data block, it can solve the problem of misjudging the actual data storage success as data storage failure in the prior art, and improve the efficiency of data storage. BRIEF DESCRIPTION OF THE DRAWINGS
[0041] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments are briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without creative work.
[0042] Figure 1 A schematic diagram of an application scenario of the data storage method provided in an embodiment of the present invention.
[0043] Figure 2 A block diagram of an electronic device provided by an embodiment of the present invention.
[0044] Figure 3 A schematic diagram of a flow chart of a data storage method provided in an embodiment of the present invention.
[0045] Figure 4 A schematic diagram of a storage process of a single data block provided by an embodiment of the present invention.
[0046] Figure 5 A schematic diagram of another storage process of a single data block provided by an embodiment of the present invention.
[0047] Figure 6 A schematic flow chart of another data storage method provided by an embodiment of the present invention.
[0048] Figure 7 A flowchart of another data storage method provided by an embodiment of the present invention.
[0049] Figure 8 A block diagram of a data storage device provided by an embodiment of the present invention.
[0050] Icons: 100 - electronic device; 101 - memory; 102 - processor; 103 - bus; 200 - data storage device; 201 - pre-storage module; 202 - fallback module; 203 - storage module; 204 - detection module; 205 - confirmation module. DETAILED DESCRIPTION
[0051] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Generally, the components of the embodiments of the present invention described and shown in the drawings here can be arranged and designed in various different configurations.
[0052] Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the invention claimed for protection, but merely represents selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0053] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, further definition and explanation thereof is not required in subsequent drawings.
[0054] In addition, the terms “first”, “second”, etc., if used, are merely used to distinguish between the descriptions and should not be understood as indicating or implying relative importance.
[0055] It should be noted that, in the absence of conflict, the features in the embodiments of the present invention may be combined with each other.
[0056] When erasure technology is used for distributed data storage, the erasure end performs erasure processing on the data to obtain multiple data blocks, and the multiple data blocks are sent to multiple data nodes for storage. When each data block is stored successfully, the erasure end confirms that the data storage is successful.
[0057] However, due to network anomalies and other reasons, the number of data blocks successfully stored learned by the erasure end is different from the number of data blocks actually stored, and the data storage is judged to have failed. At this time, the data needs to be stored again. However, the data blocks that have been successfully stored cannot be rolled back and will remain on the data node, affecting the redundancy of the data when the data is stored next time.
[0058] In order to solve the problem that data blocks remain on data nodes when the erasure end determines that data storage has failed, the technicians have improved the above method. The erasure end first sends the data blocks to the data nodes for pre-storage, and when it is learned that the number of data nodes that have successfully pre-stored is less than the preset number, it determines that the data storage has failed and sends a rollback instruction to each data node, so that each data node rolls back the data block.
[0059] When the erasure correction end learns that the number of data nodes that have successfully pre-stored is greater than the preset number, it sends a confirmation instruction to each data node that has successfully pre-stored, so that the data node confirms the pre-stored data block, so that the data block changes from the pre-stored state to the stored state, and feeds back a response message. When the erasure correction end receives the response message fed back by each data node that has successfully pre-stored, it confirms that the data is successful.
[0060] However, when the erasure correction end learns that the number of data nodes that have successfully pre-stored is greater than the preset number, due to network anomalies and other reasons, the confirmation instructions sent to each data node that has successfully pre-stored may be lost, resulting in some data nodes being unable to receive the confirmation instructions, and then the number of response messages received by the erasure correction end is less than the number of confirmation responses that should actually be received, and the data storage is determined to have failed.
[0061] At this time, the data blocks corresponding to the data nodes that have received the confirmation instructions have been successfully stored and cannot be rolled back. The data blocks corresponding to the data nodes that have not received the confirmation instructions are in the pre-storage state, resulting in inconsistency of the data blocks between the data nodes. In addition, the data blocks in the pre-storage state will remain on the data nodes. The next time the data nodes receive new pre-storage data blocks and store the pre-storage data blocks according to the confirmation instructions, it is possible that the data blocks in the pre-storage state that were previously left on the data nodes will be stored, resulting in reduced data redundancy or damage to the correctness of the data.
[0062] In view of the problem that some data nodes cannot receive confirmation instructions due to network anomalies and other reasons, and remain in the pre-stored state on the data nodes, this embodiment provides a data storage method, by sending a storage instruction to each first data node when the erasure end confirms that at least n first data nodes have successfully pre-stored data blocks, so that the first data node completes the storage of its own data block, and confirms the data success when each first data node completes the storage of its own data block. Thus, the state of the data block on each data node is guaranteed to be consistent.
[0063] Please refer to Figure 1 , Figure 1 The schematic diagram of the application scenario of the data storage method provided by the present embodiment shows: an erasure end, a client and N data nodes respectively communicating with the erasure end, and a communication connection is established between any two data nodes.
[0064] The client is used to send the data to be stored entered by the user to the erasure end, and receive the message returned by the erasure end indicating the data storage success or data storage failure. The client can be an application installed on a mobile phone, tablet, laptop or server.
[0065] The erasure end may be an application deployed on an electronic device, which is used to receive the data to be stored sent by the client, process the data to be stored to obtain N data blocks, and send the N data blocks to N data nodes for storage. The N data blocks include: n data blocks obtained by dividing the data to be stored, and m check blocks generated according to the preset erasure rules. For example, the erasure end divides the data to be stored into N data blocks: C 1 , C 2 …C N , and C 1 Send to data node 1, C 2 Send to data node 2... N Sent to data node N.
[0066] Data nodes 1 to N may be applications deployed on the server. Each server deploys a data node for storing data blocks sent by the erasure end.
[0067] Please refer to Figure 2 , Figure 2 A block diagram of an electronic device 100 provided in the present embodiment is shown, in which a correction and erasure terminal is deployed. The electronic device 100 may be, but is not limited to, a mobile phone, a tablet computer, a laptop computer, a server or other electronic device with processing capabilities, or a general-purpose computer or a computer for special purposes. The electronic device 100 includes a memory 101, a processor 102 and a bus 103. The memory 101 and the processor 102 are connected via a bus 103, for example, a disk, a ROM, or a RAM, or any combination thereof. Exemplarily, the electronic device 100 may also include program instructions stored in a ROM, a RAM, or other types of non-temporary storage media, or any combination thereof. The method of the present invention can be implemented according to these program instructions.
[0068] The memory 101 is used to store programs, such as an erasure end, and the erasure end includes a data storage device 200. The data storage device 200 includes at least one software function module that can be stored in the memory 101 in the form of software or firmware. After receiving the execution instruction, the processor 102 executes the program to implement the data storage method in this embodiment.
[0069] Among them, the memory 101 can be, but is not limited to, random access memory (Random Access Memory, RAM), read only memory (Read Only Memory, ROM), programmable read-only memory (Programmable Read-Only Memory, PROM), erasable programmable read-only memory (Erasable Programmable Read-Only Memory, EPROM), electrically erasable read-only memory (Electric Erasable Programmable Read-Only Memory, EEPROM), etc.
[0070] The processor 102 may be an integrated circuit chip with signal processing capability. In the implementation process, each step of the data storage method in this embodiment can be completed by an integrated logic circuit of hardware in the processor 102 or an instruction in the form of software. The above-mentioned processor 102 may be a general-purpose processor, including a central processing unit (CPU), a microcontroller unit (MCU), a complex programmable logic device (CPLD), a field programmable gate array (FPGA), an embedded ARM and other chips.
[0071] For ease of explanation, only one processor is described in the electronic device 100. However, it should be noted that the electronic device 100 in the present invention may also include multiple processors, so the steps performed by one processor described in the present invention may also be performed jointly or individually by multiple processors. For example, if the processor of the electronic device 100 performs step A and step B, it should be understood that step A and step B may also be performed jointly by two different processors or individually in one processor. For example, the first processor performs step A, the second processor performs step B, or the first processor and the second processor perform steps A and B together.
[0072] Based on the above content, the data storage method provided in this embodiment is introduced, and the data storage method is applied to the erasure end. Figure 3 , Figure 3 The following is a flow chart of a data storage method provided by this embodiment, which includes the following steps:
[0073] S110, sending N data blocks to N data nodes for pre-storage respectively, one data block corresponds to one data node.
[0074] In this embodiment, the erasure correction end pre-stores N data blocks, where the N data blocks include n original data blocks and m check data blocks.
[0075] The n original data blocks are obtained by dividing the data to be stored sent by the client device by the erasure correction end, and the m verification data blocks are obtained by encoding the n original data blocks according to the preset erasure correction rule.
[0076] For example, the data block X to be stored is divided into two original data blocks C 1 and C 2 , and then perform erasure coding on the original data block to obtain a check data block C 3 = C 1+ C 2 , the erasure ratio n:m=2:1. Then C 1 , C 2 and C 3 The data are sent to data node 1, data node 2, and data node 3 for pre-storage respectively.
[0077] S120, when at least n first data nodes successfully pre-store data blocks, send a storage instruction to each first data node, the storage instruction is used to instruct the first data node to complete the storage of its own data block, wherein the first data node is the data node that successfully pre-stores the data block among the N data nodes.
[0078] In this embodiment, the own data block refers to the data block pre-stored by the first data node.
[0079] After receiving the data block and pre-storing it, each data node will feedback a pre-storage message to the erasure correction end. The pre-storage message includes whether the pre-storage is successful or failed. The erasure correction end determines the number of data nodes that successfully pre-stored the data block based on each pre-storage message received.
[0080] Because, if the data to be stored is stored successfully, when the user reads the data to be stored, the erasure end only needs to read n data blocks from n data nodes, and then obtain n original data blocks through the preset erasure decoding rules, and then obtain the data to be stored.
[0081] For example, the original data block is C 1 and C 2 , the check data block is C 3 = C 1 + C 2 , store the three data blocks in the data nodes respectively. When the user reads the data to be stored, the erasure end reads two data blocks from the three data nodes at random. 1 and C 3 , by solving the following system of equations:
[0082]
[0083] Get C 1 = C 1 , C 2 = C 3 - C 1 , and then the erasure end uses C 1 and C 2 Restore the data to be stored and feed it back to the user.
[0084] Therefore, the erasure end only needs to confirm that at least n data nodes have successfully stored the data block, and then perform storage processing on the data nodes that have successfully pre-stored the data block.
[0085] The storage instruction is used to instruct the first data node to complete the storage of its own data block, which means that the first data node changes its own data block from a pre-storage state to a storage state according to the storage instruction.
[0086] It can be understood that when the data block is in a pre-storage state, the first data node can store, roll back or delete the data block according to the instructions of the erasure end. When the data block is in a storage state, the first data node cannot delete the data block, but cannot roll back. When the user initiates a data read request on the client, the erasure end can read the data block in the storage state from the first data node.
[0087] S130, when each first data node completes storage of its own data block, confirm that the data storage is successful.
[0088] In this embodiment, when the erasure end confirms that each first data node has completed the storage of its own data block, that is, when the data blocks corresponding to each first data node are in the storage state, the data storage is confirmed to be successful, and the erasure end feeds back a data storage success message to the client to prompt the user that the data storage is successful.
[0089] Compared with the prior art, the data storage method provided in this embodiment sends a storage instruction to each first data node after determining that at least n first data nodes have successfully pre-stored, so that the first data node completes the storage of its own data block, and confirms the data success when each first data node completes the storage of its own data block. This ensures that the status of the data block on each data node is consistent.
[0090] Optionally, a communication connection is established between any two data nodes; each data block includes a state identifier and a version number, and the version number represents the sequence number of the current data storage operation.
[0091] The state identifier includes a stateless state or a storage state. The state identifier of a stateless state indicates that the data block is in a pre-storage state, and the state identifier of a storage state indicates that the data block is in a storage state.
[0092] The version number represents the sequence number of the data storage operation. For example, when storing data i to be stored, N data blocks are generated according to the data i to be stored, and the version number of each data block is 1. After the data i to be stored is successfully stored, the data j to be stored is stored, and the erasure end generates N data blocks according to the data j to be stored, and the version number of each data block is 2.
[0093] After the erasure end sends the storage instruction to each first data node, the first data node can complete the storage of its own data block in the following manner.
[0094] The first data node determines in real time within a preset time whether a storage instruction is received.
[0095] If so, the first data node changes the state flag of its own data block from stateless to storage state to complete the storage of its own data block, and feeds back a response message to the erasure end.
[0096] If not, the first data node obtains the first version number and the first status identifier of each first data block, and processes its own data block according to each first version number and each first status identifier to complete the storage of its own data block, wherein one first data block corresponds to a data block pre-stored by the first data node.
[0097] In this embodiment, the preset time is calibrated by technicians based on experiments.
[0098] Please refer to Figure 4 , the erasure end will be data block C i Taking the data block i sent to data node i for storage as an example, the data storage process is described. The process of the erasure end sending data block i to data node i for storage includes the following four steps: ① The erasure end sends data block C i ② Data node i feeds back a pre-storage response; ③ After receiving the storage response fed back by data node i, the erasure end sends a storage instruction to data node i; ④ After data node i completes data block C i After storage, a response message is fed back to the erasure end.
[0099] Ideally, after the erasure end sends a storage instruction to each first data node, each first data node can receive the storage instruction within a preset time, and each first data node will change the first state identifier of its own data block from stateless to storage state, so as to change the data block from the pre-storage state to the storage state, complete the storage of its own data block, and feed back a response message to the erasure end, which will receive the response message sent by each first data node.
[0100] However, in the process of the first data node receiving the storage instruction and feeding back the response message, the network may be abnormal, for example, disconnected or delayed, resulting in some first data nodes being unable to receive the storage instruction within the preset time. Figure 4 Process ③ in the above fails; or, the response message fed back by some first data nodes is lost, that is, Figure 4 Process ④ in the above fails. Both of these situations will result in the erasure end being unable to receive every response message.
[0101] In the case where process ③ of some first data nodes fails, in the traditional technology, the number of response messages received by the erasure end is less than the number of first data nodes, and the data storage is determined to have failed. In addition, the data blocks on the first data nodes have both pre-storage status and storage status, resulting in inconsistent status of the data blocks on the first data nodes.
[0102] Furthermore, since the data blocks on the first data node that has not received the response message are in a pre-stored state, they may remain on the first data node forever. When new data is stored next time, the remaining data blocks may affect data redundancy and even cause data storage errors.
[0103] Therefore, in view of the problems existing in the prior art, the data storage method provided in this embodiment obtains the first version number and the first state identifier of each first data block for the first data node that has not received the storage instruction within the preset time, and processes its own data block according to each first version number and each first state identifier to complete the storage of its own data block. Thus, the data block pre-stored by the first data node that has not received the storage instruction is changed from the pre-storage state to the storage state, ensuring the consistency of the data blocks on each first data node.
[0104] In addition, the erasure end does not need to determine that the data storage is successful only when it receives a response message sent by each first data node. The erasure end can determine that the data storage is successful only when it receives a response message sent by at least one first data node.
[0105] Therefore, the first data node may process its own data block according to each first version number and each first state identifier in the following manner to complete the storage of its own data block:
[0106] In the first step, the first data node determines a target version number of any first data block whose first state identifier is a storage state.
[0107] In the second step, if the version number of the target data block is the same as the target version number, the first data node changes the state identifier of the target data block from stateless to storage state to complete the storage of the target data block, wherein the target data block is the data block of the first data node itself.
[0108] In this embodiment, generally speaking, the probability that each first data node fails to receive the storage instruction due to network anomalies is extremely low. Therefore, when there is no failure at the erasure end and the storage instruction is sent normally, there is at least one first data block whose first state is identified as the storage state.
[0109] The first data node that has not received the storage instruction determines to change the state identifier of the target data block from the stateless state to the storage state according to the version number of the first data block in the first state identifier not in the storage state.
[0110] If the erasure end fails before sending storage instructions to each data node, each data node will not be able to receive the storage instruction. At this time, each data node obtains the status identifier of the data block of each data node, and deletes its own data block when it determines that each status identifier is stateless. This ensures that the data block will not be stored by mistake and will not remain on the data node. At the same time, after the erasure end returns to normal, it will not receive any response message and determine that the data storage has failed.
[0111] Optional, please refer to Figure 5 , the erasure end will be data block C j Taking sending to data node j for storage as an example, the process of rollback processing on the erasure end is explained.
[0112] After the erasure end sends N data blocks to N data nodes respectively, each data node pre-stores the data block and feeds back the pre-storage response to the erasure end. If the number of pre-storage responses received by the erasure end is less than n, it is determined that the data storage has failed, and a storage failure message is fed back to the client. At this time, the erasure end needs to send a rollback instruction to each data node so that each data node rolls back its own data block to the erasure end to avoid data remaining on the data node and affecting the next data storage.
[0113] Among them, if the number of pre-stored responses received by the erasure end is less than n, the following two situations may occur.
[0114] The first one is that the number of data nodes that successfully pre-store data blocks is less than n.
[0115] The second type is that the number of data nodes that successfully pre-store data blocks is greater than or equal to n, but due to network anomalies and other reasons, the pre-stored responses fed back by some data nodes are lost, that is, Figure 4 Process ② in fails, resulting in the number of pre-stored responses received by the erasure end being less than n.
[0116] For the second case, due to network anomalies and other reasons, some data nodes may not receive the rollback instruction, that is, Figure 5 Process ⑤ in the above fails, or the data receives a rollback instruction, and the data block rollback fails, resulting in a data block rollback failure. Figure 5 Process ⑥ in the above fails, so that the data block remains on the data node. At this time, depending on the number of data nodes that failed to roll back, the following situations may occur.
[0117] The first is that the number of data nodes that failed to roll back is greater than or equal to n, that is, there are greater than or equal to n data blocks remaining on the data nodes. At this time, the remaining data may be processed as correct data.
[0118] The second type is that the number of data nodes that failed to roll back is less than n and greater than or equal to m. When storing new data next time, the remaining data blocks may be treated as the data blocks generated during the data storage and stored incorrectly. As a result, after the data is successfully stored, there are less than n and greater than or equal to m data blocks that are erroneous data blocks. As a result, when the user reads the data, the erasure end cannot decode the data blocks on the data nodes to obtain the original data blocks, resulting in data reading failure.
[0119] The third type is that the number of data nodes that failed to roll back is less than m. When storing new data next time, the remaining data blocks may be treated as the data blocks generated during the data storage and stored incorrectly, thereby reducing the redundancy of the data.
[0120] Therefore, in order to solve the problem of data blocks failing to roll back and remaining on the data node, Figure 3 Based on Figure 6 After step S110, the method may further include the following step S111 which is parallel to step S120 and step S130.
[0121] S111, when p first data nodes successfully pre-store data blocks, a rollback instruction is sent to each data node, where the rollback instruction is used to instruct the data node to roll back its own data block to the erasure end, and to delete its own data block when rollback fails, wherein p is less than n.
[0122] In this embodiment, the data block rollback failure includes that the data node does not receive the rollback instruction, or the data node receives the rollback instruction, but the data block rollback fails due to network abnormalities and other reasons.
[0123] Optionally, a data node can delete its own data block when rollback fails by the following method.
[0124] In the first step, when a data node fails to roll back its own data block, the data node obtains a second state identifier of a data block corresponding to each data node.
[0125] In the second step, if each second state identifier is stateless, the data node deletes its own data block.
[0126] In this embodiment, the data node determines in real time within a preset time whether a rollback instruction is received, and if not, obtains the status identifier of the data block corresponding to each data node.
[0127] If the status flag of each data block is "no status", the data block itself will be deleted.
[0128] If the data node receives the rollback instruction within the preset time but the rollback fails, the status identifier of the data block corresponding to each data node is obtained.
[0129] If the status flag of each data block is "no status", the data block itself will be deleted.
[0130] Optional, in Figure 6 Based on Figure 7 , after step S120, the data storage method further includes the following steps S122-S124.
[0131] S122: Determine whether a response message returned by at least one first data node is received within a preset time.
[0132] In this embodiment, starting from the moment when the erasure end sends the storage instruction, it is determined within a preset time whether a response message returned by at least one first data node is received.
[0133] S123: If a response message returned by at least one first data node is received within a preset time, it is determined that each first data node has completed the storage of its own data block.
[0134] In this embodiment, if the erasure end receives a response message returned by at least one first data node within a preset time, it confirms that each first data node has completed the storage of its own data block and confirms that the data storage is successful.
[0135] S124: If no response message is received from any first data node within a preset time, it is confirmed that the data storage has failed.
[0136] In this embodiment, the erasure end does not receive a response message returned by any of the first data nodes, indicating that an exception occurs before the erasure end sends a storage instruction to each first data node. If each data node does not receive the storage instruction within a preset time, it will delete its own data block according to the status identifier of the data block corresponding to each other data node.
[0137] Please continue to refer to Figure 4 After the erasure end sends a storage instruction to each first data node, the erasure end may fail, resulting in the inability to receive a response message fed back by any first data node, and the inability to feed back a message of successful or failed data storage to the client, resulting in the user being unable to know whether the data storage is successful.
[0138] Therefore, in order to solve the problem that after the above-mentioned erasure end sends storage instructions to each first data node, the erasure end may fail, resulting in the erasure end being unable to feedback the message of data storage failure or data storage success to the client, the electronic device 100 is also deployed with at least one slave erasure end, and the slave erasure end and the erasure end have the same function.
[0139] Therefore, please continue to refer to Figure 7 , before step S122, the data storage method also includes the following step S121.
[0140] Step S121: If an exception occurs on the erasure end, any slave erasure end is used as an erasure end so that the erasure end receives each response message.
[0141] In this embodiment, if the erasure end has not recovered after the first data node feeds back the response message, the first data node will repeatedly send the response message until the erasure end receives the response message.
[0142] Compared with the prior art, this embodiment has the following beneficial effects:
[0143] First, the data storage method provided in this embodiment sends a storage instruction to each first data node after determining that at least n first data nodes have successfully pre-stored, so that the first data node completes the storage of its own data block, and confirms the data success when each first data node completes the storage of its own data block. This solves the problem in the prior art that due to network anomalies and other reasons, some first data nodes fail to receive the storage instruction, resulting in inconsistent states of data blocks on various data nodes, and the erasure end determines that data storage has failed.
[0144] Secondly, the data storage method provided in this embodiment adds a version number to each data block. When there is a data block with a status mark of stateless on the data node, the data node can store or delete the data block according to the loading mark and version number of the data block corresponding to other data nodes, so as to ensure the consistency of the status and version number of the data blocks between the various data nodes, and ensure the consistency of the data blocks stored by the data node with the upper-level data to be stored.
[0145] Third, the data storage method provided in this embodiment can solve the problem of data blocks remaining on data nodes due to network anomalies and other reasons, thereby preventing the residual data from affecting data redundancy.
[0146] Fourth, if a failure occurs on the erasure end and the storage message cannot be sent to each data node, each data node that has not received the storage instruction can delete its own data block through the version number and status identifier of the data block corresponding to other data nodes, without storing the data block incorrectly.
[0147] Finally, the data storage method provided in this embodiment ensures that the consistency of the data blocks of the data nodes has a very low overhead when various abnormal situations occur, and the storage of the version number of the data block takes up very little, only a few bytes.
[0148] Please refer to Figure 8 , Figure 8 A block diagram of a data storage device 200 provided in this embodiment is shown. The data storage device 200 is applied to an erasure correction end, the erasure correction end establishes communication connections with N data nodes respectively, and the erasure correction end pre-stores N data blocks, the N data blocks include n original data blocks, and n is less than N; the device includes: a pre-storage module 201, a rollback module 202, a storage module 203, a detection module 204 and a confirmation module 205.
[0149] The pre-storage module 201 is used to send N data blocks to N data nodes for pre-storage respectively, and one data block corresponds to one data node.
[0150] The storage module 203 is used to send a storage instruction to each first data node when at least n first data nodes successfully pre-store data blocks, and the storage instruction is used to instruct the first data node to complete the storage of its own data block, wherein the first data node is the data node that successfully pre-stores the data block among the N data nodes.
[0151] The confirmation module 205 is used to confirm that the data storage is successful when each first data node completes the storage of its own data block.
[0152] Optionally, the rollback module 202 is used to send a rollback instruction to each data node when p first data nodes successfully pre-store data blocks, and the rollback instruction is used to instruct the data node to roll back its own data block to the correction and erasure end, and to delete the data block when the data block rollback fails, where p is less than n.
[0153] Optionally, the detection module 204 is configured to:
[0154] Within the preset time, determining whether a response message returned by at least one first data node is received;
[0155] If yes, then determine that each first data node has completed storage of its own data block;
[0156] If not, it is confirmed that the data storage has failed.
[0157] Optionally, the erasure correction end is deployed on an electronic device, and the electronic device is also deployed with at least one slave erasure correction end.
[0158] The detection module 204 is further configured to, if an abnormality occurs at the erasure end, use any slave erasure end as an erasure end, so that the erasure end receives each response message.
[0159] Those skilled in the art can clearly understand that for the convenience and brevity of description, the specific working process of the data storage device 200 described above can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.
[0160] This embodiment further provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by the processor 102, the data storage method disclosed in the above embodiment is implemented.
[0161] In summary, an embodiment of the present invention provides a data storage method, device, electronic device and storage medium. The method includes: first, sending N data blocks to N data nodes for pre-storage respectively; then, when at least n first data nodes successfully pre-store the data blocks, sending a storage instruction to each first data node, the storage instruction is used to instruct the first data node to complete the storage of its own data block; and when each first data node completes the storage of its own data block, confirming that the data storage is successful. By first sending the data block to the data node for pre-storage, and then confirming that at least n data blocks have been pre-stored successfully, and then sending a storage instruction to the data node that successfully stored the data block, so that the data node stores the data block, it can solve the problem of misjudging the actual data storage success as data storage failure in the prior art, and improve the efficiency of data storage.
[0162] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed by the present invention should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.
Claims
1. A data storage method, characterized in that: Applied to an erasure correction end, the erasure correction end establishes communication connections with N data nodes respectively, the erasure correction end pre-stores N data blocks, the N data blocks include n original data blocks, and n is less than N; the method includes: Sending the N data blocks to the N data nodes for pre-storage respectively, one data block corresponds to one data node; When at least n first data nodes successfully pre-store the data block, a storage instruction is sent to each of the first data nodes, the storage instruction is used to instruct the first data node to complete the storage of its own data block, wherein the first data node is a data node that successfully pre-stores the data block among the N data nodes; When each of the first data nodes completes storage of its own data blocks, confirming that the data storage is successful; Wherein, a communication connection is established between any two of the data nodes; each of the data blocks includes a state identifier and a version number, and the version number represents the sequence number of the current data storage operation; the first data node completes the storage of its own data block, including: The first data node determines in real time within a preset time whether the storage instruction is received; if so, the first data node changes the state identifier of its own data block from stateless to storage state to complete the storage of its own data block, and feeds back a response message to the erasure end; if not, the first data node obtains the first version number and the first state identifier of each first data block, and processes its own data block according to each first version number and each first state identifier to complete the storage of its own data block, wherein one first data block corresponds to one data block pre-stored by the first data node; The first data node processes its own data block according to each of the first version numbers and each of the first state identifiers to complete storage of its own data block, including: The first data node determines a target version number of any one of the first data blocks whose first state identifier is a storage state; if the version number of the target data block is the same as the target version number, the first data node changes the state identifier of the target data block from stateless to storage state to complete the storage of the target data block, wherein the target data block is the data block of the first data node itself.
2. The method according to claim 1, characterized in that After the step of sending the N data blocks to the N data nodes for pre-storage respectively, the method further includes: When p of the first data nodes successfully pre-store data blocks, a rollback instruction is sent to each of the data nodes, and the rollback instruction is used to instruct the data node to roll back its own data block to the correction and erasure end, and to delete its own data block when rollback fails, where p is less than n.
3. The method according to claim 2, characterized in that A communication connection is established between any two of the data nodes; The data node deletes its own data block when the rollback fails, including: When the data node fails to roll back its own data block, obtaining a second state identifier of a data block corresponding to each data node; If each of the second state identifiers is in a stateless state, the data node deletes its own data block.
4. The method according to claim 1, characterized in that: After the step of sending a storage instruction to each of the first data nodes, the method further includes: Within a preset time, determining whether a response message returned by at least one of the first data nodes is received; If yes, then determining that each of the first data nodes has completed storage of its own data blocks; If not, it is confirmed that the data storage has failed.
5. The method according to claim 4, characterized in that The erasure correction end is deployed in an electronic device, and the electronic device is further deployed with at least one slave erasure correction end; Before the step of determining whether a response message returned by at least one first data node is received within the preset time, the method further includes: If an exception occurs on the erasure end, any one of the slave erasure ends is used as the erasure end, so that the erasure end receives each of the response messages.
6. A data storage device, characterized in that: Applied to an erasure correction end, the erasure correction end establishes communication connections with N data nodes respectively, the erasure correction end stores N data blocks in advance, the N data blocks include n original data blocks, and n is less than N; the device includes: A pre-storage module, used to send the N data blocks to the N data nodes for pre-storage respectively, one data block corresponding to one data node; A storage module, configured to send a storage instruction to each of the first data nodes when at least n first data nodes successfully pre-store the data block, wherein the storage instruction is used to instruct the first data node to complete storage of its own data block, wherein the first data node is a data node that successfully pre-stores the data block among the N data nodes; A confirmation module, used for confirming that data storage is successful when each of the first data nodes completes storage of its own data blocks; Wherein, a communication connection is established between any two of the data nodes; each of the data blocks includes a state identifier and a version number, and the version number represents the sequence number of the current data storage operation; the first data node completes the storage of its own data block, including: The first data node determines in real time within a preset time whether the storage instruction is received; if so, the first data node changes the state identifier of its own data block from stateless to storage state to complete the storage of its own data block, and feeds back a response message to the erasure end; if not, the first data node obtains the first version number and the first state identifier of each first data block, and processes its own data block according to each first version number and each first state identifier to complete the storage of its own data block, wherein one first data block corresponds to one data block pre-stored by the first data node; The first data node processes its own data block according to each of the first version numbers and each of the first state identifiers to complete storage of its own data block, including: The first data node determines a target version number of any one of the first data blocks whose first state identifier is a storage state; if the version number of the target data block is the same as the target version number, the first data node changes the state identifier of the target data block from stateless to storage state to complete the storage of the target data block, wherein the target data block is the data block of the first data node itself.
7. An electronic device, characterized in that: The electronic device comprises: one or more processors; A memory for storing one or more programs, which, when executed by the one or more processors, enables the one or more processors to implement the data storage method according to any one of claims 1 to 5.
8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the data storage method according to any one of claims 1 to 5 is implemented.
