Data processing method and device, server and storage medium

Abstract

This application provides a data processing method, apparatus, server, and storage medium. When storing target data, the server splits the target data into k+m sub-data according to a preset erasure coding redundancy ratio. During the storage process, as long as N sub-data greater than or equal to k are successfully written, the server can provide feedback that the data has been successfully written, so that the user can perform subsequent operations on the target data, thereby improving the availability of the target data. Since N is greater than or equal to k, even if the remaining W-N sub-data have not been completely written, the data can be recovered from the N sub-data that have been successfully stored, thus ensuring the reliability of the target data.

Description

Technical Field

[0001] This application relates to the field of distributed storage technology, and in particular to a data processing method, apparatus, server and storage medium. Background Technology

[0002] In existing technologies, distributed storage can achieve the storage of massive amounts of data. The storage of massive amounts of data maintains high reliability through redundant storage technology.

[0003] When using erasure coding for redundant storage, the server uses erasure coding to generate k+r copies of data and stores them in k+r corresponding storage nodes. Here, k is the number of original data slices and r is the number of verification slices. When reading the stored data, it is only necessary to obtain any at least k slices of data from the above k+r storage nodes, and the original data can be recovered by calculating using the erasure coding algorithm.

[0004] In the storage process of each slice of data, a strong consistency write strategy is generally adopted to ensure data reliability. That is, the data write is considered successful only after all k+r slices of data have been written to their corresponding storage nodes. However, due to the inconsistent processing efficiency of storage nodes in handling data storage tasks, there is a tendency for write delays to occur on some storage nodes. This results in a time delay in the server's confirmation that the original data has been written successfully, leading to poor availability of the original data. Summary of the Invention

[0005] This application provides a data processing method, apparatus, server, and storage medium to solve the above-mentioned problems.

[0006] In a first aspect, this application provides a data processing method, the method being applied to a server, the server being communicatively connected to a terminal device, the server and other servers jointly deploying a distributed system, the method comprising:

[0007] Obtain the target data sent by the terminal device;

[0008] When writing the target data into the target logical object, the target data is fragmented according to a preset erasure coding redundancy ratio to obtain W sub-data; the W sub-data includes k original sub-data and m check sub-data, and the data length of each sub-data is the same;

[0009] Each of the sub-data items is written into multiple storage files corresponding to the target logical object; the number of storage files is the same as the number of sub-data items.

[0010] When at least N storage files are successfully written, a write success message is sent back to the terminal device; the N storage files correspond to the storage of N sub-data, the N sub-data including k original sub-data and r check sub-data, m≥r≥0.

[0011] In the above technical solution, when storing target data, the server splits the target data into k+m sub-data according to a preset erasure coding redundancy ratio. During the storage process, as long as N sub-data greater than or equal to k are successfully written, the server can provide feedback that the data has been successfully written, so that the user can perform subsequent operations on the target data, thereby improving the availability of the target data. Since N is greater than or equal to k, even if the remaining WN sub-data have not been completely written, the data can be recovered through the N sub-data that have been successfully stored, thus ensuring the reliability of the target data.

[0012] Optionally, the distributed system includes a client, and the target logical object records information about the data storage status.

[0013] Writing each of the sub-data into multiple storage files corresponding to the target logical object specifically includes:

[0014] The client is controlled to obtain the first information of the record data storage status;

[0015] Based on the first information and the W sub-data, generate W data write requests and transmit each data write request to the server storing the storage file;

[0016] The data write request is used to enable the server storing the storage file to obtain second information about the data storage status from the local machine. When the data write request is at the head of the current request queue of the storage file, the first information is obtained from the data write request. When the first information and the second information are consistent, the sub-data in the data write request is stored into the storage file, and a write success message is sent back to the client after storage is completed.

[0017] When the first information and the second information are inconsistent, the data write request is sorted to the end of the current request queue; the current request queue includes at least one data write request, and the first information in each data write request is different.

[0018] In the above technical solution, when the controller stores multiple data to the target logical object, it slices the data sequentially according to the order in which the data is acquired. Based on the first information recording the storage state of each data before it is stored in the target logical object, it generates multiple data write requests and transmits these write requests to the corresponding servers. After receiving at least one data write request, the server adds each data write request to a request queue and responds according to the order of the request queue. During the response process, the server retrieves the second information recording the storage state locally and compares it with the first information in the currently responded data write request. If they match, the order of the data write request in the request queue is accurate, and the sub-data in the request can be stored in the local storage file. The second information is then updated, and the corresponding write success information is fed back. If the first and second information do not match, it is determined that there is other data that has not been written before the sub-data of the data write request is written. The server then adjusts the position of the data write request in the request queue so that the server can find the data write request containing the sub-data to be written and write the data correctly, thus ensuring the consistency of the order in which each data slice is stored in each server.

[0019] Optionally, when at least N storage files are successfully written, a write success message is sent back to the terminal device, specifically including:

[0020] When at least N successful write messages for storage files are received, a mapping relationship is constructed between the identification information of the target logical object, the identification information of the target data, the write offset of the target data, and the write length of the target data, and the mapping relationship is stored in the metadata of the target logical object.

[0021] Update the data storage status information stored in the target logical object according to the write offset of the target data and the write length of the target data;

[0022] Generate and send a write success message to the terminal device.

[0023] Optionally, the information on the data storage status includes the offset of data writing in the storage space allocated to the target logical object;

[0024] Based on the write offset and write length of the target data, update the data storage status information stored in the target logical object, specifically including:

[0025] The sum of the write offset of the target data and the write length of the target data is determined as the updated data storage status information of the target logical object after the target data is written.

[0026] Optionally, after sending a successful write message to the terminal device, the method further includes:

[0027] Obtain the offset of the target logical object;

[0028] When the offset is greater than or equal to a preset encapsulation threshold, the corresponding data write offset set is obtained from each storage file. The data write offset set includes the offset of the data storage start position after each sub-data write.

[0029] Based on each set of data write offsets, the offset corresponding to the last write data that satisfies the storage file write success condition is determined as the data encapsulation length; the storage file write success condition is that at least N storage files are successfully written in the storage files corresponding to the target logical object.

[0030] The data encapsulation length is stored in the metadata of the target logical object, and the target logical object is encapsulated.

[0031] Optionally, the method further includes:

[0032] For each of the aforementioned storage files, obtain the maximum offset in the corresponding data write offset set;

[0033] When the maximum offset is greater than the data encapsulation length, a first data recovery instruction is generated and sent to the server storing the storage file; the first data recovery instruction is used to cause the server storing the storage file to truncate the data length in the storage file to the data encapsulation length;

[0034] When the maximum offset is less than the data encapsulation length, a second data recovery instruction is generated and sent to the server storing the storage file; the second data recovery instruction is used to enable the server storing the storage file to obtain the data after the maximum offset from other storage, and to perform data recovery based on the data after each maximum offset.

[0035] In the above technical solution, after data encapsulation, it can be determined whether the stored data needs to be corrected based on the maximum offset of the data in each storage file and the data encapsulation length. During the correction process, when the maximum offset is greater than the data encapsulation length, it is determined that some sub-files in the storage file have failed to be written to at least WN other storage files. Since the user cannot use these sub-files to recover the original data, the data length in the storage file needs to be truncated. Data within the data encapsulation length is determined to be validly stored data to prevent the server from repeatedly storing the sub-data when rewriting failed data, thereby improving the utilization of storage space. When the maximum offset is less than the data encapsulation length, it is determined that there is a data write failure when storing data in the storage file. Data recovery can be performed using data at the corresponding position in other storage files to fill in the missing data in the current storage file, ensuring strong data reliability.

[0036] Optionally, the method further includes:

[0037] Obtain the identification information of the data to be read from the terminal device;

[0038] The identifier information of the data to be read is queried in the mapping relationship of each metadata storage to obtain the identifier information of the corresponding target logical object, the data write offset and the data write length;

[0039] Obtain sub-data starting from the data write offset and with a data length equal to the data write length from at least R storage files corresponding to the identification information of the target logical object;

[0040] The data to be read is obtained by processing each of the sub-data using erasure coding algorithms;

[0041] The R storage files each store R sub-data items, and the R sub-data items include k+mr sub-data items.

[0042] Optionally, the method further includes:

[0043] Obtain the identification information of the data to be deleted from the terminal device;

[0044] The identifier information of the data to be deleted is queried in the mapping relationship of each of the metadata storages to obtain the identifier information, data write offset and data write length of the corresponding target logical object;

[0045] Write the data corresponding to the data to be deleted to the offset and delete it, or write the data corresponding to the data to be deleted to the length and set it to 0.

[0046] Secondly, this application provides a data processing apparatus, comprising:

[0047] The acquisition module is used to acquire the target data sent by the terminal device;

[0048] The processing module is used to divide the target data into data fragments according to a preset erasure coding redundancy ratio to obtain W sub-data; the W sub-data includes k original sub-data and m check sub-data, and the data length of each sub-data is the same;

[0049] The processing module is further configured to write each of the sub-data into multiple storage files corresponding to the target logical object; the number of storage files is the same as the number of sub-data.

[0050] The processing module is also used to send a write success message to the terminal device when at least N storage files are successfully written; the N storage files correspond to storing N sub-data, the N sub-data including k original sub-data and r check sub-data, m≥r≥0.

[0051] Thirdly, this application provides a server, including: a processor and a memory communicatively connected to the processor;

[0052] The memory stores computer instructions;

[0053] The processor is used to implement the methods involved in the first aspect when executing computer instructions.

[0054] Fourthly, this application provides a computer-readable storage medium storing computer instructions, which, when executed by a processor, are used to implement the method involved in the first aspect.

[0055] This application provides a data processing method, apparatus, server, and storage medium. When storing target data, the server splits the target data into k+m sub-data according to a preset erasure coding redundancy ratio. During the storage process, as long as N sub-data greater than or equal to k are successfully written, the server can provide feedback that the data has been successfully written, so that the user can perform subsequent operations on the target data, thereby improving the availability of the target data. Since N is greater than or equal to k, even if the remaining WN sub-data have not been completely written, the data can be recovered from the N sub-data that have been successfully stored, thus ensuring the reliability of the target data. Attached Figure Description

[0056] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0057] Figure 1 This is a scene diagram of data distribution storage provided in accordance with an exemplary embodiment of this application;

[0058] Figure 2This is a schematic diagram of a distributed storage system storing data according to an exemplary embodiment of this application;

[0059] Figure 3 This is a schematic diagram of a data processing method provided in this application according to an exemplary embodiment;

[0060] Figure 4 This is a schematic diagram illustrating the distributed writing of target data according to an exemplary embodiment of this application;

[0061] Figure 5 This is an interaction diagram of a data processing method provided in this application according to an exemplary embodiment;

[0062] Figure 6 This is an interaction diagram of a data recovery method provided in this application according to an exemplary embodiment;

[0063] Figure 7 This is a schematic diagram illustrating data recovery according to an exemplary embodiment of this application;

[0064] Figure 8 This is a schematic diagram of the structure of a data processing apparatus provided in accordance with an exemplary embodiment of this application;

[0065] Figure 9 This is a schematic diagram of the structure of a server provided according to an embodiment of this application.

[0066] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concepts of this application to those skilled in the art through reference to particular embodiments. Detailed Implementation

[0067] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0068] Figure 1 A scenario diagram for distributed data storage, such as Figure 1 As shown, the system includes a terminal device 10 and a server cluster 20, which are connected communicatively. The server cluster 20 includes multiple servers 201, which act as storage nodes and are connected in pairs to form a distributed storage system capable of distributively storing data.

[0069] Figure 2 A diagram illustrating distributed data storage in a distributed storage system. (Example) Figure 2 As shown, server 20 includes client 301 and management node 302. Server 20 provides KV storage 303 service and can communicate with other storage nodes 304 through client 301. When the distributed storage system stores data, terminal device 10 calls client 301 to transmit data to server 201. After server 201 requests a target logical object extent 305 from management node 302, it can write the data into the storage node 304 corresponding to the target logical object 305.

[0070] During data writing, server 201 slices the data and stores the sliced ​​sub-data in different storage nodes 304. During data storage, redundant storage technology is used to maintain high reliability, ensuring that even if some storage nodes become unavailable, the complete data can still be retrieved from other storage nodes. Redundant storage technologies include multi-replica storage and erasure coding.

[0071] When using erasure coding technology for redundant storage, server 201 uses erasure coding technology to generate k+r copies of data and store them in the corresponding k+r storage nodes. Here, k is the number of original data slices and r is the number of verification slices. When reading the stored data, it is only necessary to obtain any at least k slices of data from the above k+r storage nodes, and the original data can be recovered by calculating through the erasure coding algorithm.

[0072] During the storage of data in each slice, a strong consistency write strategy is generally adopted to ensure data reliability. That is, only after all k+r slice data has been written to their corresponding storage nodes will a successful write message be sent to the terminal device 10, so that the terminal device 10 can perform operations on the stored data, such as querying and deleting. However, due to the inconsistent processing efficiency of storage nodes in handling data storage tasks, write delays can easily occur on some storage nodes. This leads to a time delay in the server's confirmation that the original data has been written successfully, resulting in poor availability of the original data.

[0073] To address the aforementioned problems, this application provides a data processing method, apparatus, server, and storage medium. The technical concept of this application is to improve data availability by reducing data reliability during the data storage process, while ensuring that data can be recovered using sub-data stored in multiple storage files.

[0074] Figure 3 This is a flowchart illustrating a data processing method provided according to an exemplary embodiment of this application. Figure 3 As shown, the method includes:

[0075] S101, The server obtains the target data sent by the terminal device.

[0076] The target data is the data to be stored in the distributed system.

[0077] In one embodiment, the distributed system includes a client, and the terminal device sends target data to the server by calling the client in the server.

[0078] S102. When the server writes the target data into the target logical object, it performs data fragmentation on the target data according to the preset erasure code redundancy ratio to obtain W sub-data.

[0079] The target logical object is an object created by the management node in the server. When the storage space allocated to the target logical object is not full and the terminal device writes data again, the target logical object can be appended.

[0080] When the preset erasure coding redundancy ratio is k:m, the server will generate W sub-data from the target data according to the preset erasure coding redundancy ratio. The W sub-data include k original sub-data and m check sub-data, and the data length of each sub-data is the same.

[0081] S103. The server writes each sub-data into multiple storage files corresponding to the target logical object.

[0082] Each storage file is stored on a storage node, meaning that each storage file on other servers corresponds to the target logical object.

[0083] The server can obtain the correspondence between the target logical object and each storage file. After generating W sub-data using the obtained target data, the server writes the W sub-data into the corresponding W storage files according to the correspondence between the target logical object and each storage file.

[0084] After the storage file pairs have finished storing the sub-data, a success message is sent back to the client on the server.

[0085] S104. When at least N storage files are successfully written, the server sends a write success message to the terminal device.

[0086] Because the efficiency of storing sub-data is not exactly the same for each storage node, the time when the storage information is received from each storage node is also not exactly the same.

[0087] When the client obtains storage information for at least N storage files, it can be determined that the target data has been successfully stored in the distributed system, and the server sends a success message to the terminal device.

[0088] The N sub-data items include k original sub-data items and r verification sub-data items, where m ≥ r ≥ 0.

[0089] Since N is greater than or equal to k, when the terminal device obtains the original data, the client can use these N sub-data and erasure coding technology to recover the target data and provide the target data to the terminal device.

[0090] After the server sends a successful write message to the terminal device via the client, the storage nodes that failed to write will continue to complete the data writing to the stored file to ensure data reliability.

[0091] In the above technical solution, when storing target data, the server splits the target data into k+m sub-data according to a preset erasure coding redundancy ratio. During the storage process, as long as N sub-data greater than or equal to k are successfully written, the server can provide feedback that the data has been successfully written, so that the user can perform subsequent operations on the target data, thereby improving the availability of the target data. Since N is greater than or equal to k, even if the remaining WN sub-data have not been completely written, the data can be recovered through the N sub-data that have been successfully stored, thus ensuring the reliability of the target data.

[0092] Figure 4 This is a schematic diagram illustrating the distributed writing of target data according to an exemplary embodiment of this application, such as... Figure 4 As shown, when client 301 in server 201 is invoked by terminal device 10, it obtains target data ③ and writes target data ③ into target logical object 305. Since other data has been written into target logical object 305 before target data ③ is written, appending data needs to be performed on target logical object 305 according to the writing order.

[0093] When writing to the target logical object 305, server 201 performs data slicing with a redundancy ratio of 2:1, obtaining three sub-data ③', including two original sub-data and one verification sub-data. Server 201 transmits the above three sub-data ③' to three storage nodes 304 respectively, and stores them in storage files 3041 within the storage nodes 304. The order in which the sub-data are stored in each storage file 3041 is the same as the order in which the client 301 stores the data in the target logical object 305.

[0094] After storage is completed, each storage node 304 sends a successful write message to the client 301. After confirming that at least two data entries have been successfully written, the client 301 can store the mapping relationship between the target data, the target logical object, and related storage information in the KV storage 303, and send a successful write message to the terminal device 10 so that the terminal device 10 can perform subsequent operations on the stored data.

[0095] Storage nodes 304 that have not yet completed the write process will continue to write the sub-data ③' they have obtained after the client 301 sends a write success message to the terminal device 10, thus ensuring the reliability of the data in each storage node 304.

[0096] The process of writing target data to each storage node is explained in detail below.

[0097] Figure 5 This is an interaction diagram of a data processing method provided according to another exemplary embodiment of this application. (See diagram below.) Figure 5 As shown, it includes:

[0098] S201, The first server controls the client to obtain the first information on the data storage status of the record.

[0099] In this embodiment, the first server is the server that obtains the target data, and the second server is the server that contains storage files and stores the sub-data after the target data has been fragmented.

[0100] The first piece of information that records the data storage status includes the data write offset in the target logical object.

[0101] The data write offset in the target logical object represents the current usage length of the target logical object.

[0102] If the target logical object is a newly created object of the management node and has not yet been written with data, the data write offset is 0; if the target logical object has already been written with data, the data write offset is the data write offset updated according to the length of the written data after the previous data was written.

[0103] For example: If the target logical object has been written with data once, and the write length of that data is 10, then the current data write offset is 10.

[0104] In one embodiment, the first information is stored locally on the first server; in another embodiment, the first information is stored on another server that serves as a key-value store. No specific limitation is made here.

[0105] S202. The first server generates W data write requests based on the first information and W sub-data.

[0106] S203, The first server sends a data write request to the second server.

[0107] S204. The second server obtains the second information on the storage status of the record data from the local machine.

[0108] The second server has a storage file that corresponds to the target logical object, and the order in which the data is stored is the same as the order in which the first server writes data to the target logical object through the client.

[0109] The second piece of information includes the data write offset in the storage file, which represents the current length of the storage space currently used by the storage file.

[0110] If the second server can successfully store data every time the first server transmits data to it, then the first information and the second information are consistent.

[0111] S205. The second server obtains a data write request from the head of the current request queue located in the storage file, and obtains the first information from the data write request.

[0112] When the second server receives a data write request, it adds the request to the request queue and processes it according to the first-in, first-out principle. That is, the request at the head of the request queue is the earliest request received, and the second server will respond to that request first.

[0113] In some cases, when the second server responds to requests slowly and the terminal device continuously transmits multiple data for storage, the second server's request queue contains multiple requests that it has received consecutively.

[0114] To ensure the accuracy of the data write order, when responding to each data write request in the queue, the first information needs to be obtained from the data write request and compared with the second information obtained locally. This comparison determines whether the data write offset of the current data write request is consistent with the data write offset of the storage file for this write, thus verifying whether the current data write request order is consistent.

[0115] If the first information and the second information are consistent, proceed to step S206; if the first information and the second information are inconsistent, proceed to step S208.

[0116] S206. When the first information and the second information are consistent, the second server will write the sub-data in the request into the storage file.

[0117] When the first and second information are consistent, it means that the data to be written to the storage file in the second server is the data in the current data write request, and the sub-data in the data write request is stored in the storage file.

[0118] S207. After storage is completed, the second server sends a successful write message to the first server.

[0119] S208. When the first information and the second information are inconsistent, the second server will sort the data write request to the end of the current request queue of the storage file.

[0120] When the first and second information are inconsistent, it means that the order in which multiple data write requests entered the request queue of the second server is inconsistent with the order in which they were transmitted by the first server. The data write request at the head of the current request queue cannot be responded to. It is necessary to sort the requests in the request queue to find the data write request that matches the first and second information.

[0121] In one embodiment, when the first and second information of a data write request located at the head of the request queue are inconsistent, the request is moved to the tail of the request queue. The request in the original second position in the request queue is then moved to the head of the request queue, and step S205 is entered so that the second server can verify the first and second information in the request until a request with the correct writing order is found.

[0122] S209. When the first server obtains write success information for at least N storage files, construct a mapping relationship between the identification information of the target logical object, the identification information of the target data, the write offset of the target data, and the write length of the target data, and store the mapping relationship in the metadata of the target logical object.

[0123] The first server can obtain the identification information of the target data when the terminal transmits the target data to it.

[0124] The write offset of the target data is obtained from the metadata of the target logical object. If the target logical object is a newly created object of the management node, the write offset of the target data is 0; if the target logical object has already been written with data, the write offset of the target data is the sum of the lengths of the data already written to the target logical object.

[0125] The write length of the target data is determined based on the data length of the target data and the number of data fragments.

[0126] S210. The first server updates the data storage status information stored in the target logical object according to the write offset of the target data and the write length of the target data.

[0127] The first server determines the sum of the write offset of the target data and the write length of the target data as the data storage status information of the updated target logical object after the target data is written.

[0128] S211. The first server generates and sends a write success message to the terminal device.

[0129] In the above technical solution, when the controller stores multiple data to the target logical object, it slices the data sequentially according to the order in which the data is acquired. Based on the first information recording the storage state of each data before it is stored in the target logical object, it generates multiple data write requests and transmits these write requests to the corresponding servers. After receiving at least one data write request, the server adds each data write request to a request queue and responds according to the order of the request queue. During the response process, the server retrieves the second information recording the storage state locally and compares it with the first information in the currently responded data write request. If they match, the order of the data write request in the request queue is accurate, and the sub-data in the request can be stored in the local storage file. The second information is then updated, and the corresponding write success information is fed back. If the first and second information do not match, it is determined that there is other data that has not been written before the sub-data of the data write request is written. The server then adjusts the position of the data write request in the request queue so that the server can find the data write request containing the sub-data to be written and write the data correctly, thus ensuring the consistency of the order in which each data slice is stored in each server.

[0130] Once the storage space allocated to the target logical object is full, the target logical object needs to be encapsulated, and after encapsulation, each storage file needs to be checked and the corresponding data restored.

[0131] The encapsulation process of the target logical object is explained below.

[0132] Figure 6 This is an interaction diagram of a data recovery method provided in this application according to an exemplary embodiment, such as... Figure 6 As shown, the method includes:

[0133] S401, The first server obtains the offset of the target logical object.

[0134] The offset of the target logical object is the offset updated in its corresponding metadata after the last data storage of the target logical object is completed. The method for updating the offset of the target logical object has already been explained in step S210 and will not be repeated here.

[0135] S402. When the offset is greater than or equal to the preset encapsulation threshold, the first server retrieves the corresponding data from the storage files of each second server and writes it into the offset set.

[0136] When the offset is greater than or equal to the preset encapsulation threshold, the first server encapsulates the target logical object.

[0137] The first server calls the encapsulated interface of the management node so that the first server can obtain the corresponding data from each of the second servers and write it to the offset set.

[0138] The data write offset set includes the offset of the data storage starting position after each sub-data write.

[0139] The data write offset is generated by each second server based on the write result of each sub-data. When the data write is successful, the second server storing each storage file will store the updated offset after the data write in its local data write offset set, and use the latest stored offset as the second information. When the data write fails, the second server cannot store the offset corresponding to the data in the data write offset set, and the second information cannot be updated. Therefore, the second server cannot store subsequently obtained sub-data.

[0140] S403. The first server determines the data encapsulation length based on the set of data write offsets and the offset corresponding to the last write data that meets the conditions for successful write of the storage file.

[0141] The condition for successful writing of storage files is that at least N storage files corresponding to the target logical object are successfully written.

[0142] In other words, the first server counts the offsets of each storage file after each sub-data write, determines the number of successful data writes each time, and obtains the data offsets updated after at least N storage files successfully stored sub-files last time, which is used as the data encapsulation length.

[0143] The server can recover complete data from data in each storage file that is less than or equal to the data encapsulation length.

[0144] like Figure 7 As shown, the maximum length of data stored in 5 storage files is specified. The data length of the first and second storage files is cl2, the data length of the third storage file is cl1, the data length of the fourth storage file is cl3, and the data length of the fifth storage file is 0.

[0145] If the condition for successful writing of a storage file is that at least three storage files are successfully written, then the first, second, and fourth storage files are determined to have been successfully written. The minimum data length of the three storage files is the data encapsulation length, and the data length is cl2.

[0146] S404. The first server stores the data encapsulation length into the metadata of the target logical object and encapsulates the target logical object.

[0147] S405. For each storage file, the first server obtains the maximum offset in the corresponding data write offset set.

[0148] In other words, the first server obtains the maximum storage data length of each storage file.

[0149] After obtaining the maximum offset of each storage file, the first server compares it with the data encapsulation length.

[0150] When the maximum offset is greater than the data encapsulation length, for example Figure 7 The maximum offset of the fourth storage file is determined, and step S406 is used for data recovery.

[0151] When the maximum offset is less than the data encapsulation length, for example Figure 7 The maximum offset of the third and fifth storage files is determined, and the process proceeds to step S409 for data recovery.

[0152] When the maximum offset equals the data encapsulation length, for example Figure 7 The maximum offset of the first and second storage files indicates that data storage does not require data recovery.

[0153] S406. When the maximum offset is greater than the data encapsulation length, the first server generates the first data recovery instruction.

[0154] If the maximum offset of the storage file is greater than the data encapsulation length, it is determined that the storage file has stored other data after the data encapsulation length. However, if the number of times this data is successfully stored in conjunction with other storage files is less than N, it means that the data has not been successfully stored and the extra data stored in the storage file needs to be truncated.

[0155] Based on this, the first server generates the first data recovery instruction.

[0156] S407, The first server sends the first data recovery command to the second server.

[0157] S408. In response to the first data recovery command, the second server truncates the data length in the storage file to the data encapsulation length.

[0158] exist Figure 7 In the embodiment shown, data in the fourth storage file that is greater than cl2 and less than cl3 ​​is truncated, and only data that is less than or equal to cl2 is determined to be valid data.

[0159] For truncated data, the storage file corresponding to the new target logical object can be rewritten after the first server requests the new target logical object.

[0160] S409. When the maximum offset is less than the data encapsulation length, the first server generates a second data recovery instruction.

[0161] If the maximum offset is less than the data encapsulation length, it is determined that the storage file failed to store at least one sub-data, but at least N other storage files were successfully stored. The failed data needs to be recovered to ensure the data reliability of each storage file.

[0162] Based on this, the first server generates a second data recovery command.

[0163] S410, The first server sends a second data recovery command to the second server.

[0164] S411. In response to the second data recovery instruction, the second server retrieves the data after the maximum offset from other storage and performs data recovery based on the data after each maximum offset.

[0165] exist Figure 7 In the illustrated embodiment, it is necessary to use data greater than cl1 and less than or equal to cl2 from the first, second, and fourth storage files to recover the missing data in the third storage file, so that the data length of the third storage file is also the encapsulation length.

[0166] It is also necessary to recover all the data in the fifth storage file from the data in the first, second and fourth storage files that are less than or equal to cl2, and then perform a full write to the fifth storage file.

[0167] In the above technical solution, after data encapsulation, it can be determined whether the stored data needs to be corrected based on the maximum offset of the data in each storage file and the data encapsulation length. During the correction process, when the maximum offset is greater than the data encapsulation length, it is determined that some sub-files in the storage file have failed to be written to at least WN other storage files. Since the user cannot use these sub-files to recover the original data, the data length in the storage file needs to be truncated. Data within the data encapsulation length is determined to be validly stored data to prevent the server from repeatedly storing the sub-data when rewriting failed data, thereby improving the utilization of storage space. When the maximum offset is less than the data encapsulation length, it is determined that there is a data write failure when storing data in the storage file. Data recovery can be performed using data at the corresponding position in other storage files to fill in the missing data in the current storage file, ensuring strong data reliability.

[0168] After the data is stored, the terminal device can perform operations on the data, such as reading and deleting.

[0169] In one embodiment, when the terminal device reads the stored data, the first server obtains the identification information of the data to be read from the terminal device, queries the mapping relationship of each metadata storage for the identification information of the data to be read, and obtains the identification information of the corresponding target logical object, the data write offset, and the data write length.

[0170] Considering the possibility of write failures during the data writing process, sub-data, starting from the data write offset and with a length equal to the data write length, is obtained from at least R storage files corresponding to the identifier information of the target logical object. Each sub-data is then processed using an erasure coding algorithm to obtain the data to be read. Specifically, the R storage files store R sub-data, which include k+mr sub-data.

[0171] In another embodiment, when the terminal device deletes stored data, the first server obtains the identification information of the data to be deleted from the terminal device, queries the mapping relationship of each metadata storage for the identification information of the data to be deleted, obtains the identification information of the corresponding target logical object, the data write offset and the data write length, deletes the data write offset corresponding to the data to be deleted, or sets the data write length corresponding to the data to be deleted to 0.

[0172] During the subsequent data processing, each secondary server will retrieve data with a non-zero write length based on the mapping relationship associated with the target logical object and re-store it. Therefore, the previously deleted data will no longer be retained during the re-storage process.

[0173] Figure 8 This is a schematic diagram of the structure of a data processing apparatus 500 according to an embodiment of the present application. The data processing apparatus 500 includes an acquisition module 501 and a processing module 502, wherein...

[0174] The acquisition module 501 is used to acquire the target data sent by the terminal device;

[0175] Processing module 502 is used to slice the target data according to a preset erasure code redundancy ratio when writing the target data into the target logical object, and obtain W sub-data; the W sub-data includes k original sub-data and m check sub-data, and each sub-data has the same data length;

[0176] Processing module 502 is also used to write each sub-data into multiple storage files corresponding to the target logical object; the number of storage files is the same as the number of sub-data.

[0177] The processing module 502 is also used to send a write success message to the terminal device when at least N storage files are successfully written; the N storage files correspond to the storage of N sub-data, and the N sub-data include k original sub-data and r check sub-data, where m≥r≥0.

[0178] In one feasible embodiment, the processing module 502 is specifically used for:

[0179] Each piece of data is written into multiple storage files corresponding to the target logical object, specifically including:

[0180] Control the client to obtain the first information about the record data storage status;

[0181] Based on the first information and W sub-data, generate W data write requests and transmit each data write request to the server storing the storage file;

[0182] The data write request is used to enable the server storing the storage file to obtain the second information of the record data storage status from the local machine. When the data write request is at the head of the current request queue of the storage file, the first information is obtained from the data write request. When the first information and the second information are consistent, the sub-data in the data write request is stored into the storage file, and the write success information is sent back to the client after the storage is completed.

[0183] When the first information and the second information are inconsistent, the data write request is sorted to the end of the current request queue; the current request queue includes at least one data write request, and the first information in each data write request is different.

[0184] In one feasible embodiment, the processing module 502 is specifically used for:

[0185] When at least N storage files have been successfully written, a mapping relationship is constructed between the identification information of the target logical object, the identification information of the target data, the write offset of the target data, and the write length of the target data, and the mapping relationship is stored in the metadata of the target logical object.

[0186] Update the data storage status information stored in the target logical object based on the write offset and write length of the target data;

[0187] Generate and send a write success message to the terminal device.

[0188] In one feasible embodiment, the processing module 502 is specifically used for:

[0189] The sum of the write offset of the target data and the write length of the target data is determined as the data storage status information of the target logical object after the target data is written.

[0190] In one feasible embodiment, the processing module 502 is further configured to:

[0191] Get the offset of the target logical object;

[0192] When the offset is greater than or equal to the preset encapsulation threshold, the corresponding data is obtained from each storage file and written to the offset set. The data writing offset set includes the offset of the data storage start position after each sub-data write.

[0193] Based on the set of data write offsets, the offset corresponding to the last data that satisfies the storage file write success condition is determined as the data encapsulation length; the storage file write success condition is that at least N storage files are successfully written in the storage file corresponding to the target logical object.

[0194] The data encapsulation length is stored in the metadata of the target logical object, and the target logical object is encapsulated.

[0195] In one feasible embodiment, the processing module 502 is further configured to:

[0196] For each storage file, obtain the maximum offset in the corresponding data write offset set;

[0197] When the maximum offset is greater than the data encapsulation length, a first data recovery instruction is generated and sent to the server storing the storage file; the first data recovery instruction is used to cause the server storing the storage file to truncate the data length in the storage file to the data encapsulation length;

[0198] When the maximum offset is less than the data encapsulation length, a second data recovery instruction is generated and sent to the server storing the storage file. The second data recovery instruction is used to enable the server storing the storage file to retrieve the data after the maximum offset from other storage devices and perform data recovery based on the data after each maximum offset.

[0199] In one feasible embodiment, the processing module 502 is further configured to:

[0200] Obtain the identification information of the data to be read from the terminal device;

[0201] Query the identification information of the data to be read in the mapping relationship of each metadata storage to obtain the identification information of the corresponding target logical object, the data write offset and the data write length;

[0202] Obtain sub-data starting from the data write offset and with a data length equal to the data write length from at least R storage files corresponding to the identification information of the target logical object;

[0203] The erasure coding algorithm is used to process each sub-data to obtain the data to be read;

[0204] R storage files correspond to R sub-data items, and the R sub-data items include k+mr sub-data items.

[0205] In one feasible embodiment, the processing module 502 is further configured to:

[0206] Obtain the identification information of the data to be deleted from the terminal device;

[0207] Query the identification information of the data to be deleted in the mapping relationship of each metadata storage to obtain the identification information of the corresponding target logical object, the data write offset and the data write length;

[0208] Write the data corresponding to the data to be deleted to the offset for deletion, or write the data corresponding to the data to be deleted to the length set to 0.

[0209] Figure 9 This is a schematic diagram of the structure of a server according to an embodiment of this application. The server 600 includes a memory 601 and a processor 602. The memory 601 stores computer instructions executable by the processor. The memory 601 may include high-speed random access memory (RAM), and may also include non-volatile memory (NVM), such as at least one disk drive, or a USB flash drive, external hard drive, read-only memory, disk, or optical disc, etc.

[0210] When executing computer instructions, processor 602 implements the various steps of the data processing method with the server as the execution entity in the above embodiments. For details, please refer to the relevant descriptions in the foregoing method embodiments. The processor 602 can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. A general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in this invention can be directly manifested as being executed by a hardware processor, or executed by a combination of hardware and software modules within the processor.

[0211] Optionally, the memory 601 can be either independent or integrated with the processor 602. When the memory 601 is configured independently, the electronic device 600 also includes a bus for connecting the memory 601 and the processor 602. This bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. The bus can be categorized as an address bus, data bus, control bus, etc. For ease of illustration, the buses shown in the accompanying drawings are not limited to a single bus or a single type of bus.

[0212] This application also provides a computer-readable storage medium storing computer instructions, which, when executed by a processor, implement the various steps of the data processing method described above.

[0213] This application also provides a computer program product, including computer instructions, which, when executed by a processor, implement the various steps of the data processing method described above.

[0214] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this application are indicated by the following claims.

[0215] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this application is limited only by the appended claims.

Claims

1. A data processing method, characterized in that, The method is applied to a server, which is communicatively connected to a terminal device, and the server jointly deploys a distributed system with other servers. The method includes: Obtain the target data sent by the terminal device; When writing the target data into the target logical object, the target data is fragmented according to a preset erasure coding redundancy ratio to obtain W sub-data; the W sub-data includes k original sub-data and m check sub-data, and the data length of each sub-data is the same; Each of the sub-data items is written into multiple storage files corresponding to the target logical object; the number of storage files is the same as the number of sub-data items. When at least N storage files are successfully written, a write success message is sent back to the terminal device; the N storage files correspond to the storage of N sub-data, and the N sub-data include k original sub-data and r check sub-data, where m≥r≥0; After sending a successful write message to the terminal device, the method further includes: Obtain the offset of the target logical object; When the offset is greater than or equal to a preset encapsulation threshold, the corresponding data write offset set is obtained from each storage file. The data write offset set includes the offset of the data storage start position after each sub-data write. Based on each set of data write offsets, the offset corresponding to the last write data that satisfies the storage file write success condition is determined as the data encapsulation length; the storage file write success condition is that at least N storage files are successfully written in the storage files corresponding to the target logical object. The data encapsulation length is stored in the metadata of the target logical object, and the target logical object is encapsulated. For each of the aforementioned storage files, obtain the maximum offset in the corresponding data write offset set; When the maximum offset is greater than the data encapsulation length, a first data recovery instruction is generated and sent to the server storing the storage file; the first data recovery instruction is used to cause the server storing the storage file to truncate the data length in the storage file to the data encapsulation length; When the maximum offset is less than the data encapsulation length, a second data recovery instruction is generated and sent to the server storing the storage file; the second data recovery instruction is used to enable the server storing the storage file to obtain the data after the maximum offset from other storage, and to perform data recovery based on the data after each maximum offset.

2. The method according to claim 1, characterized in that, The distributed system has a client; Writing each of the sub-data into multiple storage files corresponding to the target logical object specifically includes: The client is controlled to obtain the first information of the record data storage status; Based on the first information and the W sub-data, generate W data write requests and transmit each data write request to the server storing the storage file; The data write request is used to enable the server storing the storage file to obtain second information about the data storage status from the local machine. When the data write request is at the head of the current request queue of the storage file, the first information is obtained from the data write request. When the first information and the second information are consistent, the sub-data in the data write request is stored into the storage file, and a write success message is sent back to the client after storage is completed. When the first information and the second information are inconsistent, the data write request is sorted to the end of the current request queue; the current request queue includes at least one data write request, and the first information in each data write request is different.

3. The method according to claim 2, characterized in that, When at least N storage files are successfully written, a write success message is sent to the terminal device, specifically including: When at least N successful write messages for storage files are received, a mapping relationship is constructed between the identification information of the target logical object, the identification information of the target data, the write offset of the target data, and the write length of the target data, and the mapping relationship is stored in the metadata of the target logical object. Update the data storage status information stored in the target logical object according to the write offset of the target data and the write length of the target data; Generate and send a write success message to the terminal device.

4. The method according to claim 3, characterized in that, The information on the data storage status includes the offset of data being written in the storage space allocated to the target logical object; Based on the write offset and write length of the target data, update the data storage status information stored in the target logical object, specifically including: The sum of the write offset of the target data and the write length of the target data is determined as the updated data storage status information of the target logical object after the target data is written.

5. The method according to claim 1, characterized in that, The method further includes: Obtain the identification information of the data to be read from the terminal device; The identification information of the data to be read is queried in the mapping relationship of each of the metadata storages to obtain the identification information of the corresponding target logical object, the data write offset and the data write length; Obtain sub-data starting from the data write offset and with a data length equal to the data write length from at least R storage files corresponding to the identification information of the target logical object; The data to be read is obtained by processing each of the sub-data using erasure coding algorithms; The R storage files each store R sub-data items, and the R sub-data items include k+mr sub-data items.

6. The method according to claim 1, characterized in that, The method further includes: Obtain the identification information of the data to be deleted from the terminal device; The identifier information of the data to be deleted is queried in the mapping relationship of each metadata storage to obtain the identifier information of the corresponding target logical object, the data write offset and the data write length; Write the data corresponding to the data to be deleted to the offset and delete it, or write the data corresponding to the data to be deleted to the length and set it to 0.

7. A data processing apparatus, characterized in that, include: The acquisition module is used to acquire the target data sent by the terminal device; The processing module is used to divide the target data into data fragments according to a preset erasure coding redundancy ratio to obtain W sub-data; the W sub-data includes k original sub-data and m check sub-data, and the data length of each sub-data is the same; The processing module is further configured to write each of the sub-data into multiple storage files corresponding to the target logical object; the number of storage files is the same as the number of sub-data. The processing module is also used to send a write success message to the terminal device when at least N storage files are successfully written; the N storage files correspond to storing N sub-data, and the N sub-data include k original sub-data and r check sub-data, where m≥r≥0; The processing module is also used to obtain the offset of the target logical object; When the offset is greater than or equal to a preset encapsulation threshold, the corresponding data write offset set is obtained from each storage file. The data write offset set includes the offset of the data storage start position after each sub-data write. Based on each set of data write offsets, the offset corresponding to the last write data that satisfies the storage file write success condition is determined as the data encapsulation length. The condition for successful writing of the storage file is that at least N storage files corresponding to the target logical object are successfully written. The data encapsulation length is stored in the metadata of the target logical object, and the target logical object is encapsulated. The processing module is also used to obtain the maximum offset in the corresponding data write offset set for each of the storage files; When the maximum offset is greater than the data encapsulation length, a first data recovery instruction is generated and sent to the server storing the storage file; The first data recovery instruction is used to instruct the server storing the stored file to truncate the data length in the stored file to the data encapsulation length; When the maximum offset is less than the data encapsulation length, a second data recovery instruction is generated and sent to the server storing the storage file; The second data recovery instruction is used to instruct the server storing the stored file to retrieve the data after the maximum offset from other storage, and to perform data recovery based on the data after each maximum offset.

8. A server, characterized in that, include: A processor and a memory communicatively connected to the processor; The memory stores computer-executed instructions; The processor, when executing the computer execution instructions, is used to implement the method as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that, when executed by a processor, are used to implement the method as described in any one of claims 1 to 6.

Images