Data processing method and device, server and storage medium

Abstract

The invention provides a data processing method and device, a server and a storage medium, when the server stores target data, the target data is split into k + m sub-data according to a preset erasure code redundancy ratio, in the storage process, information of successful data writing can be fed back as long as N sub-data greater than or equal to k is successfully written, and the data writing efficiency is improved. N is greater than or equal to k, so that data recovery can be completed through the successfully stored N sub-data even if the remaining W-N sub-data are not completely written, and the reliability of the target data is guaranteed.

Description

Technical Field

[0001] The present application relates to the field of distributed storage technology, and in particular to a data processing method, device, server and storage medium. Background Art

[0002] In the prior art, distributed storage can realize the storage of massive data. The storage of massive data maintains high reliability through redundant storage technology.

[0003] When using erasure coding technology for redundant storage, the server uses erasure coding technology to generate k+r copies of data and store them in the corresponding k+r storage nodes, where k is the number of slices of the original data and r is the number of slices of the check slices. When reading the stored data, you only need to obtain any at least k slices of data from the above k+r storage nodes to restore the original data through the erasure coding algorithm.

[0004] In the process of storing each slice data, a strong consistency writing strategy is generally adopted to ensure data reliability, that is, the data writing is determined to be successful only after the k+r slice data are all written to the corresponding storage node. However, due to the inconsistent processing efficiency of storage nodes for data storage tasks, there is a tendency for some storage nodes to have writing delays, resulting in a time delay in the server confirming that the original data has been written successfully, and the availability of the original data is poor. Summary of the invention

[0005] The present application provides a data processing method, device, server and storage medium to solve the above-mentioned problems.

[0006] In a first aspect, the present application provides a data processing method, which is applied to a server, the server is communicatively connected with a terminal device, and the server and other servers jointly deploy a distributed system, and the method includes:

[0007] Acquiring target data sent by the terminal device;

[0008] When writing the target data into the target logical object, the target data is segmented according to the preset erasure code redundancy ratio to obtain W sub-data; the W sub-data include k original sub-data and m check sub-data, and the data length of each sub-data is the same;

[0009] Writing each of the sub-data into a plurality of storage files corresponding to the target logical object; the number of the storage files is the same as the number of the sub-data;

[0010] When at least N storage files are written successfully, write success information is fed back to the terminal device; the N storage files store N sub-data correspondingly, and the N sub-data include k original sub-data and r check sub-data, m≥r≥0.

[0011] In the above technical solution, when storing target data, the server splits the target data into k+m sub-data according to the preset erasure code redundancy ratio. During the storage process, it only needs to successfully write N sub-data greater than or equal to k to feedback the information that the data is written successfully, so that the user can perform subsequent operations on the target data, thereby improving the availability of the target data. Since N is greater than or equal to k, even if the remaining WN sub-data have not been completely written, data recovery can be completed through the N sub-data that have been successfully stored, thereby ensuring the reliability of the target data.

[0012] Optionally, the distributed system is provided with a client, and the target logical object records information on data storage status;

[0013] Writing each of the sub-data into a plurality of storage files corresponding to the target logical object specifically includes:

[0014] Control the client to obtain first information recording data storage status;

[0015] Generate W data writing requests according to the first information and the W sub-data, and transmit each of the data writing requests to a server storing the storage file;

[0016] The data write request is used to enable the server storing the storage file to obtain second information recording the data storage status from the local computer. When the data write request is at the head of the current request queue of the storage file, the first information is obtained from the data write request. When the first information and the second information are consistent, the sub-data in the data write request is stored in the storage file, and after the storage is completed, a write success message is fed back to the client.

[0017] When the first information and the second information are inconsistent, the data write request is sorted to the end of the current request queue; the current request queue includes at least one data write request, and the first information in each data write request is different.

[0018] In the above technical solution, when the controller stores multiple data to the target logical object, it will slice the data in sequence according to the order in which the data is obtained, and generate multiple data write requests according to the first information that records the storage status of each data before being stored in the target logical object, and transmit the write request to the corresponding server. After obtaining at least one data write request, the server adds each data write request to the request queue and responds in the order of the request queue. During the response process, the second information that records the storage status is obtained locally and compared with the first information in the currently responded data write request. When they are consistent, the order of the data write request in the request queue is accurate, and the sub-data in the request can be stored in the local storage file, the second information is updated, and the corresponding write success information is fed back. When the first information and the second information are inconsistent, it is determined that there are other data that have not been written before the sub-data of the data write request is written, and the position of the data write request in the request queue is adjusted, so that the server can find the data write request containing the sub-data to be written and write the data correctly, thereby ensuring the sequential consistency of storing each data slice in each server.

[0019] Optionally, when at least N storage files are written successfully, feeding back writing success information to the terminal device specifically includes:

[0020] When the write success information of at least N storage files is obtained, a mapping relationship among the identification information of the target logical object, the identification information of the target data, the write offset of the target data, and the write length of the target data is constructed, and the mapping relationship is stored in the metadata of the target logical object;

[0021] updating the data storage status information stored in the target logical object according to the write offset of the target data and the write length of the target data;

[0022] Generate and send write success information to the terminal device.

[0023] Optionally, the information of the data storage state includes an offset of data written in the storage space allocated to the target logical object;

[0024] Updating the data storage status information stored in the target logical object according to the write offset of the target data and the write length of the target data, specifically including:

[0025] The sum of the write offset of the target data and the write length of the target data is determined as the updated data storage state information of the target logical object after the target data is written.

[0026] Optionally, after feeding back writing success information to the terminal device, the method further includes:

[0027] Obtaining the offset of the target logical object;

[0028] When the offset is greater than or equal to a preset encapsulation threshold, a corresponding data write offset set is obtained from each storage file, wherein the data write offset set includes an offset of a data storage start position after each sub-data is written;

[0029] According to each of the data write offset sets, the offset corresponding to the last write data satisfying the storage file write success condition is determined as the data encapsulation length; the storage file write success condition is that at least N storage files are written successfully in the storage file corresponding to the target logical object;

[0030] The data encapsulation length is stored in the metadata of the target logical object, and the target logical object is encapsulated.

[0031] Optionally, the method further comprises:

[0032] For each of the storage files, obtaining the maximum offset in the corresponding data write offset set;

[0033] When the maximum offset is greater than the data encapsulation length, a first data recovery instruction is generated, and the first data recovery instruction is sent to the server storing the storage file; the first data recovery instruction is used to make the server storing the storage file truncate the data length in the storage file to the data encapsulation length;

[0034] When the maximum offset is less than the data encapsulation length, a second data recovery instruction is generated and sent to the server storing the storage file; the second data recovery instruction is used to enable the server storing the storage file to obtain the data after the maximum offset from other memories, and perform data recovery based on the data after each maximum offset.

[0035] In the above technical scheme, after data encapsulation, it can be determined whether the stored data needs to be corrected based on the maximum offset of the data in each storage file and the data encapsulation length. During the correction process, when the maximum offset is greater than the data encapsulation length, it is determined that some sub-files in the storage file have failed to be written in at least WN other storage files. Users cannot use these sub-files to restore the original data, and it is necessary to truncate the data length in the storage file, and determine the data within the data encapsulation length as valid stored data to prevent the server from repeatedly storing the sub-data when the data that failed to be written is rewritten, thereby improving the utilization rate of the storage space; when the maximum offset is less than the data encapsulation length, it is determined that there is a data write failure when storing data in the storage file, and the data at the corresponding position in other storage files can be used for data recovery to make up for the missing data in the current storage file, thereby ensuring the strong reliability of the data.

[0036] Optionally, the method further comprises:

[0037] Acquire identification information of the data to be read from the terminal device;

[0038] Inquiring the identification information of the data to be read in the mapping relationship stored in each metadata, obtaining the identification information of the corresponding target logical object, the data writing offset and the data writing length;

[0039] Acquire sub-data starting from the data writing offset and having a data length equal to the data writing length from at least R storage files corresponding to the identification information of the target logical object;

[0040] Processing each of the sub-data using an erasure code algorithm to obtain the data to be read;

[0041] The R storage files store R sub-data correspondingly, and the R sub-data include k+mr sub-data.

[0042] Optionally, the method further comprises:

[0043] Acquire identification information of the data to be deleted from the terminal device;

[0044] Inquiring the identification information of the to-be-deleted data in the mapping relationship stored in each metadata, obtaining the identification information, data writing offset and data writing length of the corresponding target logical object;

[0045] The data write offset corresponding to the data to be deleted is deleted, or the data write length corresponding to the data to be deleted is set to 0.

[0046] In a second aspect, the present application provides a data processing device, including:

[0047] An acquisition module, used to acquire target data sent by a terminal device;

[0048] A processing module, configured to perform data sharding on the target data according to a preset erasure code redundancy ratio to obtain W sub-data; the W sub-data include k original sub-data and m check sub-data, and the data length of each sub-data is the same;

[0049] The processing module is further used to write each sub-data into a plurality of storage files corresponding to the target logical object; the number of the storage files is the same as the number of the sub-data;

[0050] The processing module is also used to feed back write success information to the terminal device when at least N storage files are written successfully; the N storage files store N sub-data correspondingly, and the N sub-data include k original sub-data and r check sub-data, m≥r≥0.

[0051] In a third aspect, the present application provides a server, including: a processor and a memory communicatively connected to the processor;

[0052] Memory stores computer instructions;

[0053] When executing the computer instructions, the processor is used to implement the method involved in the first aspect.

[0054] In a fourth aspect, the present application provides a computer-readable storage medium, in which computer instructions are stored. When the computer instructions are executed by a processor, they are used to implement the method involved in the first aspect.

[0055] The present application provides a data processing method, device, server and storage medium. When storing target data, the server splits the target data into k+m sub-data according to a preset erasure code redundancy ratio. During the storage process, only N sub-data greater than or equal to k need to be successfully written to feedback information that the data is written successfully, so that the user can perform subsequent operations on the target data, thereby improving the availability of the target data. Since N is greater than or equal to k, even if the remaining WN sub-data have not been completely written, data recovery can be completed through the N sub-data that have been successfully stored, thereby ensuring the reliability of the target data. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0057] Figure 1 A scene graph for data distribution storage provided by the present application according to an exemplary embodiment;

[0058] Figure 2A schematic diagram of data storage in a distributed storage system provided by the present application according to an exemplary embodiment;

[0059] Figure 3 A schematic diagram of a data processing method provided by the present application according to an exemplary embodiment;

[0060] Figure 4 A schematic diagram of distributed writing of target data provided by the present application according to an exemplary embodiment;

[0061] Figure 5 An interactive diagram of a data processing method provided by the present application according to an exemplary embodiment;

[0062] Figure 6 An interactive diagram of a data recovery method provided by the present application according to an exemplary embodiment;

[0063] Figure 7 A schematic diagram of data recovery provided by the present application according to an exemplary embodiment;

[0064] Figure 8 A schematic diagram of the structure of a data processing device provided by the present application according to an exemplary embodiment;

[0065] Fig. 9 A schematic diagram of the structure of a server provided in accordance with an embodiment of the present application.

[0066] The above drawings have shown clear embodiments of the present application, which will be described in more detail later. These drawings and text descriptions are not intended to limit the scope of the present application in any way, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0067] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0068] Figure 1 A scene graph for storing data distribution, such as Figure 1 As shown, it includes a terminal device 10 and a server cluster 20, which are communicatively connected to each other. The server cluster 20 includes multiple servers 201, which are used as storage nodes and are communicatively connected to each other to form a distributed storage system, which can store data in a distributed manner.

[0069] Figure 2 The following is a schematic diagram of distributed storage system for distributed storage of data. Figure 2 As shown, the server 20 is provided with a client 301 and a management node 302. A server 20 provides a KV storage 303 service, and the server 20 can communicate with other storage nodes 304 through the client 301. When the distributed storage system stores data, the terminal device 10 calls the client 301 to transmit the data to a server 201. After the server 201 requests a target logical object extent 305 from the management node 302, the data can be written into the storage node 304 corresponding to the target logical object 305.

[0070] When writing data, the server 201 slices the data and stores each sliced ​​sub-data in different storage nodes 304. In the process of data storage, it is necessary to use redundant storage technology to maintain high reliability to prevent the complete data from other storage nodes when some storage nodes cannot obtain data. Redundant storage technology includes multi-copy storage technology, erasure code technology, etc.

[0071] When using erasure coding technology for redundant storage, server 201 uses erasure coding technology to generate k+r copies of data and stores them in corresponding k+r storage nodes, where k is the number of slices of the original data and r is the number of slices of the check slices. When reading the stored data, it is only necessary to obtain any at least k slices of data from the above k+r storage nodes to restore the original data through calculation using the erasure coding algorithm.

[0072] In the process of storing each slice data, a strong consistency writing strategy is generally adopted to ensure data reliability, that is, after all k+r slice data are written in the corresponding storage nodes, the information of successful data writing is fed back to the terminal device 10, so that the terminal device 10 can operate the above stored data, such as query, delete, etc. However, due to the inconsistent processing efficiency of storage nodes for data storage tasks, there is a tendency for some storage nodes to be delayed in writing, resulting in a time delay in the server determining that the original data is successfully written, and the availability of the original data is poor.

[0073] In order to solve the above problems, the present application provides a data processing method, device, server and storage medium. The technical concept of the present application is: in the process of data storage, on the basis of ensuring that the data can be restored using sub-data stored in multiple storage files, the reliability of the data is reduced to improve the availability of the data.

[0074] Figure 3 FIG. 1 is a flow chart of a data processing method provided by the present application according to an exemplary embodiment. Figure 3 As shown, the method includes:

[0075] S101. The server obtains target data sent by the terminal device.

[0076] The target data is the data to be stored in the distributed system.

[0077] In one embodiment, a client is provided in the distributed system, and the terminal device sends target data to the server by calling the client in the server.

[0078] S102: When the server writes the target data into the target logical object, it partitions the target data according to the preset erasure code redundancy ratio to obtain W sub-data.

[0079] The target logical object is an object created by the management node in the server. When the storage space allocated to the target logical object is not fully written and the terminal device writes data again, additional writing can be performed on the target logical object.

[0080] When the preset erasure code redundancy ratio is k:m, the server generates W sub-data from the target data according to the preset erasure code redundancy ratio, wherein the W sub-data include k original sub-data and m check sub-data, and the data length of each sub-data is the same.

[0081] S103: The server writes each sub-data into multiple storage files corresponding to the target logical object.

[0082] Each storage file is stored in each storage node, that is, each storage file in other servers is in a corresponding relationship with the target logical object.

[0083] The server may obtain the correspondence between the target logical object and each storage file, and after generating W sub-data using the obtained target data, write the W sub-data into the corresponding W storage files according to the correspondence between the target logical object and each storage file.

[0084] After the storage of the file to the sub-data storage is completed, a write success message is fed back to the client in the server.

[0085] S104: When at least N storage files are written successfully, the server feeds back writing success information to the terminal device.

[0086] Since the efficiency of storing sub-data in each storage node is not exactly the same, the time when the storage information is fed back by each storage node is also not exactly the same.

[0087] When the client obtains the storage information of at least N storage files, it can be determined that the target data is successfully stored in the distributed system, and the server feeds back the write success information to the terminal device.

[0088] The N sub-data include k original sub-data and r check sub-data, m≥r≥0.

[0089] Since N is greater than or equal to k, when the terminal device obtains the original data, the client can use the N sub-data and the erasure coding technology to restore the target data and provide the target data to the terminal device.

[0090] After the server feeds back the write success information to the terminal device through the client, the storage nodes that have not written successfully will continue to complete the data writing to the storage file to ensure the reliability of the data.

[0091] In the above technical solution, when storing target data, the server splits the target data into k+m sub-data according to the preset erasure code redundancy ratio. During the storage process, it only needs to successfully write N sub-data greater than or equal to k to feedback the information that the data is written successfully, so that the user can perform subsequent operations on the target data, thereby improving the availability of the target data. Since N is greater than or equal to k, even if the remaining WN sub-data have not been completely written, data recovery can be completed through the N sub-data that have been successfully stored, thereby ensuring the reliability of the target data.

[0092] Figure 4 This is a schematic diagram of distributed writing of target data provided by the present application according to an exemplary embodiment, such as Figure 4 As shown, when the client 301 in the server 201 is called by the terminal device 10, the target data ③ is obtained, and the client 301 writes the target data ③ into the target logical object 305. Since other data has been written into the target logical object 305 before the target data ③ is written, the target logical object 305 needs to be additionally written in the writing order.

[0093] When writing the target logical object 305, the server 201 slices the data according to the redundancy ratio of 2:1, and obtains three sub-data ③', including 2 original sub-data and 1 check sub-data. The server 201 transmits the above three sub-data ③' to the three storage nodes 304 respectively, and stores them in the storage files 3041 in the storage nodes 304. The order of storing the sub-data in each storage file 3041 is the same as the order in which the client 301 stores the data in the target logical object 305.

[0094] After storage is completed, each storage node 304 feeds back information of successful writing to the client 301. After determining that at least two data are written successfully, the client 301 can store the mapping relationship between the target data and the target logical object and the related storage information in the KV storage 303, and feed back information of successful writing to the terminal device 10, so that the terminal device 10 can perform subsequent operations on the stored data.

[0095] The storage nodes 304 that have not completed writing will continue to write the sub-data ③′ they have obtained after the client 301 feeds back writing success information to the terminal device 10 , thereby ensuring the reliability of the data in each storage node 304 .

[0096] The following is a detailed explanation of the process of writing target data to each storage node.

[0097] Figure 5 FIG. 1 is an interactive diagram of a data processing method provided by the present application according to another exemplary embodiment. Figure 5 As shown, including:

[0098] S201. A first server controls a client to obtain first information recording a data storage status.

[0099] In this embodiment, the first server is a server for obtaining target data, and the second server is a server that contains a storage file and stores sub-data after the target data is segmented.

[0100] The first information recording the data storage state includes a data write offset in the target logical object.

[0101] The data write offset in the target logical object indicates the currently used length of the target logical object.

[0102] If the target logical object is a newly created object of the management node and no data has been written to it, the data write offset is 0; if the target logical object has data written to it, the data write offset is the data write offset updated according to the length of the written data after the last data was written.

[0103] For example, if the target logical object has been written with data once and the write length of the data is 10, then the current data write offset is 10.

[0104] In one embodiment, the first information is stored locally on the first server. In another embodiment, the first information is stored in other servers serving as KV storage, which is not specifically limited here.

[0105] S202: The first server generates W data writing requests according to the first information and the W sub-data.

[0106] S203: The first server sends a data write request to the second server.

[0107] S204: The second server obtains second information recording the data storage status locally.

[0108] The second server is provided with a storage file, which corresponds to the target logical object, and the order of storing data in the storage file is the same as the order in which the first server writes data to the target logical object through the client.

[0109] The second information includes a data write offset in the storage file, where the data write offset indicates a currently used length of the storage space of the storage file.

[0110] If the second server can successfully store data each time the first server transmits data to it, the first information and the second information are consistent.

[0111] S205. The second server obtains a data write request from the head end of the current request queue for storing files, and obtains first information from the data write request.

[0112] When receiving a data write request, the second server adds the request to the request queue and processes it according to the first-in-first-out principle, that is, the request at the head of the request queue is the earliest request obtained, and the second server will respond to the request first.

[0113] In some cases, when the second server responds to requests slowly and the terminal device continuously transmits multiple data for storage, there are multiple requests continuously obtained by the second server in the request queue of the second server.

[0114] In order to ensure the accuracy of the data writing order, it is necessary to first obtain the first information from the data writing request when responding to each data writing request in the queue, compare it with the second information obtained locally, and determine whether the data writing offset of the current data writing request is consistent with the data writing offset of the storage file for this write, so as to verify whether the order of the current data writing requests is consistent.

[0115] If the first information and the second information are consistent, the process proceeds to step S206; if the first information and the second information are inconsistent, the process proceeds to step S208.

[0116] S206: When the first information and the second information are consistent, the second server stores the sub-data in the data write request into the storage file.

[0117] When the first information and the second information are consistent, it means that the data currently to be written into the storage file in the second server is the data in the current data write request, and the sub-data in the data write request is stored in the storage file.

[0118] S207: After the storage is completed, the second server feeds back a write success message to the first server.

[0119] S208: When the first information and the second information are inconsistent, the second server sorts the data write request to the end of the current request queue for storing the file.

[0120] When the first information and the second information are inconsistent, it means that the order in which multiple data write requests enter the request queue of the second server is inconsistent with the order of transmission by the first server, and the data write request at the head of the current request queue cannot be responded to. It is necessary to sort the requests in the request queue to find the data write request that is consistent with the first information and the second information.

[0121] In one embodiment, when the first information and the second information of the data write request located at the head end of the request queue are inconsistent, it is queued to the tail end of the request queue, and the request located at the original second position in the request queue is advanced to the head end of the request queue, entering step S205 so that the second server verifies the first information and the second information in the request until a request with accurate write order is found.

[0122] S209. When the first server obtains the write success information of at least N storage files, it constructs a mapping relationship among the identification information of the target logical object, the identification information of the target data, the write offset of the target data, and the write length of the target data, and stores the mapping relationship in the metadata of the target logical object.

[0123] The first server may obtain identification information of the target data when the terminal transmits the target data to the first server.

[0124] The write offset of the target data is obtained from the metadata of the target logical object. If the target logical object is a newly created object of the management node, the write offset of the target data is 0; if the target logical object has been written with data, the write offset of the target data is the sum of the lengths of the data written to the target logical object.

[0125] The write length of the target data is determined according to the data length of the target data and the number of data slices.

[0126] S210. The first server updates the data storage status information stored in the target logical object according to the write offset of the target data and the write length of the target data.

[0127] The first server determines the sum of the write offset of the target data and the write length of the target data as the updated data storage state information of the target logical object after the target data is written.

[0128] S211. The first server generates and sends write success information to the terminal device.

[0129] In the above technical solution, when the controller stores multiple data to the target logical object, it will slice the data in sequence according to the order in which the data is obtained, and generate multiple data write requests according to the first information that records the storage status of each data before being stored in the target logical object, and transmit the write request to the corresponding server. After obtaining at least one data write request, the server adds each data write request to the request queue and responds in the order of the request queue. During the response process, the second information that records the storage status is obtained locally and compared with the first information in the currently responded data write request. When they are consistent, the order of the data write request in the request queue is accurate, and the sub-data in the request can be stored in the local storage file, the second information is updated, and the corresponding write success information is fed back. When the first information and the second information are inconsistent, it is determined that there are other data that have not been written before the sub-data of the data write request is written, and the position of the data write request in the request queue is adjusted, so that the server can find the data write request containing the sub-data to be written and write the data correctly, thereby ensuring the sequential consistency of storing each data slice in each server.

[0130] When the storage space allocated to the target logical object is full, the target logical object needs to be encapsulated, and after the encapsulation is completed, each storage file is checked and the corresponding data is restored.

[0131] The encapsulation process of the target logical object is explained below.

[0132] Figure 6 This is an interactive diagram of a data recovery method provided by the present application according to an exemplary embodiment, such as Figure 6 As shown, the method includes:

[0133] S401: The first server obtains an offset of a target logical object.

[0134] The offset of the target logical object is the offset updated on the corresponding metadata after the last data storage of the target logical object is completed. The method for updating the offset of the target logical object has been explained in step S210 and will not be repeated here.

[0135] S402: When the offset is greater than or equal to a preset encapsulation threshold, the first server obtains a corresponding data write offset set from the storage files of each second server.

[0136] When the offset is greater than or equal to the preset encapsulation threshold, the first server encapsulates the target logical object.

[0137] The first server calls the encapsulation interface of the management node so that the first server obtains the corresponding data write offset set from each second server.

[0138] The data write offset set includes the offset of the data storage start position after each sub-data is written.

[0139] The data write offset is generated by each second server according to the writing result of each sub-data. When the data is written successfully, the second server storing each storage file will store the offset updated after the data is written in the local data write offset set, and use the latest stored offset as the second information: when the data write fails, the second server cannot store the offset corresponding to the data in the data write offset set, and the second information cannot be updated. Based on this, the sub-data obtained subsequently cannot be stored in the second server.

[0140] S403: The first server determines the offset corresponding to the last write data satisfying the storage file write success condition as the data encapsulation length according to each data write offset set.

[0141] The storage file writing success condition is that at least N storage files are written successfully in the storage files corresponding to the target logical object.

[0142] That is, the first server counts the offset updated after each sub-data is written to each storage file, determines the number of successful data writes each time, and obtains the data offset updated after the last time at least N storage files successfully stored sub-files as the data encapsulation length.

[0143] The server can restore the complete data based on the data in each storage file that is less than or equal to the data encapsulation length.

[0144] like Figure 7 As shown, the maximum length of the data stored in 5 storage files is included, wherein the data length of the first storage file and the second storage file is cl2, the data length of the third storage file is cl1, the data length of the fourth storage file is cl3, and the data length of the fifth storage file is 0.

[0145] When the storage file writing success condition is that at least 3 storage files are written successfully, it is determined that the first storage file, the second storage file and the fourth storage file are written successfully, and the minimum data length of the three storage files is the data encapsulation length, and the data length is cl2.

[0146] S404: The first server stores the data encapsulation length in the metadata of the target logical object, and encapsulates the target logical object.

[0147] S405: For each storage file, the first server obtains the maximum offset in the corresponding data write offset set.

[0148] That is, the first server obtains the maximum storage data length of each storage file.

[0149] After obtaining the maximum offset of each storage file, the first server compares it with the data encapsulation length.

[0150] When the maximum offset is greater than the data encapsulation length, for example Figure 7 The maximum offset of the fourth storage file in the process proceeds to step S406 for data recovery.

[0151] When the maximum offset is less than the data encapsulation length, for example Figure 7 The maximum offset of the third storage file and the fifth storage file in the process proceeds to step S409 for data recovery.

[0152] When the maximum offset is equal to the data encapsulation length, for example Figure 7 The maximum offset of the first storage file and the second storage file in the data storage indicates that no data recovery is required.

[0153] S406: When the maximum offset is greater than the data encapsulation length, the first server generates a first data recovery instruction.

[0154] When the maximum offset of a storage file is greater than the data encapsulation length, it is determined that the storage file stores other data after the data encapsulation length, but the amount of the data successfully stored in conjunction with other storage files is less than N, indicating that the data is not stored successfully and the extra data stored in the storage file needs to be intercepted.

[0155] Based on this, the first server generates a first data recovery instruction.

[0156] S407: The first server sends a first data recovery instruction to the second server.

[0157] S408. In response to the first data recovery instruction, the second server truncates the data length in the storage file to the data encapsulation length.

[0158] exist Figure 7 In the illustrated embodiment, data greater than cl2 and less than cl3 ​​in the fourth storage file are truncated, and only data less than or equal to cl2 is determined as valid data.

[0159] For the truncated data, after the first server requests a new target logical object, the storage file corresponding to the new target logical object may be rewritten.

[0160] S409: When the maximum offset is less than the data encapsulation length, the first server generates a second data recovery instruction.

[0161] When the maximum offset is less than the data encapsulation length, it is determined that the storage file has failed to store at least one sub-data, but at least N other storage files have been stored successfully. It is necessary to recover the data that failed to be stored to ensure the data reliability of each storage file.

[0162] Based on this, the first server generates a second data recovery instruction.

[0163] S410: The first server sends a second data recovery instruction to the second server.

[0164] S411. In response to the second data recovery instruction, the second server obtains data after the maximum offset from other memories, and performs data recovery based on the data after each maximum offset.

[0165] exist Figure 7 In the illustrated embodiment, the missing data in the third storage file needs to be recovered using data in the first storage file, the second storage file and the fourth storage file that is greater than cl1 and less than or equal to cl2, so that the data length of the third storage file is also the encapsulation length.

[0166] It is also necessary to restore all the data in the fifth storage file from the data that is less than or equal to cl2 in the first storage file, the second storage file, and the fourth storage file, and write the entire data to the fifth storage file.

[0167] In the above technical scheme, after data encapsulation, it can be determined whether the stored data needs to be corrected based on the maximum offset of the data in each storage file and the data encapsulation length. During the correction process, when the maximum offset is greater than the data encapsulation length, it is determined that some sub-files in the storage file have failed to be written in at least WN other storage files. Users cannot use these sub-files to restore the original data, and it is necessary to truncate the data length in the storage file, and determine the data within the data encapsulation length as valid stored data to prevent the server from repeatedly storing the sub-data when the data that failed to be written is rewritten, thereby improving the utilization rate of the storage space; when the maximum offset is less than the data encapsulation length, it is determined that there is a data write failure when storing data in the storage file, and the data at the corresponding position in other storage files can be used for data recovery to make up for the missing data in the current storage file, thereby ensuring the strong reliability of the data.

[0168] After the data is stored, the terminal device can operate the data, such as reading, deleting, etc.

[0169] In one embodiment, when the terminal device reads the stored data, the first server obtains the identification information of the data to be read from the terminal device, queries the identification information of the data to be read in the mapping relationship stored in each metadata, and obtains the identification information of the corresponding target logical object, the data write offset and the data write length.

[0170] Considering the possibility of write failure during data writing, sub-data starting from the data write offset and having a data length equal to the data write length are obtained from at least R storage files corresponding to the identification information of the target logical object, and each sub-data is processed using an erasure code algorithm to obtain the data to be read. Among them, R storage files store R sub-data correspondingly, and the R sub-data include k+mr sub-data.

[0171] In another embodiment, when the terminal device deletes stored data, the first server obtains identification information of the data to be deleted from the terminal device, queries the identification information of the data to be deleted in the mapping relationship stored in each metadata, obtains the identification information, data write offset and data write length of the corresponding target logical object, deletes the data write offset corresponding to the data to be deleted, or sets the data write length corresponding to the data to be deleted to 0.

[0172] In the subsequent data sorting process, each second server will obtain data with a data write length not equal to 0 according to the mapping relationship associated with the target logical object for re-storage, and the previously deleted data will no longer be retained in the re-storage process.

[0173] Figure 8 This is a schematic diagram of the structure of a data processing device provided according to an embodiment of the present application. The data processing device 500 includes an acquisition module 501 and a processing module 502, wherein:

[0174] The acquisition module 501 is used to acquire the target data sent by the terminal device;

[0175] The processing module 502 is used to, when writing the target data into the target logical object, perform data sharding on the target data according to the preset erasure code redundancy ratio to obtain W sub-data; the W sub-data include k original sub-data and m check sub-data, and the data length of each sub-data is the same;

[0176] The processing module 502 is further used to write each sub-data into a plurality of storage files corresponding to the target logical object; the number of the storage files is the same as the number of the sub-data;

[0177] The processing module 502 is also used to feedback write success information to the terminal device when at least N storage files are written successfully; the N storage files store N sub-data correspondingly, and the N sub-data include k original sub-data and r check sub-data, m≥r≥0.

[0178] In a feasible embodiment, the processing module 502 is specifically used for:

[0179] Each sub-data is written into multiple storage files corresponding to the target logical object, including:

[0180] Control the client to obtain the first information of recording data storage status;

[0181] Generate W data write requests according to the first information and the W sub-data, and transmit each data write request to a server storing the storage file;

[0182] The data write request is used to enable the server storing the storage file to obtain the second information recording the data storage status from the local computer. When the data write request is at the head of the current request queue of the storage file, the first information is obtained from the data write request. When the first information and the second information are consistent, the sub-data in the data write request is stored in the storage file, and after the storage is completed, the write success information is fed back to the client.

[0183] When the first information and the second information are inconsistent, the data write request is sorted to the end of the current request queue; the current request queue includes at least one data write request, and the first information in each data write request is different.

[0184] In a feasible embodiment, the processing module 502 is specifically used for:

[0185] When the write success information of at least N storage files is obtained, a mapping relationship among the identification information of the target logical object, the identification information of the target data, the write offset of the target data, and the write length of the target data is constructed, and the mapping relationship is stored in the metadata of the target logical object;

[0186] Update the data storage status information stored in the target logical object according to the write offset of the target data and the write length of the target data;

[0187] Generates and sends write success information to the terminal device.

[0188] In a feasible embodiment, the processing module 502 is specifically used for:

[0189] The sum of the write offset of the target data and the write length of the target data is determined as the updated data storage state information of the target logical object after the target data is written.

[0190] In a feasible embodiment, the processing module 502 is further configured to:

[0191] Get the offset of the target logical object;

[0192] When the offset is greater than or equal to the preset encapsulation threshold, a corresponding data write offset set is obtained from each storage file, where the data write offset set includes the offset of the data storage start position after each sub-data is written;

[0193] According to each data write offset set, the offset corresponding to the last write data that meets the storage file write success condition is determined as the data encapsulation length; the storage file write success condition is that at least N storage files are written successfully in the storage file corresponding to the target logical object;

[0194] The data encapsulation length is stored in the metadata of the target logical object, and the target logical object is encapsulated.

[0195] In a feasible embodiment, the processing module 502 is further configured to:

[0196] For each storage file, obtain the maximum offset of the corresponding data write offset set;

[0197] When the maximum offset is greater than the data encapsulation length, a first data recovery instruction is generated and sent to the server storing the storage file; the first data recovery instruction is used to cause the server storing the storage file to truncate the data length in the storage file to the data encapsulation length;

[0198] When the maximum offset is less than the data encapsulation length, a second data recovery instruction is generated and sent to the server storing the storage file; the second data recovery instruction is used to enable the server storing the storage file to obtain the data after the maximum offset from other memories and perform data recovery based on the data after each maximum offset.

[0199] In a feasible embodiment, the processing module 502 is further configured to:

[0200] Obtaining identification information of the data to be read from the terminal device;

[0201] Inquiring the identification information of the data to be read in the mapping relationship stored in each metadata, obtaining the identification information of the corresponding target logical object, the data write offset and the data write length;

[0202] Acquire sub-data starting from the data writing offset and having a data length equal to the data writing length from at least R storage files corresponding to the identification information of the target logical object;

[0203] Use the erasure code algorithm to process each sub-data to obtain the data to be read;

[0204] The R storage files store R sub-data correspondingly, and the R sub-data include k+mr sub-data.

[0205] In a feasible embodiment, the processing module 502 is further configured to:

[0206] Obtaining identification information of the data to be deleted from the terminal device;

[0207] Query the identification information of the data to be deleted in the mapping relationship stored in each metadata, and obtain the identification information, data writing offset and data writing length of the corresponding target logical object;

[0208] Delete the data write offset corresponding to the data to be deleted, or set the data write length corresponding to the data to be deleted to 0.

[0209] Fig. 9 A schematic diagram of the structure of a server provided according to an embodiment of the present application. The server 600 includes a memory 601 and a processor 602, and the memory 601 is used to store computer instructions executable by the processor. The memory 601 may include a high-speed random access memory (Random Access Memory, RAM), and may also include a non-volatile memory (Non-Volatile Memory, NVM), such as at least one disk memory, and may also be a USB flash drive, a mobile hard disk, a read-only memory, a disk or an optical disk, etc.

[0210] The processor 602 implements each step of the data processing method with the server as the execution subject in the above embodiment when executing computer instructions. For details, please refer to the relevant description in the above method embodiment. The processor 602 can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), etc. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in the invention can be directly embodied as a hardware processor for execution, or can be executed by a combination of hardware and software modules in the processor.

[0211] Optionally, the memory 601 can be independent or integrated with the processor 602. When the memory 601 is independently provided, the electronic device 600 further includes a bus for connecting the memory 601 and the processor 602. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, the bus in the drawings of the present application is not limited to only one bus or one type of bus.

[0212] An embodiment of the present application further provides a computer-readable storage medium, in which computer instructions are stored. When a processor executes the computer instructions, each step of the data processing method in the above embodiment is implemented.

[0213] An embodiment of the present application also provides a computer program product, including computer instructions, which implement the various steps of the data processing method in the above embodiment when executed by a processor.

[0214] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the invention disclosed herein. The present application is intended to cover any modification, use or adaptation of the present application, which follows the general principles of the present application and includes common knowledge or customary techniques in the art that are not disclosed in the present application. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present application are indicated by the following claims.

[0215] It should be understood that the present application is not limited to the precise structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present application is limited only by the appended claims.

Claims

1. A data processing method, characterized in that: The method is applied to a server, the server is in communication connection with a terminal device, the server and other servers jointly deploy a distributed system, and the method includes: Acquiring target data sent by the terminal device; When writing the target data into the target logical object, the target data is segmented according to the preset erasure code redundancy ratio to obtain W sub-data; the W sub-data include k original sub-data and m check sub-data, and the data length of each sub-data is the same; Writing each of the sub-data into a plurality of storage files corresponding to the target logical object; the number of the storage files is the same as the number of the sub-data; When at least N storage files are written successfully, write success information is fed back to the terminal device; the N storage files store N sub-data correspondingly, and the N sub-data include k original sub-data and r check sub-data, m≥r≥0.

2. The method according to claim 1, characterized in that The distributed system is provided with a client; Writing each of the sub-data into a plurality of storage files corresponding to the target logical object specifically includes: Control the client to obtain first information recording data storage status; Generate W data writing requests according to the first information and the W sub-data, and transmit each of the data writing requests to a server storing the storage file; The data write request is used to enable the server storing the storage file to obtain second information recording the data storage status from the local computer. When the data write request is at the head of the current request queue of the storage file, the first information is obtained from the data write request. When the first information and the second information are consistent, the sub-data in the data write request is stored in the storage file, and after the storage is completed, a write success message is fed back to the client. When the first information and the second information are inconsistent, the data write request is sorted to the end of the current request queue; the current request queue includes at least one data write request, and the first information in each data write request is different.

3. The method according to claim 2, characterized in that When at least N storage files are written successfully, feedback of writing success information is sent to the terminal device, specifically including: When the write success information of at least N storage files is obtained, a mapping relationship among the identification information of the target logical object, the identification information of the target data, the write offset of the target data, and the write length of the target data is constructed, and the mapping relationship is stored in the metadata of the target logical object; updating the data storage status information stored in the target logical object according to the write offset of the target data and the write length of the target data; Generate and send write success information to the terminal device.

4. The method according to claim 3, characterized in that The information of the data storage status includes the offset of data written in the storage space allocated to the target logical object; Updating the data storage status information stored in the target logical object according to the write offset of the target data and the write length of the target data, specifically including: The sum of the write offset of the target data and the write length of the target data is determined as the updated data storage state information of the target logical object after the target data is written.

5. The method according to claim 1, characterized in that After feeding back the writing success information to the terminal device, the method further includes: Obtaining the offset of the target logical object; When the offset is greater than or equal to a preset encapsulation threshold, a corresponding data write offset set is obtained from each storage file, wherein the data write offset set includes an offset of a data storage start position after each sub-data is written; According to each of the data write offset sets, the offset corresponding to the last write data satisfying the storage file write success condition is determined as the data encapsulation length; the storage file write success condition is that at least N storage files are written successfully in the storage file corresponding to the target logical object; The data encapsulation length is stored in the metadata of the target logical object, and the target logical object is encapsulated.

6. The method according to claim 5, characterized in that The method further comprises: For each of the storage files, obtaining the maximum offset in the corresponding data write offset set; When the maximum offset is greater than the data encapsulation length, a first data recovery instruction is generated, and the first data recovery instruction is sent to the server storing the storage file; the first data recovery instruction is used to make the server storing the storage file truncate the data length in the storage file to the data encapsulation length; When the maximum offset is less than the data encapsulation length, a second data recovery instruction is generated and sent to the server storing the storage file; the second data recovery instruction is used to enable the server storing the storage file to obtain the data after the maximum offset from other memories, and perform data recovery based on the data after each maximum offset.

7. The method according to claim 5, characterized in that The method further comprises: Acquire identification information of the data to be read from the terminal device; Inquiring the identification information of the to-be-read data in the mapping relationship stored in each metadata, obtaining the identification information of the corresponding target logical object, the data writing offset and the data writing length; Acquire sub-data starting from the data writing offset and having a data length equal to the data writing length from at least R storage files corresponding to the identification information of the target logical object; Processing each of the sub-data using an erasure code algorithm to obtain the data to be read; The R storage files store R sub-data correspondingly, and the R sub-data include k+mr sub-data.

8. The method according to claim 1, characterized in that The method further comprises: Acquire identification information of the data to be deleted from the terminal device; Inquiring the identification information of the to-be-deleted data in the mapping relationship stored in each metadata, obtaining the identification information, data writing offset and data writing length of the corresponding target logical object; The data write offset corresponding to the data to be deleted is deleted, or the data write length corresponding to the data to be deleted is set to 0.

9. A data processing device, characterized in that: include: An acquisition module, used to acquire target data sent by a terminal device; A processing module, configured to perform data sharding on the target data according to a preset erasure code redundancy ratio to obtain W sub-data; the W sub-data include k original sub-data and m check sub-data, and the data length of each sub-data is the same; The processing module is further used to write each sub-data into a plurality of storage files corresponding to the target logical object; the number of the storage files is the same as the number of the sub-data; The processing module is also used to feed back write success information to the terminal device when at least N storage files are written successfully; the N storage files store N sub-data correspondingly, and the N sub-data include k original sub-data and r check sub-data, m≥r≥0.

10. A server, characterized in that: include: A processor and a memory communicatively connected to the processor; The memory stores computer-executable instructions; The processor is configured to implement the method according to any one of claims 1 to 8 when executing the computer-executable instructions.

11. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, which are used to implement the method according to any one of claims 1 to 8 when executed by a processor.

Images