Data processing method and apparatus

Abstract

This specification provides a data processing method and apparatus. The data processing method includes: receiving a marking operation instruction for a data file and storing marking operation information associated with the marking operation instruction in an operation log; updating a data operation table associated with the data file based on the marking operation information recorded in the operation log when the operation log meets log information processing conditions; generating a data operation request for a corresponding target data node based on the updated data operation table and sending the data operation request to the target data node. By updating the data operation table according to the operation log, determining the target data node based on the data operation table, and performing the marking operation at the target data node, collaborative processing of data files based on marking operation instructions is achieved.

Description

Technical Field

[0001] The embodiments in this specification relate to the field of computer technology, and in particular to a data processing method. Background Technology

[0002] With the continuous development of internet technology, distributed storage systems are now widely used to achieve reliable massive data storage. Distributed storage systems extensively employ a three-replica mechanism, such as key-value distributed storage systems, to store data in different locations to improve data storage reliability. Current distributed storage systems consist of a three-layer architecture: a bottom-layer distributed file system, a middle-layer key-value storage engine, and an upper-layer computing engine layer. In this structure, to ensure data reliability, data deletion and modification are performed using an append-only approach. Therefore, in multi-layered distributed storage systems, both the middle key-value storage engine layer and the bottom single-machine storage engine layer use this method to process data. Subsequent memory release requires garbage collection at each layer, leading to greater write amplification, increased garbage data, and ultimately impacting overall system performance.

[0003] Therefore, how to reduce write amplification, reduce garbage data, lower storage costs, and improve system performance during garbage collection in distributed storage systems are urgent problems that need to be solved. Summary of the Invention

[0004] In view of the above, embodiments of this specification provide a data processing method. One or more embodiments of this specification also relate to a data processing apparatus, a data processing system, a computing device, a computer-readable storage medium, and a computer program, to address the technical deficiencies existing in the prior art.

[0005] According to a first aspect of the embodiments of this specification, a data processing method is provided, comprising:

[0006] Receive marking operation instructions for data files, and store the marking operation information associated with the marking operation instructions in the operation log;

[0007] If the operation log meets the log information processing conditions, update the data operation table associated with the data file according to the marked operation information recorded in the operation log;

[0008] Based on the updated data operation table, a data operation request is generated for the corresponding target data node, and the data operation request is sent to the target data node.

[0009] According to a second aspect of the embodiments of this specification, a data processing apparatus is provided, comprising:

[0010] The receiving module is configured to receive marking operation instructions for data files and store the marking operation information associated with the marking operation instructions in the operation log;

[0011] The update module is configured to update the data operation table associated with the data file according to the marked operation information recorded in the operation log when the operation log meets the log information processing conditions;

[0012] The sending module is configured to generate a data operation request for the corresponding target data node based on the updated data operation table, and send the data operation request to the target data node.

[0013] According to a third aspect of the embodiments of this specification, a data processing system is provided, comprising:

[0014] The server receives marking operation instructions for data files and stores the marking operation information associated with the marking operation instructions in the operation log; when the operation log meets the log information processing conditions, it updates the data operation table associated with the data file according to the marking operation information recorded in the operation log; it generates a data operation request for the corresponding target data node based on the updated data operation table and sends the data operation request to the target data node.

[0015] The target data node is used to mark the data range to be operated on in the local storage space based on the data operation request, and generate status update information based on the marking result and send it to the server.

[0016] The server is also used to receive the operation status information and record it in the updated data operation table.

[0017] According to a fourth aspect of the embodiments of this specification, a computing device is provided, comprising:

[0018] Memory and processor;

[0019] The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions, which, when executed by the processor, implement the steps of the above-described data processing method.

[0020] According to a fifth aspect of the embodiments of this specification, a computer-readable storage medium is provided that stores computer-executable instructions, which, when executed by a processor, implement the steps of the data processing method described above.

[0021] According to a sixth aspect of the embodiments of this specification, a computer program is provided, wherein when the computer program is executed in a computer, it causes the computer to perform the steps of the above-described data processing method.

[0022] One embodiment of this specification receives a marking operation instruction for a data file and stores the marking operation information associated with the instruction in an operation log. When the operation log meets the log information processing conditions, the data operation table associated with the data file is updated according to the marking operation information recorded in the operation log. Based on the updated data operation table, a data operation request for the corresponding target data node is generated and sent to the target data node. By updating the data operation table according to the operation log, determining the target data node based on the data operation table, and performing the marking operation at the target data node, collaborative processing of data files based on marking operation instructions is achieved. Attached Figure Description

[0023] Figure 1 This is a schematic diagram of the structure of a data processing method provided in one embodiment of this specification;

[0024] Figure 2-1 This is a flowchart illustrating a data processing method provided in one embodiment of this specification;

[0025] Figure 2-2 This is a schematic diagram of a data processing method provided in one embodiment of this specification;

[0026] Figure 3 This is a flowchart illustrating the processing procedure of a data processing method provided in one embodiment of this specification.

[0027] Figure 4 This is a schematic diagram of the structure of a data processing apparatus provided in one embodiment of this specification;

[0028] Figure 5 This is a schematic diagram of the structure of a data processing system provided in one embodiment of this specification;

[0029] Figure 6 This is a structural block diagram of a computing device provided in one embodiment of this specification. Detailed Implementation

[0030] Many specific details are set forth in the following description to provide a full understanding of this specification. However, this specification can be implemented in many other ways than those described herein, and those skilled in the art can make similar extensions without departing from the spirit of this specification. Therefore, this specification is not limited to the specific implementations disclosed below.

[0031] The terminology used in one or more embodiments of this specification is for the purpose of describing particular embodiments only and is not intended to be limiting of the one or more embodiments of this specification. The singular forms “a,” “described,” and “the” as used in one or more embodiments of this specification and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used in one or more embodiments of this specification refers to and includes any or all possible combinations of one or more associated listed items.

[0032] It should be understood that although the terms first, second, etc., may be used to describe various information in one or more embodiments of this specification, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, first may also be referred to as second without departing from the scope of one or more embodiments of this specification, and similarly, second may also be referred to as first. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to a determination."

[0033] First, the terms and concepts used in one or more embodiments of this specification will be explained.

[0034] Append-only: Only supports append-only writes; does not support random modification writes.

[0035] File: A contiguous block of stored data, uniquely identified by its filename or file ID. Each file consists of one or more chunks.

[0036] Chunk: A Chunk is a continuous range of data stored in a file. Each Chunk can be a multi-copy Chunk or an erasure-coded Chunk.

[0037] Replica: A replica is a contiguous block of data stored in a chunk on a single-machine storage engine. A multi-replica chunk is identical to every other replica; an erasure-coded chunk stores data shards that satisfy erasure coding slicing and arrangement rules, and each shard contains different data.

[0038] KV storage engine: refers to an engine built on top of a distributed file system that stores data in a key-value manner and provides access to data in a key manner.

[0039] ZNS-SSD (Zoned Namespace) - (Solid State Disk) is developed based on SSD. It realizes the migration of FTL (Flash Translation Layer) from inside the SSD to the upper host layer, opening up the inside of the SSD to the host, so that users can flexibly have their own specific FTL according to their needs.

[0040] This specification provides an information processing method, and also relates to an information processing apparatus, a computing device, an information processing system, a computer-readable storage medium, and a computer program, which will be described in detail in the following embodiments.

[0041] Currently, a three-tier architecture is widely used in various distributed storage systems such as object storage, file storage, table storage, and block storage, as well as database systems such as distributed analytical databases, distributed transactional databases, and hybrid analytical-transactional databases. The bottom layer uses an append-only distributed file system for highly reliable and available persistent storage. The middle layer uses various key-value (KV) storage engines as row-oriented, column-oriented, or hybrid engines. The upper layer is the front-end access layer or computation engine layer. In this architecture, data deletion, modification, and updates are all implemented by writing logs in an append-only manner to fully utilize the access characteristics of the storage medium and improve write throughput. Then, a background garbage collection (GC) process releases the physical space of overwritten or deleted data. Therefore, in this architecture, KV layer GC optimization is a key technology and a direction for continuous optimization and breakthroughs by relevant technical personnel.

[0042] As more and more underlying distributed file systems support append-only storage media such as ZNS-SSD, the single-machine storage layer of the distributed file system also adopts a key-value storage structure. Data deletion, modification, and updates are implemented by writing logs to append-only, relying on background garbage collection (GC) to truly release the physical space of overwritten or deleted data. Therefore, the independent GC approach of the intermediate key-value storage engine layer and the underlying distributed file system layer, lacking coordination, will lead to greater write amplification, increased garbage data, and thus affect the overall system performance, cost, and reduce the system's competitiveness.

[0043] In view of this, see Figure 1The diagram illustrates the structure. In a distributed file system, when deleting data in a file, the data to be deleted can be marked. When writing data to the data file, the data to be written can be written to the storage location of the marked data to be deleted. The operation of marking the data to be deleted is called the trim operation. The data file includes multiple consecutive data intervals such as chunk1, chunk2, and chunk3, and each consecutive data interval includes multiple data blocks (replicas). When performing a trim operation on the data stored in the data file, at least one consecutive data interval can be determined in the data file based on the trim operation request. The data interval associated with the trim operation, the seqId, and other operation information are recorded in the operation log, where the seqId is an auto-incrementing ID in the operation log. Each log entry written to the operation log is assigned a seqId. When executing the trim operation, the log information recorded in the operation log is read, and the read log information is converted into a marked operation information in the data operation table. The data operation table is then updated, that is, the offset, length, and other marked operation information corresponding to the chunkId are written to the data operation table. Based on a marker operation information stored in the data operation table, the corresponding data node in the distributed file system can be determined, and the data operation request can be sent to the data node, which will then perform the trim operation.

[0044] In the append-only distributed file system, a new Trim interface supporting strong data consistency has been added. The Trim interface includes key information such as the unique identifier of the data file (fileId), the offset within the file, the length of the Trim, and other interface parameters. For data that is no longer needed, the distributed file system's Trim interface is called. If the Trim call returns successfully, when the application tries to read the Trimed data, it returns 0, indicating that the data has been trimmed and cannot be read. If the Trim call returns unsuccessfully, it means the data has not been trimmed, and when reading the Trimed data, the original data before trimming is returned. Furthermore, a log file has been added for the Trim operation. Successful Trim operations are recorded in the log file; otherwise, a Trim call failure is returned.

[0045] See Figure 2-1 , Figure 2-1 A flowchart of a data processing method according to an embodiment of this specification is shown. The method is applied to a server and specifically includes the following steps.

[0046] Step S202: Receive a marking operation instruction for the data file and store the marking operation information associated with the marking operation instruction in the operation log.

[0047] Specifically, a data file is a file type in a computer system that can be used to store various types of data. In a distributed file system, a data file has a file identifier and consists of one or more data ranges, each of which includes multiple data blocks. Tagging instructions are computer instructions used to tag data within a data file. These instructions typically carry tagging information, including information about the data range to be tagged and the sequence number (ID) of the tagging information stored in the operation log. The operation log records this tagging information.

[0048] Therefore, in a distributed file system, marking data files can be performed by receiving a marking operation instruction for the data file, determining the marking operation information associated with the marking operation instruction based on the data file information carried in the marking operation instruction, and storing the marking operation information in the operation log associated with the data file, so that subsequent marking operation processing of the data file can be carried out based on the marking operation information stored in the operation log.

[0049] For example, in the scenario of deleting user A's order information 'a' stored in a distributed file system, since user A's order information 'a' may be stored in multiple replicas within the data files contained in the distributed file system, during data marking operations, the data contained in multiple replicas can be marked. That is, order information 'a' will be split into three data components: data 1, data 2, and data 3, and stored in three different consecutive data intervals in the data file. Therefore, upon receiving a deletion instruction for user A's order information 'a', the relevant data interval information for user A's order information 'a' can be stored in the operation log, awaiting the deletion processing of user A's order information 'a'.

[0050] Furthermore, when storing the marking operation information in the operation log, the marking operation information can be stored in an operation log that has a unique correspondence with the data file, or it can be stored in an operation log that stores the marking operation information corresponding to multiple data files. The specific implementation is as follows:

[0051] The marking operation information associated with the marking operation instruction is stored in the operation log associated with the data file, wherein the operation log is created based on the historical operation instructions of the data file; or, the target operation log associated with the data file is used as the operation log, and the marking operation information associated with the marking operation instruction is stored in the operation log, wherein the target operation log is used to store marking operation information associated with at least two data files.

[0052] Specifically, historical operation instructions refer to computer instructions used to store operation information in the operation log before the marked operation information is stored in the operation log; the target operation log refers to the operation log that is associated with at least two data files, that is, the marked operation information corresponding to the operation instructions submitted based on each of the at least two data files is stored in the target operation log.

[0053] Based on this, when storing marking operation information in the operation log, the marking operation information can be stored in an operation log that has a unique correspondence with the data file, or it can be stored in an operation log that stores marking operation information corresponding to multiple data files. Upon receiving a marking operation instruction, the marking operation information associated with the instruction can be stored in the operation log associated with the data file. The operation log can be created based on historical operation instructions for the data file. If no operation log is associated with a data file, it can be created and the marking operation information stored there. Alternatively, the target operation log associated with the data file can be used as the operation log, and the marking operation information associated with the instruction can be stored there. The target operation log is used to store marking operation information associated with at least two data files.

[0054] In practical implementation, when storing the tagging operation information corresponding to a data file in the operation log, a unique operation log can be identified for each data file, and the data file can be stored in that specific operation log. If no operation log exists for a data file, an operation log can be created for that data file, and the tagging operation information corresponding to the data file can be stored in the created operation log. Alternatively, multiple data files can share a single operation log, using a many-to-one approach, storing the tagging operation information corresponding to multiple data files in one operation log. This improves the utilization rate of the operation log.

[0055] Continuing with the previous example, when storing the data range information related to user A's order information 'a' in the operation log, different operation logs can be written based on the different data files corresponding to user A's order information 'a'. In practical applications, the log information corresponding to the deletion command submitted for data file A can be stored in operation log A, which is uniquely associated with data file A. Alternatively, the log information corresponding to the deletion command submitted for data file A can be stored in operation log B. Operation log B is used to store log information corresponding to multiple data files, such as data file A, data file B, and data file C.

[0056] In summary, when storing tagging operation information in the operation log, the tagging operation information can be stored in an operation log that has a unique correspondence with the data files, or it can be stored in an operation log that stores tagging operation information corresponding to multiple data files, thereby improving the flexibility of tagging operation information storage.

[0057] Step S204: If the operation log meets the log information processing conditions, update the data operation table associated with the data file according to the marked operation information recorded in the operation log.

[0058] Specifically, after storing the marking operation information associated with the marking operation instructions in the operation log, the marking operation information recorded in the operation log can be processed if the operation log meets the log information processing conditions. The log information processing conditions are pre-set conditions used to determine whether the marking operation information stored in the operation log can be processed. In this embodiment, the log information processing conditions include: determining that the operation log meets the log information processing conditions when the number of marking operation information stored in the operation log reaches a set threshold; and setting a preset log processing period, determining that the operation log meets the log information processing conditions when the preset log processing period is reached. The data operation table is used to store relevant information about the data to be processed corresponding to the data file, including but not limited to data range information, offset address information, length information, and processing status information.

[0059] Based on this, the operation log is judged from multiple dimensions such as the log processing cycle and the amount of information in the log to determine whether the operation log meets the log information processing conditions. If the operation log meets the log information processing conditions, the marked operation information stored in the operation log is read, and the data operation table associated with the data file is updated according to the marked operation information recorded in the operation log. The data associated with the marked operation information is recorded in the data operation table.

[0060] Using the previous example, when the number of log messages recorded in the operation log reaches the preset threshold of 20, it is determined that the operation log meets the log message processing conditions. The log messages are read from the operation log in the order of writing: delete the log message corresponding to order information a of user A: data range 1-data 1. Based on the "data range 1-data 1" recorded in the log message, a record is generated in the data operation table: data range 1-offset 1-length 1-state empty. This record is then stored in the data operation table corresponding to the data file.

[0061] Furthermore, after storing the marking operation information associated with the marking operation instructions in the operation log, the data operation table associated with the data file can be updated based on the marking operation information stored in the operation log. The specific implementation is as follows:

[0062] Read the marked operation information recorded in the operation log, and determine the data interval information associated with the data file based on the marked operation information, wherein the data interval information is associated with the data interval contained in the data file; determine the data block operation information based on the data interval information, and update the data operation table associated with the data file based on the data block operation information.

[0063] Specifically, data interval information refers to the interval information corresponding to the data interval contained in the data file. The data file contains at least one continuous data interval, and the marked operation information stored in the operation log contains data interval information. A continuous data interval corresponds to multiple data blocks. Correspondingly, data block operation information refers to the operation information associated with the data to be deleted, including but not limited to the offset address, length, and operation status of the data block.

[0064] Based on this, after storing the marking operation information associated with the marking operation instruction in the operation log, and provided that the operation log meets the log information processing conditions, the marking operation information recorded in the operation log is read. The data range information associated with the data file is determined based on the data range associated with the marking operation information. The data block operation information corresponding to the marking operation instruction is determined based on the data range information, and the data operation table associated with the data file is updated based on the data block operation information.

[0065] Following the previous example, after determining the log information recorded in the operation log: data range 1 - data 1, the storage location of data 1 in data range 1 is determined based on data range 1 and data 1. The relevant information of the data block corresponding to the log information is determined: data range 1 - offset 1 - length 1 - status empty, and this information is recorded in the data operation table.

[0066] In summary, by reading the marking operation information to determine the data range information associated with the marking operation information, and then updating the data operation table based on the data range information associated with the marking operation information, the collaborative operation of the operation log and the data operation table can be achieved.

[0067] Furthermore, after storing the marking operation information associated with the marking operation instruction in the operation log, and enabling the addition of new marking operation information to the operation log, the newly added marking operation information can be processed. It is also possible to query the log information contained in the operation log according to a preset log query period and process the newly added marking operation information found in the query. The specific implementation is as follows:

[0068] If new marked operation information is added to the operation log, the operation log is determined to meet the log information processing conditions, and the step of updating the data operation table associated with the data file according to the marked operation information recorded in the operation log is executed; or, the log information contained in the operation log is queried according to a preset log query cycle, and if the marked operation information is determined to be new marked operation information according to the query result, the operation log is determined to meet the log information processing conditions, and the step of updating the data operation table associated with the data file according to the marked operation information recorded in the operation log is executed.

[0069] Specifically, the preset log query cycle refers to the pre-set query cycle for operation logs. The operation logs are periodically queried according to the preset log query cycle to determine the marked operation information in the operation logs that needs to be processed, and then the marked operation information is processed.

[0070] Based on this, after storing the marking operation information associated with the marking operation instruction in the operation log, it is determined that new marking operation information has been added to the operation log. At this time, it is determined that the operation log meets the log information processing conditions, and the step of updating the data operation table associated with the data file according to the marking operation information recorded in the operation log is executed; or, after storing the marking operation information associated with the marking operation instruction in the operation log, the log information contained in the operation log is periodically queried according to a preset log query cycle. If the marking operation information is determined to be new marking operation information based on the query results, it is determined that the operation log meets the log information processing conditions, and the step of updating the data operation table associated with the data file according to the marking operation information recorded in the operation log is executed.

[0071] Continuing with the previous example, the operation log corresponding to data file A records log information such as data range 1-data1, data range 2-data2, and data range 3-data3 corresponding to the order information a of user A to be deleted. In addition, the processing status of each log information in the operation log is also recorded. When the log information data range 1-data1 is processed, it will be marked as "completed". Therefore, when processing the log information in the operation log, a new log information: data range 3-data3 can be added to the operation log and processed. Alternatively, a query period of 10 minutes can be set for the log, and the operation log can be queried once every 10 minutes. When a new log information: data range 1-data4 is found in the operation log, this log information can be processed.

[0072] In summary, adding marker operation information to the operation log and periodically querying the operation log according to a preset log query cycle can both confirm that the operation log meets the log information processing conditions when the query results are obtained. This allows the data operation table associated with the data file to be updated based on the marker operation information recorded in the operation log, thereby improving the flexibility of processing the operation log.

[0073] Step S206: Generate a data operation request for the corresponding target data node based on the updated data operation table, and send the data operation request to the target data node.

[0074] Specifically, after updating the data operation table associated with the data file based on the marking operation information recorded in the operation log, a data operation request for the corresponding target data node can be generated based on the information recorded in the data operation table. The data operation request is then sent to the target data node, where the target data node is the node used to store the data in the data file. In this embodiment, the target data node can be a single-machine storage engine, and the data marking operation is performed by the target data node. The data operation request is a request submitted to the target data node to inform the target data node to perform the marking operation on the data.

[0075] Based on this, after updating the data operation table associated with the data file according to the marking operation information recorded in the operation log, a data operation request for the corresponding target data node is generated based on the updated data operation table, and the data operation request is sent to the target data node. After receiving the data operation request, the target data node performs marking operations on the data stored in the data node.

[0076] Continuing with the previous example, after updating the data operation table by recording the relevant information of the data block corresponding to the log information: data range 1 - offset 1 - length 1 - empty status, record 1 stored in the data operation table is read: data range 1 - offset 1 - length 1 - empty status. Based on this record, a data deletion request is generated and sent to multiple data nodes in the distributed file system corresponding to data range 1, including the primary data node and multiple replica data nodes. The data nodes then perform the deletion operation on the data block corresponding to record 1. It should be noted that the deletion operation on the data block only marks the data, rather than deleting the data from the data block. Marking the data indicates that when other data is stored, it can be stored in the storage location corresponding to the marked data.

[0077] Furthermore, after the data operation table is updated, a data operation request can be generated based on the data block operation information stored in the data operation table, and the data operation request can be sent to the target data node corresponding to the data block. The target data node then performs the marking operation, as specifically implemented below:

[0078] Read the data block operation information from the updated data table; generate a data operation request corresponding to the target data node based on the physical address information contained in the data block operation information, wherein the data operation request is associated with the data block of the data file, and the data block is stored in the target data node.

[0079] Specifically, data block operation information refers to the descriptive information recorded in the data operation table used to process the data to be deleted. This includes data range information, as well as the offset address, offset length, and processing status information associated with the data range information. Among these, the offset address information and offset length are the physical address information, used to determine the data block and the data to be deleted in the target data node.

[0080] Based on this, after updating the data operation table according to the log information recorded in the operation log, the recorded data block operation information can be read from the updated data table. Using the physical address information contained in the read data block operation information, the target data node is determined among the data nodes in the distributed file system. A corresponding data operation request is generated for the target data node, and this request is associated with a data block in the data file. The data block is stored on the target data node. Subsequently, the data operation request is sent to the target data node, which then performs marking operations on the data stored in the data block.

[0081] Continuing with the previous example, record 1 stored in the data operation table is read: Data Range 1 - Offset 1 - Length 1 - State Empty. Based on the physical address information Offset 1 - Length 1, a data deletion request is generated for the data block in data node 1. This request is sent to the primary data node 10, and replica data nodes 21 and 22, which correspond to user A's order information 'a', within the distributed file system. The data deletion request is then sent to each data node, and each data node executes the data deletion operation, thus marking the data stored in the data block within that data node.

[0082] In summary, by reading the physical address information contained in the data block operation information, generating a data operation request for the corresponding target data node, and then sending the data operation request to the target data node, the accuracy of target data node determination is improved.

[0083] Furthermore, after sending the data operation request to the target data node, the target data node can execute the data operation request and obtain the execution result. Considering that the target data node may fail to execute the data operation request, in order to record the data operation status in real time, operation status information can be generated based on the execution result and recorded in the data operation table. The specific implementation is as follows:

[0084] Based on the execution result of the data operation request executed by the target data node, the operation status information is determined and recorded in the updated data operation table.

[0085] Specifically, after the data operation request is sent to the target data node, the target data node can execute the data operation request and obtain the execution result. The execution result is then recorded in the data operation table. The operation status information refers to the execution status corresponding to the marked operation information stored in the data operation table, including execution success and execution failure. If the target data node executes the marked operation successfully, the status information corresponding to the marked operation information in the data operation table is recorded as execution success; if the target data node executes the marked operation unsuccessfully, the status information corresponding to the marked operation information in the data operation table is recorded as execution failure.

[0086] Based on this, after the target data node receives the data operation request, it executes the data operation request, obtains the execution result corresponding to the data operation request, determines the operation status information for updating the data operation table based on the execution result, and records the operation status information into the updated data operation table, thereby realizing the update of the operation status corresponding to the marked operation information in the data operation table.

[0087] Following the previous example, after the primary data node 10 and replica data nodes 21 and 22 have all completed the data marking operation and the operation is successful, record 1 of the data operation table is updated. That is, the record 1 stored in the data operation table, "Data Range 1-Offset 1-Length 1-State Empty", is updated to "Data Range 1-Offset 1-Length 1-Marking Successful". If the operation fails, record 1 of the data operation table is updated. That is, the record 1 stored in the data operation table, "Data Range 1-Offset 1-Length 1-State Empty", is updated to "Data Range 1-Offset 1-Length 1-Marking Failed".

[0088] In summary, the target data node executes the data operation request to complete the data operation. The execution result is recorded in the data operation table in the form of operation status information. This allows the data operation request to be regenerated based on the data operation table if the execution of the data operation request fails. Thus, retrying after failure ensures that the data can be operated successfully.

[0089] Furthermore, when the target data node executes a data marking operation request, or after the data marking operation is completed, the data stored in the data file can be read. Upon receiving a data read request for the data file, the data operation table is queried according to the data read request. Based on the query results, the feedback result corresponding to the data read request can be determined. The specific implementation is as follows:

[0090] Receive a data read request for the data file; query the data operation table based on the data read request; determine the feedback result corresponding to the data read request based on the query result.

[0091] Specifically, a data read request refers to a computer instruction used to read data stored in a data file; the feedback result refers to the data read result corresponding to the data read request. The data to be read may be in an unreadable or readable state. If the data to be read has completed a marking operation, is in the process of a marking operation, or the marking operation has failed, the data to be read is in an unreadable state and cannot be read. If the data to be read has not undergone a marking operation, it means that there is no log information in the operation log associated with the data to be read. At this time, the data to be read is in a readable state, and the feedback result is the data that has been read.

[0092] Based on this, when marking a data file, data can also be read from the data file. When a data read request for a data file is received, the data operation table is queried based on the data read request, and the feedback result corresponding to the data read request is determined according to the operation status information in the data operation table.

[0093] Using the previous example, when querying user B's order information b, a data read request is submitted for the data file A storing the order information b. The data operation table A associated with data file A is queried, and the read result is determined based on the record information of the data to be read in the data operation table A.

[0094] In summary, when reading data stored in the data file, the feedback result is determined based on the status information recorded in the data operation table, thereby providing the data corresponding to the data reading request and ensuring the timeliness of the response to the data reading request.

[0095] Furthermore, when reading data stored in a data file based on a data read request, different data read results will be returned for each data read request because the operation status of the marked operation information recorded in the data operation table is different. The specific implementation is as follows:

[0096] If, based on the query results, it is determined that the data operation table contains the marker operation information corresponding to the data read request, a prompt message is generated and used as the feedback result, wherein the prompt message indicates that the data to be read corresponding to the data read request is in an unreadable state; if, based on the query results, it is determined that the data operation table does not contain the marker operation information corresponding to the data read request, a node data read request is sent to the data node corresponding to the data read request, and the original data returned by the data node in response to the data read request is used as the feedback result.

[0097] Specifically, the prompt message refers to the information fed back based on the data read request, which is used to indicate to the data reader that the data to be read is currently in an unreadable state. The prompt message can be a text prompt or a number 0 to indicate that the data read has failed. The node data read request is used to read the data to be read stored in the data node. The original data refers to the data in the data node that did not successfully perform the marking operation.

[0098] Based on this, the data operation table is queried according to the data read request. If the marking operation information corresponding to the data read request is found in the data operation table, it indicates that the data to be read corresponding to the data read request is in a marking state, a marking completed state, or a marking start state. Even if the marking of the data to be read fails, it will still be re-marked until it is successfully marked, meaning that data to be read in any state will be successfully marked. Therefore, when a data read request is received, if the marking operation information corresponding to the data read request is stored in the data operation table, it can be determined that the data to be read is in an unreadable state. A prompt message is generated based on the unreadable data and used as the feedback result corresponding to the data read request. If the marking operation information corresponding to the data read request is not found in the data operation table, it indicates that the data to be read is in a readable state. A node data read request is sent to the data node corresponding to the data read request, and the original data returned by the data node in response to the data read request is used as the feedback result corresponding to the data read request.

[0099] Continuing with the previous example, when querying user B's order information b, a data read request is submitted to the data file A storing the order information b. The data operation table A associated with data file A is queried. If the data operation table A records the data corresponding to order information b, it means that the data corresponding to order information b has been deleted, is being deleted, or has started deleting. At this time, the data is in an unreadable state, and a message such as "read failed" or "data unreadable" is returned, or the number 0 is returned to indicate that the data read failed. If the data operation table A does not record order information b, it means that the data corresponding to order information b has not been deleted by the data node corresponding to order information b. At this time, the data is in a readable state, and the data node corresponding to the data returns the original data corresponding to order information b stored in the data node.

[0100] In summary, upon receiving a data read request, different feedback information is determined based on the different operation states corresponding to the marked operation information in the data operation table. When the data to be read is in an unreadable state, a prompt message is provided; when the original data stored in the data node fails to execute the marking operation, the original data is provided, thereby improving the timeliness of data feedback.

[0101] Furthermore, since a data file is associated with multiple data nodes, when determining the target data node, the first data node can be identified from the data nodes contained in the distributed file system based on the file identifier of the data file. Then, the second data node associated with the first data node can be identified from the data nodes contained in the distributed file system. The specific implementation is as follows:

[0102] The file identifier corresponding to the data file is determined, and a first data node is determined in the data nodes contained in the distributed file system based on the file identifier, and a second data node associated with the first data node is determined in the data nodes contained in the distributed file system; the first data node and the second data node are used as target data nodes.

[0103] Specifically, a file identifier refers to the identification information such as a data file ID used to uniquely identify a data file in a distributed file system; a first data node refers to at least one data node in the distributed file system that has an association with a data file. The first data node can be the primary data node associated with the data file, and correspondingly, the second data node can be the secondary data node associated with the data file. There is at least one second data node, and the first data node and the second data node constitute the target data node corresponding to the data file.

[0104] Based on this, when determining the target data node corresponding to the marker operation information stored in the data operation table, the file identifier corresponding to the data file is determined, and the first data node is determined in the data nodes contained in the distributed file system according to the file identifier, and the second data node associated with the first data node is determined in the data nodes contained in the distributed file system; the first data node and the second data node are used as the target data node.

[0105] Continuing with the previous example, in a distributed system, data files can be stored across multiple replicas. Based on the file identifier of the data file, the primary data node 10, and replica data nodes 21 and 22 corresponding to the data file can be identified. Primary data node 10, replica data nodes 21 and 22 are the target data nodes corresponding to the data file.

[0106] In summary, based on the file identifier corresponding to the data file, the first data node and the second data node are determined from the data nodes contained in the distributed file system, and the first data node and the second data node are used as the target data nodes corresponding to the data file.

[0107] Furthermore, upon receiving a request to read target data from a data file, the first data node associated with the data file is determined. If the first data node is unavailable, data is received from the second data node. The specific implementation is as follows:

[0108] Receive a target data read request; send the target data read request to the first data node; if the first data node is unavailable, receive second target data fed back by the second data node in response to the target data read request, wherein the second target data is the same as the first target data stored by the first data node.

[0109] Specifically, the first target data refers to the target data stored in the first data node corresponding to the target data read request, and the second target data refers to the target data stored in the second data node. The first target data and the second target data are the same, which are the same data stored in the first data node and the second data node respectively.

[0110] Based on this, upon receiving a target data read request for a data file, the target data read request is sent to the first data node corresponding to the data file. If the first data node is unavailable, the system receives second target data from the second data node in response to the target data read request. The second target data is identical to the first target data stored on the first data node.

[0111] In practice, when a data node malfunctions or crashes, the data operation table can fully record the data marking operations for each data range. Therefore, when executing a data replication task, it can be based on the data operation table, filtering out marking operation information in a failed execution state. Data corresponding to marking operation information in a successful execution state in the data operation table is replicated, while data corresponding to marking operation information in a failed execution state is not replicated.

[0112] Continuing with the previous example, in a distributed system, data files can be stored across multiple data replicas. Based on the file identifier, the primary data node 20, replica data nodes 21 and 22 corresponding to the data file can be identified. Primary data node 20, replica data nodes 21 and 22 are the target data nodes corresponding to the data file. When a query request for order information c for user C is received, the query request is sent to the primary data node 20, replica data nodes 21 and 22 corresponding to order information c. If primary data node 20 fails, the system receives the data corresponding to order information c from replica data nodes 21 or 22.

[0113] In summary, when reading data stored in a data file, the system prioritizes reading the first target data stored in the first data node. If the first data node is determined to be unavailable, the system receives the second target data fed back from the second data node. This improves the success rate of data reading and avoids data reading failures caused by data node unavailability.

[0114] Furthermore, after storing the marking operation information in the data operation table, since there are cases where data marking operations fail when performed based on the marking operation information in the data operation table, the marking operation information that failed during the execution of data marking operations can be determined by periodically querying the data operation table. The specific implementation is as follows:

[0115] The data operation table is queried based on a preset data operation table query cycle; the target physical address information is determined based on the query result, wherein the target physical address information is the physical address information corresponding to the target data block operation information in the data operation table whose operation status is operation failure; a data reoperation request is generated based on the target physical address information and sent to the data node corresponding to the target data block operation information; the execution result of the data node executing the data reoperation request is received, and the data operation table is updated based on the execution result.

[0116] Specifically, the preset data operation table query cycle refers to a pre-defined query period for the data operation table. The table can be queried at fixed intervals to identify marking operation information that has failed. Based on this information, the data marking operation can be re-executed. The target physical address information refers to the physical address information corresponding to the retrieved marking status information. The data re-operation request is a data operation request generated based on the retrieved marking operation information, used to re-execute the data marking operation based on the marking operation information that has failed.

[0117] Based on this, when querying the data operation table, a preset query period is established. The data operation table is queried according to this preset period to obtain query results. The target physical address information is determined based on the query results. This target physical address information corresponds to the physical address of the target data block operation information in the data operation table whose operation status is "operation failed." A data reoperation request is generated based on the target physical address information and sent to the data node corresponding to the target data block operation information. The execution result of the data node executing the data reoperation request is received, and the data operation table is updated based on the execution result.

[0118] In practice, when performing data marking operations, since data files correspond to multiple data copies and data nodes may fail, some data marking operations may succeed, some may fail, or all data may fail to be marked. In this scenario, the data operation table can be queried periodically to find the data segments that failed to be marked, and the marking operation can be re-performed on these segments until all data blocks corresponding to the data operation table have completed the marking operation.

[0119] Continuing with the previous example, when performing data deletion or data query operations on a data file, the data operation table is queried every five minutes according to a preset query cycle of 5 minutes. The query checks for a record in the data operation table whose operation status is "operation failed": "Data Range 1 - Offset 1 - Length 1 - Mark Failed". The physical address information "Offset 1 - Length 1" is determined, a retry request is generated, and the retry request is sent to the data node corresponding to the record "Data Range 1 - Offset 1 - Length 1 - Mark Failed". Then, after the data node performs the operation on the data to be processed, the record "Data Range 1 - Offset 1 - Length 1 - Mark Failed" in the data operation table is updated according to the operation result. If the operation is successful, "Mark Failed" is updated to "Mark Successful"; if the operation fails, the data operation table is not updated.

[0120] In summary, by querying the data operation table based on a preset data operation table query cycle, and then identifying the physical address information corresponding to the target data block operation information whose operation status is failed, a data reoperation request is generated based on the target physical address information and sent to the data node corresponding to the target data block operation information, thereby completing the data marking operation on the target data block and improving the success rate of the data marking operation.

[0121] Furthermore, if it is determined that the data marking operation corresponding to the log information stored in the operation log was executed successfully, considering the limited storage space of the operation log, the corresponding log information marking the successful execution of the operation can be deleted from the operation log. The specific implementation is as follows:

[0122] Delete the operation log information in the operation log that is in the execution completed state; or, if the target data operation table associated with the operation log has completed the persistence operation, delete the target operation log information associated with the target data operation table in the operation log.

[0123] Specifically, the target data operation table refers to the data operation table that has completed the persistence operation. In this embodiment, the persistence operation can be to record the marked operation information stored in the data operation table to achieve the goal of data backup, so that when the data node crashes, the execution status before the data node crashes can be determined according to the persistent data operation table.

[0124] Based on this, if there is operation log information in the operation log that is in the execution completed state, delete the operation log information in the operation log that is in the execution completed state; or, if the target data operation table associated with the operation log completes the persistence operation, delete the target operation log information associated with the target data operation table in the operation log.

[0125] In practice, the log information recorded in the operation log is arranged in ascending order. After the data operation table corresponding to the log information recorded in the operation log is persisted, the log information corresponding to the data operation table in the operation log can be deleted. After the data marking operation is completed in the log information contained in the operation log, the log information that has completed the marking operation can be deleted.

[0126] Continuing with the previous example, the operation log records log information 1, log information 2, log information 3, etc. Each log information has a corresponding execution status, indicating whether the log information has been completed. When log information 1 is in the completed execution status in the operation log, log information 1 is deleted from the operation log. Alternatively, the log information recorded in the operation log can be persisted, generating a checkpoint corresponding to the log information. That is, the currently existing log information in the operation log is recorded as a snapshot, generating the corresponding storage file. After confirming that log information 1 and log information 2 have been persisted in the operation log, log information 1 and log information 2 are deleted from the operation log.

[0127] like Figure 2-2 As shown, Figure 2-2 This diagram illustrates a data processing method according to one embodiment of this specification. After the client receives a user's marking operation on a file, the marking operation is persistently recorded in the operation log. When it is necessary to process the log information stored in the operation log, the log information is read from the operation log and converted into marking operations on the data blocks contained in the data replica corresponding to the data range. The marking operations on the data replica data blocks are sent to the corresponding data nodes, which then execute the marking operations on the data, thereby completing the marking operation on the file.

[0128] In summary, the embodiments of this specification provide a data processing method that receives marking operation instructions for data files and stores the marking operation information associated with the instructions in an operation log. When the operation log meets the log information processing conditions, the method updates the data operation table associated with the data file based on the marking operation information recorded in the operation log. Based on the updated data operation table, a data operation request for the corresponding target data node is generated and sent to the target data node. By updating the data operation table according to the operation log, determining the target data node based on the data operation table, and performing the marking operation at the target data node, collaborative processing of data files based on marking operation instructions is achieved.

[0129] The following is in conjunction with the appendix Figure 3 Taking the application of the data processing method provided in this specification in a distributed file system as an example, the data processing method will be further explained. Figure 3 A flowchart illustrating the processing procedure of a data processing method according to an embodiment of this specification is shown, specifically including the following steps.

[0130] Step S302: Receive a marking operation instruction for the data file.

[0131] In scenarios involving the deletion of user personal data, a data deletion command is received for a data file containing the user's personal data.

[0132] Step S304: Store the marking operation information associated with the marking operation instruction in the operation log.

[0133] The data deletion instruction determines the data range information corresponding to the user's personal data, carried within the instruction. To improve data storage security and reliability, the user's personal data is stored using a distributed file storage system. Therefore, multiple copies of the user's personal data exist, stored in multiple data blocks corresponding to data files. These data blocks are managed by data nodes, with one data node capable of managing multiple data blocks. The data range information corresponding to the user's personal data is stored in the operation log corresponding to the data file.

[0134] Step S306: If the operation log meets the log information processing conditions, determine the data range information associated with the data file based on the marked operation information.

[0135] When the number of log messages stored in the operation log reaches the set threshold, or when new log messages are added to the operation log, it can be confirmed that the operation log meets the log message processing conditions. Then, the data range for storing user personal data can be determined based on the log messages stored in the operation log.

[0136] Step S308: Determine the data block operation information based on the data range information, and update the data operation table associated with the data file based on the data block operation information.

[0137] Read the log information stored in the operation log, and complete the recording of the data operation table according to the data range information corresponding to the log information. Record the data block ID, offset, length and processing status contained in the data range information in the data operation table. When no data in the data operation table is processed, the processing status is empty.

[0138] Step S310: Generate a data operation request for the corresponding target data node based on the updated data operation table.

[0139] Read the data range ID, offset, length, and other marker operation information stored in the data operation table, and generate a data operation request for the user's personal data based on the marker operation information.

[0140] Step S312: Send the data operation request to the target data node.

[0141] The data operation request is sent to the data node corresponding to the data block that stores the user's personal data in the data range, and the data node performs the marking operation on the user's personal data.

[0142] Step S314: Determine the operation status information based on the execution result of the data operation request executed by the target data node.

[0143] The designated data nodes perform a marking operation on the user's personal data stored in the data nodes. If the marking operation is successful, the execution result is determined to be successful; if the marking operation fails, the execution result is determined to be unsuccessful.

[0144] Step S316: Record the operation status information into the updated data operation table.

[0145] If the execution result is successful, update the status information in the corresponding flag operation information in the data operation table to "execution successful"; if the flag operation fails, update the status information in the corresponding flag operation information in the data operation table to "execution failed".

[0146] Step S318: Receive a data read request for the data file.

[0147] When reading user personal data, receive data read requests for data files that store user personal data.

[0148] Step S320: Query the data operation table according to the data read request.

[0149] Step S322: If the data operation table contains the marker operation information corresponding to the data read request, generate the prompt information corresponding to the data read request, and use the prompt information as the feedback result corresponding to the data read request.

[0150] When the data operation table contains the marking operation information corresponding to the data read request, it means that the marking operation on the user's personal data in the data file corresponding to this marking operation information has been recorded in the operation log in the form of log information. Based on this, it can be considered that the marking operation corresponding to the data has been successful. When a data read request is received, since the data operation table contains the marking operation information corresponding to the data read request, it can be determined that the user's personal data to be read is in an unreadable state. Therefore, a prompt message such as "read failed" or "0" indicating that it cannot be read is returned.

[0151] Step S324: If the data operation table does not contain the marker operation information corresponding to the data read request, send a node data read request to the data node corresponding to the data read request, and use the original data fed back by the data node in response to the data read request as the feedback result.

[0152] If the data operation table does not contain the marking operation information corresponding to the data read request, it means that the operation log does not record the log information corresponding to the marking operation information. That is, the user personal data to be read corresponding to the data read request has not been successfully marked and is in a readable state. Therefore, a data read request can be sent to the data node that stores the user personal data to be read, and the data node will return the user personal data stored in the data node.

[0153] Step S326: Query the data operation table based on the preset data operation table query cycle.

[0154] Since the data operation table contains flagged operations that are in a failed state, it is necessary to re-flag these operations. This can be done by querying the data operation table to identify the flagged operations that are in a failed state.

[0155] Step S328: Generate a data reoperation request based on the query results and send it to the data node corresponding to the operation information of the target data block.

[0156] When a marking operation information in a marked failure state is found in the data operation table, a data reoperation request is generated based on this marking operation information and sent to the data node.

[0157] Step S330: Receive the execution result of the data node's data reoperation request, and update the data operation table based on the execution result.

[0158] When a data node receives a data re-operation request, it re-marks the user's personal data stored in the data node. Based on the marked results, it updates the marked operation information in the data operation table. That is, if the operation is successful, the status information in the marked operation information in the data operation table is updated to "operation successful"; if the operation fails, the data operation table is not processed. This continues until all operation statuses corresponding to marked operation information in the data operation table are in the "execution successful" state.

[0159] In summary, by receiving marking operation instructions for data files and storing the marking operation information associated with these instructions in an operation log; when the operation log meets the log information processing conditions, updating the data operation table associated with the data file based on the marking operation information recorded in the operation log; generating a data operation request for the corresponding target data node based on the updated data operation table, and sending the data operation request to the target data node; updating the data operation table based on the operation log, determining the target data node based on the data operation table, and performing the marking operation at the target data node, collaborative processing of data files based on marking operation instructions is achieved.

[0160] Corresponding to the above method embodiments, this specification also provides data processing apparatus embodiments. Figure 4 A schematic diagram of the structure of a data processing apparatus according to one embodiment of this specification is shown. Figure 4 As shown, the device includes:

[0161] The receiving module 402 is configured to receive a marking operation instruction for a data file and store the marking operation information associated with the marking operation instruction in the operation log;

[0162] Update module 404 is configured to update the data operation table associated with the data file according to the marked operation information recorded in the operation log when the operation log meets the log information processing conditions;

[0163] The sending module 406 is configured to generate a data operation request for the corresponding target data node based on the updated data operation table, and send the data operation request to the target data node.

[0164] In an optional embodiment, the update module 404 is further configured to:

[0165] Read the marking operation information recorded in the operation log, and determine the data interval information associated with the data file based on the marking operation information, wherein the data interval information is associated with the data interval contained in the data file;

[0166] The data block operation information is determined based on the data range information, and the data operation table associated with the data file is updated based on the data block operation information.

[0167] In an optional embodiment, the sending module 406 is further configured to:

[0168] Read the data block operation information from the updated data table;

[0169] Based on the physical address information contained in the data block operation information, a data operation request corresponding to the target data node is generated, wherein the data operation request is associated with a data block of the data file, and the data block is stored in the target data node.

[0170] In an optional embodiment, the receiving module 402 is further configured to:

[0171] The marking operation information associated with the marking operation instruction is stored in the operation log associated with the data file, wherein the operation log is created based on the historical operation instructions of the data file; or...

[0172] The target operation log associated with the data file is used as the operation log, and the marking operation information associated with the marking operation instruction is stored in the operation log. The target operation log is used to store marking operation information associated with at least two data files.

[0173] In an optional embodiment, the update module 404 is further configured to:

[0174] If new flagged operation information is added to the operation log, it is determined that the operation log meets the log information processing conditions, and the step of updating the data operation table associated with the data file according to the flagged operation information recorded in the operation log is executed; or,

[0175] The operation log contains log information according to a preset log query cycle. If the query result determines that the marked operation information is newly added marked operation information, the operation log meets the log information processing conditions, and the step of updating the data operation table associated with the data file according to the marked operation information recorded in the operation log is executed.

[0176] In an optional embodiment, the sending module 406 is further configured to:

[0177] Based on the execution result of the data operation request executed by the target data node, the operation status information is determined and recorded in the updated data operation table.

[0178] In an optional embodiment, the recording module 406 is further configured to:

[0179] Receive a data read request for the data file;

[0180] Based on the data read request, query the data operation table;

[0181] The feedback result corresponding to the data reading request is determined based on the query results.

[0182] In an optional embodiment, the recording module 406 is further configured to:

[0183] If, based on the query results, it is determined that the data operation table contains marker operation information corresponding to the data read request, a prompt message is generated and the prompt message is used as the feedback result, wherein the prompt message indicates that the data to be read corresponding to the data read request is in an unreadable state;

[0184] If, based on the query results, it is determined that the data operation table does not contain the tag operation information corresponding to the data read request, a node data read request is sent to the data node corresponding to the data read request, and the original data returned by the data node in response to the data read request is used as the feedback result.

[0185] In an optional embodiment, the sending module 406 is further configured to:

[0186] Determine the file identifier corresponding to the data file, determine the first data node in the data nodes contained in the distributed file system based on the file identifier, and determine the second data node associated with the first data node in the data nodes contained in the distributed file system;

[0187] The first data node and the second data node are used as target data nodes.

[0188] In an optional embodiment, the recording module 406 is further configured to:

[0189] Receive the target data read request;

[0190] Send the target data read request to the first data node;

[0191] If the first data node is unavailable, the system receives second target data from the second data node in response to the target data read request, wherein the second target data is the same as the first target data stored by the first data node.

[0192] In an optional embodiment, the recording module 406 is further configured to:

[0193] The data operation table is queried based on a preset data operation table query cycle;

[0194] The target physical address information is determined based on the query results. The target physical address information is the physical address information corresponding to the target data block operation information whose operation status is operation failure in the data operation table.

[0195] Based on the target physical address information, a data reoperation request is generated and sent to the data node corresponding to the target data block operation information;

[0196] Receive the execution result of the data node executing the data reoperation request, and update the data operation table based on the execution result.

[0197] In an optional embodiment, the recording module 406 is further configured to:

[0198] Delete the operation log information that is in the execution completed state from the operation log; or...

[0199] If the target data operation table associated with the operation log completes the persistence operation, the target operation log information associated with the target data operation table will be deleted from the operation log.

[0200] This specification provides a data processing apparatus that receives marking operation instructions for data files and stores the marking operation information associated with the instructions in an operation log. When the operation log meets the log information processing conditions, the apparatus updates the data operation table associated with the data file based on the marking operation information recorded in the operation log. Based on the updated data operation table, a data operation request for a corresponding target data node is generated and sent to the target data node. By updating the data operation table according to the operation log, determining the target data node based on the data operation table, and performing the marking operation at the target data node, collaborative processing of data files based on marking operation instructions is achieved.

[0201] The above is an illustrative scheme of a data processing apparatus according to this embodiment. It should be noted that the technical solution of this data processing apparatus and the technical solution of the data processing method described above belong to the same concept. For details not described in detail in the technical solution of the data processing apparatus, please refer to the description of the technical solution of the data processing method described above.

[0202] Corresponding to the above method embodiments, this specification also provides data processing system embodiments. Figure 5 A schematic diagram of the structure of a data processing system according to one embodiment of this specification is shown. Figure 5As shown, the data processing system 500 includes: a server 510 and a target data node 520;

[0203] Server 510 is used to receive marking operation instructions for data files and store the marking operation information associated with the marking operation instructions in the operation log; when the operation log meets the log information processing conditions, it updates the data operation table associated with the data file according to the marking operation information recorded in the operation log; it generates a data operation request for the corresponding target data node 520 based on the updated data operation table and sends the data operation request to the target data node 520.

[0204] The target data node 520 is used to mark the data range to be operated on in the local storage space based on the data operation request, and generate status update information based on the marking result and send it to the server 510.

[0205] The server 510 is also used to receive the operation status information and record it in the updated data operation table.

[0206] Based on this, the server receives marking operation instructions for data files and stores the marking operation information associated with the instructions in the operation log. If the operation log meets the log information processing conditions, the server updates the data operation table associated with the data file according to the marking operation information recorded in the operation log. Based on the updated data operation table, a data operation request is generated for the corresponding target data node 520, and the data operation request is sent to the target data node 520. The target data node 520 marks the data range to be operated on in its local storage space based on the data operation request, and generates status update information based on the marking results, which is then sent to the server 510. The server 510 receives the operation status information and records it in the updated data operation table, where the data range to be operated on is the data range to be deleted in the data node.

[0207] This specification provides a data processing system that receives marking operation instructions for data files and stores the marking operation information associated with the instructions in an operation log. When the operation log meets the log information processing conditions, the system updates the data operation table associated with the data file based on the marking operation information recorded in the operation log. Based on the updated data operation table, a data operation request for the corresponding target data node is generated and sent to the target data node. By updating the data operation table according to the operation log, determining the target data node based on the data operation table, and performing the marking operation at the target data node, collaborative processing of data files based on marking operation instructions is achieved.

[0208] The above is an illustrative scheme of a data processing system according to this embodiment. It should be noted that the technical solution of this data processing system and the technical solution of the data processing method described above belong to the same concept. For details not described in detail in the technical solution of the data processing system, please refer to the description of the technical solution of the data processing method described above.

[0209] Figure 6 A structural block diagram of a computing device 600 according to one embodiment of this specification is shown. The components of the computing device 600 include, but are not limited to, a memory 610 and a processor 620. The processor 620 is connected to the memory 610 via a bus 630, and a database 650 is used to store data.

[0210] The computing device 600 also includes an access device 640, which enables the computing device 600 to communicate via one or more networks 660. Examples of these networks include a Public Switched Telephone Network (PSTN), a Local Area Network (LAN), a Wide Area Network (WAN), a Personal Area Network (PAN), or a combination of communication networks such as the Internet. The access device 640 may include one or more of any type of wired or wireless network interface (e.g., a Network Interface Card (NIC)), such as an IEEE 802.11 Wireless Local Area Network (WLAN) interface, a Wi-MAX interface, an Ethernet interface, a Universal Serial Bus (USB) interface, a cellular network interface, a Bluetooth interface, a Near Field Communication (NFC) interface, and so on.

[0211] In one embodiment of this specification, the above-described components of the computing device 600 and Figure 6 Other components, not shown, can also be connected to each other, for example, via a bus. It should be understood that... Figure 6 The block diagram of the computing device shown is for illustrative purposes only and is not intended to limit the scope of this specification. Those skilled in the art can add or replace other components as needed.

[0212] The computing device 600 can be any type of stationary or mobile computing device, including mobile computers or mobile computing devices (e.g., tablet computers, personal digital assistants, laptop computers, notebook computers, netbooks, etc.), mobile phones (e.g., smartphones), wearable computing devices (e.g., smartwatches, smart glasses, etc.) or other types of mobile devices, or stationary computing devices such as desktop computers or PCs. The computing device 600 can also be a mobile or stationary server.

[0213] The processor 620 is configured to execute the following computer-executable instructions, which, when executed by the processor, implement the steps of the above-described data processing method.

[0214] The above is an illustrative scheme of a computing device according to this embodiment. It should be noted that the technical solution of this computing device and the technical solution of the data processing method described above belong to the same concept. For details not described in detail in the technical solution of the computing device, please refer to the description of the technical solution of the data processing method described above.

[0215] An embodiment of this specification also provides a computer-readable storage medium storing computer-executable instructions that, when executed by a processor, implement the steps of the above-described data processing method.

[0216] The above is an illustrative embodiment of a computer-readable storage medium. It should be noted that the technical solution of this storage medium and the technical solution of the data processing method described above belong to the same concept. Details not described in detail in the technical solution of the storage medium can be found in the description of the technical solution of the data processing method described above.

[0217] An embodiment of this specification also provides a computer program, wherein when the computer program is executed in a computer, it causes the computer to perform the steps of the above-described data processing method.

[0218] The above is an illustrative example of a computer program according to this embodiment. It should be noted that the technical solution of this computer program and the technical solution of the data processing method described above belong to the same concept. Details not described in detail in the technical solution of the computer program can be found in the description of the technical solution of the data processing method described above.

[0219] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0220] The computer instructions include computer program code, which may be in the form of source code, object code, executable file, or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording media, USB flash drive, portable hard drive, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in the computer-readable medium may be appropriately added to or subtracted according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media may not include electrical carrier signals and telecommunication signals.

[0221] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that the embodiments in this specification are not limited to the described order of actions, because according to the embodiments in this specification, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in this specification are all preferred embodiments, and the actions and modules involved are not necessarily essential to the embodiments in this specification.

[0222] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0223] The preferred embodiments disclosed above are merely illustrative of this specification. The optional embodiments do not exhaustively describe all details, nor do they limit the invention to the specific implementations described. Clearly, many modifications and variations can be made based on the embodiments described herein. These embodiments are selected and specifically described in this specification to better explain the principles and practical applications of the embodiments, thereby enabling those skilled in the art to better understand and utilize this specification. This specification is limited only by the claims and their full scope and equivalents.

Claims

1. A data processing method, applied on a server side, comprising: Receive marking operation instructions for data files, and store the marking operation information associated with the marking operation instructions in the operation log; If the operation log meets the log information processing conditions, update the data operation table associated with the data file according to the marked operation information recorded in the operation log; Based on the updated data operation table, a data operation request is generated for the corresponding target data node, and the data operation request is sent to the target data node.

2. The method according to claim 1, wherein updating the data operation table associated with the data file based on the marked operation information recorded in the operation log comprises: Read the marking operation information recorded in the operation log, and determine the data interval information associated with the data file based on the marking operation information, wherein the data interval information is associated with the data interval contained in the data file; The data block operation information is determined based on the data range information, and the data operation table associated with the data file is updated based on the data block operation information.

3. The method according to claim 2, wherein generating the data operation request for the corresponding target data node based on the updated data operation table includes: Read the data block operation information from the updated data table; Based on the physical address information contained in the data block operation information, a data operation request corresponding to the target data node is generated, wherein the data operation request is associated with a data block of the data file, and the data block is stored in the target data node.

4. The method according to claim 1, wherein storing the marking operation information associated with the marking operation instruction in the operation log includes: The marking operation information associated with the marking operation instruction is stored in the operation log associated with the data file, wherein the operation log is created based on the historical operation instructions of the data file; or... The target operation log associated with the data file is used as the operation log, and the marking operation information associated with the marking operation instruction is stored in the operation log. The target operation log is used to store marking operation information associated with at least two data files.

5. The method according to claim 1, wherein updating the data operation table associated with the data file according to the marked operation information recorded in the operation log when the operation log meets the log information processing conditions includes: If new marked operation information is added to the operation log, it is determined that the operation log meets the log information processing conditions, and the step of updating the data operation table associated with the data file according to the marked operation information recorded in the operation log is executed; or, The operation log contains log information according to a preset log query cycle. If the query result determines that the marked operation information is newly added marked operation information, the operation log meets the log information processing conditions, and the step of updating the data operation table associated with the data file according to the marked operation information recorded in the operation log is executed.

6. The method according to claim 1, further comprising, after the step of sending the data operation request to the target data node, the method as described in claim 1: Based on the execution result of the data operation request executed by the target data node, the operation status information is determined and recorded in the updated data operation table.

7. The method according to claim 6, further comprising: Receive a data read request for the data file; Based on the data read request, query the data operation table; The feedback result corresponding to the data reading request is determined based on the query results.

8. The method according to claim 7, wherein determining the feedback result corresponding to the data reading request based on the query result includes: If, based on the query results, it is determined that the data operation table contains marker operation information corresponding to the data read request, a prompt message is generated and the prompt message is used as the feedback result, wherein the prompt message indicates that the data to be read corresponding to the data read request is in an unreadable state; If, based on the query results, it is determined that the data operation table does not contain the tag operation information corresponding to the data read request, a node data read request is sent to the data node corresponding to the data read request, and the original data returned by the data node in response to the data read request is used as the feedback result.

9. The method according to claim 1, wherein determining the target data node includes: Determine the file identifier corresponding to the data file, determine the first data node in the data nodes contained in the distributed file system based on the file identifier, and determine the second data node associated with the first data node in the data nodes contained in the distributed file system; The first data node and the second data node are used as target data nodes.

10. The method of claim 9, further comprising: Receive the target data read request; Send the target data read request to the first data node; If the first data node is unavailable, the system receives second target data from the second data node in response to the target data read request, wherein the second target data is the same as the first target data stored by the first data node.

11. The method of claim 6, further comprising: The data operation table is queried based on a preset data operation table query cycle; The target physical address information is determined based on the query results. The target physical address information is the physical address information corresponding to the target data block operation information whose operation status is operation failure in the data operation table. Based on the target physical address information, a data reoperation request is generated and sent to the data node corresponding to the target data block operation information; Receive the execution result of the data node executing the data reoperation request, and update the data operation table based on the execution result.

12. A data processing system, comprising: The server is used to receive marking operation instructions for data files and store the marking operation information associated with the marking operation instructions in the operation log; If the operation log meets the log information processing conditions, update the data operation table associated with the data file according to the marked operation information recorded in the operation log; generate a data operation request for the corresponding target data node based on the updated data operation table, and send the data operation request to the target data node; The target data node is used to mark the data range to be operated on in the local storage space based on the data operation request, and generate status update information based on the marking result and send it to the server. The server is also used to receive the operation status information and record it in the updated data operation table.

13. A computing device, comprising: Memory and processor; The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions, which, when executed by the processor, implement the steps of the data processing method according to any one of claims 1 to 11.

14. A computer-readable storage medium storing computer-executable instructions that, when executed by a processor, implement the steps of the data processing method according to any one of claims 1 to 11.

Images