Data processing method, apparatus, device, and medium

Abstract

The present specification provides a data processing method, device, equipment and medium. When a data write request for writing to-be-written data in a target data file is received, a target write position of the to-be-written data in the target data file and an end position of stored data in the target data file are obtained, a data part with a position after the end position of the stored data as a start position is split from the to-be-written data based on the target write position and the end position of the stored data as target data to be written, to generate a to-be-written file corresponding to the target data, and then the to-be-written file is stored in the target data file according to an indication of a target data storage position. Since the data is written in the form of a file, there is no need to modify or overwrite the written file, but the effect of adding data after the stored data can be achieved, that is, the effect of a data append write operation can be achieved.

Claims

1. A data processing method, comprising: In response to receiving a data write request for writing data to be written into a target data file, the target write position of the data to be written in the target data file and the end position of the stored data in the target data file are obtained; the target write position includes the start write position of the data to be written in the target data file and the length of the data to be written corresponding to the data to be written; Based on the target write position of the data to be written and the end position of the stored data in the target data file, a first positional relationship between the target write position and the end position of the stored data is determined; Based on the first positional relationship, a portion of the data to be written is extracted from the data to be written, with the starting position being the position after the end position of the stored data. This portion is then used as target data. If the first positional relationship indicates that the starting position is the same as the end position of the stored data, then the entire data to be written is used as the target data. If the first positional relationship indicates that the starting position is before the end position of the stored data, then the target data is obtained from the data to be written based on the starting position, the length of the data to be written, and the length of the stored data. Generate the file to be written corresponding to the target data in the data to be written; According to the indication of the target data storage location, the file to be written is stored in the target data file.

2. The method according to claim 1, wherein the target data is obtained from the data to be written based on the first positional relationship, the target write position, and the end position of the stored data, comprising: If the first positional relationship indicates that the start writing position is after the end position of the stored data, then it is not necessary to obtain the target data to be written from the data to be written.

3. The method according to claim 2, wherein obtaining the target data from the data to be written based on the start writing position, the length of the data to be written, and the length of the stored data includes: Based on the start writing position and the length of the data to be written, determine the end writing position of the data to be written in the target data file; Determine a second positional relationship between the end of the write position and the end position of the stored data; Based on the second positional relationship, the end of writing position, and the end of stored data position, the target data is obtained from the data to be written.

4. The method according to claim 3, wherein obtaining the target data from the data to be written based on the second positional relationship, the end-of-write position, and the end-of-stored-data position comprises: If the second positional relationship indicates that the end of writing is before the end of the stored data, or if the second positional relationship indicates that the end of writing is the same as the end of the stored data, then it is not necessary to obtain the target data from the data to be written. If the second positional relationship indicates that the end-of-write position is after the end-of-stored-data position, then the first data to be written in the data to be written is verified based on the target stored data in the target data file, so as to obtain the target data based on the verification result and the second data to be written in the data to be written; wherein, the target stored data is the data in the target data file whose storage position is between the start-of-write position and the end-of-stored-data position, the first data to be written is the data in the data to be written whose data position is between the start-of-write position and the end-of-stored-data position, and the second data to be written is the data in the data to be written whose data position is between the end-of-stored-data position and the end-of-write position.

5. The method according to claim 4, wherein obtaining the target data based on the verification result and the second data to be written in the data to be written comprises: If the verification result indicates that the target stored data is inconsistent with the first data to be written, then there is no need to obtain the target data; If the verification result indicates that the target stored data is consistent with the first data to be written, then the full amount of data in the second data to be written is used as the target data to be obtained.

6. The method according to claim 1, wherein storing the file to be written into the target data file according to the indication of the target data storage location comprises: The file to be written is stored in a region of the target data file starting from the target data storage location, so that the file to be written becomes a sub-file of the target data file.

7. The method according to claim 1, wherein after storing the file to be written into the target data file according to the indication of the target data storage location, the method further comprises: Update the length of the stored data in the target data file.

8. The method according to claim 1, wherein the target data file includes multiple sub-files, and the data in different sub-files is written in different batches; The method further includes: In response to receiving a data read request for reading data to be read from a target data file, the storage location of each sub-file in the target data file is obtained; Based on the storage locations of each sub-file in the target data file and the data storage location of the data to be read, a target sub-file for storing the data to be read is determined; Based on the data storage location of the data to be read, the data to be read is obtained from the target sub-file.

9. A data processing apparatus, comprising: The acquisition module is configured to, in response to receiving a data write request for writing data to be written into a target data file, acquire the target write position of the data to be written in the target data file and the end position of the stored data in the target data file; the target write position includes the start write position of the data to be written in the target data file and the length of the data to be written corresponding to the data to be written; A generation module is configured to determine a first positional relationship between the target write position and the end position of the stored data in the target data file, based on the target write position of the data to be written and the end position of the stored data in the target data file; according to the first positional relationship, a portion of data starting from the position after the end position of the stored data is extracted from the data to be written as target data, and a file to be written corresponding to the target data in the data to be written is generated; wherein, if the first positional relationship indicates that the start write position is consistent with the end position of the stored data, then the entire data in the data to be written is used as the target data; if the first positional relationship indicates that the start write position is before the end position of the stored data, then the target data is obtained from the data to be written based on the start write position, the length of the data to be written, and the length of the stored data; The writing module is used to store the file to be written into the target data file according to the indication of the target data storage location.

10. A computing device, comprising: processor; Memory used to store processor-executable instructions; The processor implements the method as described in any one of claims 1-8 by executing the executable instructions.

11. A computer-readable storage medium having stored thereon computer instructions that, when executed by a processor, implement the method as described in any one of claims 1-8.

Images