Data processing method, device and computer equipment

Abstract

The present application relates to a data processing method, apparatus, and computer device. The method includes: upon determining that target additional data of a target additional write object exists in an object storage system, obtaining a data archiving record of the target additional write object from the object storage system based on the object identifier of the target additional write object, wherein the data archiving record is used to record data information of the archived data of the target additional write object, where the archived data is data that has been stored in the archiving storage system; storing the target additional data in the archiving storage system based on the data archiving record; and after the target additional data has been stored in the archiving storage system, updating the data archiving record based on the data information of the target additional data, and deleting the target additional data in the object storage system. The use of this method can improve the flexibility and reliability of data processing.

Description

Technical Field

[0001] The present application relates to the technical field of data storage, and in particular to a data processing method, apparatus, and computer equipment. Background Art

[0002] As object storage systems become more widely used, append write operations are becoming increasingly common to ensure data upload efficiency. Existing append write operations generally split the data to be uploaded into the first uploaded data and subsequent appended data. Subsequently, after the first uploaded data is uploaded to the object storage system, each subsequent appended data is uploaded to the object storage system in sequence.

[0003] However, existing append write operations can only be performed on data objects in the object storage system, and cannot be performed on data objects that have been archived in an external archiving storage system, which reduces the flexibility of data processing. Summary of the Invention

[0004] Based on this, it is necessary to provide a data processing method, device and computer equipment that can improve data processing flexibility in response to the above technical problems.

[0005] In a first aspect, the present application provides a data processing method, comprising:

[0006] When it is determined that target additional data of the target additional write object exists in the object storage system, obtaining a data archiving record of the target additional write object from the object storage system according to the object identifier of the target additional write object; wherein the data archiving record is used to record data information of archived data of the target additional write object, where the archived data is data that has been stored in the archiving storage system;

[0007] According to the data archiving records, the target additional data is stored in the archiving storage system;

[0008] After the target additional data has been stored in the archive storage system, the data archive record is updated according to the data information of the target additional data, and the target additional data in the object storage system is deleted.

[0009] In an embodiment of the present application, data archiving records are introduced. By storing the target additional data in the archiving storage system according to the data archiving records, the problem in the prior art that it is impossible to perform additional write operations on objects that have been archived in an external archiving storage system is solved, thereby improving the flexibility and reliability of data processing.

[0010] In one embodiment, the data information of the archived data includes a data storage location in the archive storage system; storing the target additional data in the archive storage system according to the data archive record includes:

[0011] According to the data storage location, a target storage location of the target additional data in the archive storage system is determined; and according to the target storage location, the target additional data is stored in the archive storage system.

[0012] In the embodiment of the present application, the target storage location of the target additional data in the archival storage system is determined by the data storage location, so that the consistency of the additional data in the archival storage system can be ensured.

[0013] In one embodiment, storing the target additional data in the archival storage system according to the target storage location includes:

[0014] The data archiving time is determined according to the delayed archiving period and the data uploading time of the target additional data to the object storage system; when the data archiving time is reached, the target additional data is stored in the target storage location in the archiving storage system.

[0015] In an embodiment of the present application, by determining the data archiving time based on the delayed archiving period of the target additional data and the data upload time, and storing the target additional data in the archiving storage system at the data archiving time, the data access efficiency and storage cost of the object storage system can be guaranteed.

[0016] In one embodiment, the method further comprises:

[0017] Obtain first data summary information of the stored target additional data fed back by the archiving storage system; determine second data summary information corresponding to the target additional data; and determine data archiving reliability of the target additional data based on consistency between the first data summary information and the second data summary information.

[0018] In the embodiment of the present application, the reliability of data archiving can be ensured by determining the data archiving reliability of the target additional data based on the consistency between the first data summary information and the second data summary information.

[0019] In one embodiment, determining that target append data of a target append-write object exists in the object storage system includes:

[0020] Obtaining a storage identifier of the target append-write object based on the object identifier of the target append-write object; if the storage identifier is a hybrid storage identifier, determining that target append-write data of the target append-write object exists in the object storage system; wherein the hybrid storage identifier is used to indicate that data of the target append-write object is stored in both the object storage system and the archive storage system;

[0021] Accordingly, after the target additional data has been stored in the archival storage system, the method further includes:

[0022] The storage identifier is changed to an archive storage identifier; the archive storage identifier is used to indicate that the data of the target append-write object is stored in the archive storage system.

[0023] In the embodiment of the present application, the data storage status of the target append-write object is determined by the storage identifier of the target append-write object, which can ensure the reliability of the data storage status determination.

[0024] In one embodiment, obtaining a data archive record of a target append-write object from an object storage system according to an object identifier of the target append-write object includes:

[0025] If the data archiving record of the target append-write object is not obtained from the object storage system based on the object identifier of the target append-write object, a data archiving record for the target append-write object is created; based on the data storage situation of the archiving storage system, the data storage location of the data of the target append-write object in the archiving storage system is determined; and the data storage location is added to the data archiving record.

[0026] In an embodiment of the present application, when there is no data archiving record for the target appended write object in the object storage system, the reliability of the data archiving record construction can be ensured by creating a data archiving record for the target appended write object and adding the determined data storage location to the data archiving record.

[0027] In one embodiment, the method further comprises:

[0028] In response to a data read request for target archived data in the archived data of the target append-write object, a data archive record of the target append-write object is obtained; based on the data archive record, the target archived data is obtained from the archive storage system, and a temporary data copy corresponding to the target archived data is constructed in the object storage system; based on the temporary data copy, the data read request is executed.

[0029] In an embodiment of the present application, by constructing a temporary data copy corresponding to the target archived data in the object storage system to execute a data read request, the data read request can be responded to quickly, thereby improving the processing efficiency of the data read request.

[0030] In one embodiment, the method further comprises:

[0031] The copy deletion time is determined according to the copy construction time of the temporary data copy; when the copy deletion time is reached, it is determined whether the data read request is completed; if so, the temporary data copy in the object storage system is deleted.

[0032] In an embodiment of the present application, by configuring a corresponding copy deletion time for each temporary data copy and deleting the temporary data copy in the object storage system when the copy deletion time is reached, the available capacity in the object storage system can be guaranteed.

[0033] In a second aspect, the present application further provides a data processing device, comprising:

[0034] a record acquisition module configured to, upon determining that target additional data of a target additional write object exists in the object storage system, acquire a data archive record of the target additional write object from the object storage system based on the object identifier of the target additional write object; wherein the data archive record is configured to record data information of archived data of the target additional write object, where the archived data is data that has been stored in the archive storage system;

[0035] A data archiving module is used to store target additional data into an archiving storage system according to data archiving records;

[0036] The record update module is used to update the data archiving record according to the data information of the target additional data after the target additional data has been stored in the archiving storage system, and delete the target additional data in the object storage system.

[0037] In a third aspect, the present application further provides a computer device comprising a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

[0038] When it is determined that target additional data of the target additional write object exists in the object storage system, obtaining a data archiving record of the target additional write object from the object storage system according to the object identifier of the target additional write object; wherein the data archiving record is used to record data information of archived data of the target additional write object, where the archived data is data that has been stored in the archiving storage system;

[0039] According to the data archiving records, the target additional data is stored in the archiving storage system;

[0040] After the target additional data has been stored in the archive storage system, the data archive record is updated according to the data information of the target additional data, and the target additional data in the object storage system is deleted.

[0041] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the following steps are implemented:

[0042] When it is determined that target additional data of the target additional write object exists in the object storage system, obtaining a data archiving record of the target additional write object from the object storage system according to the object identifier of the target additional write object; wherein the data archiving record is used to record data information of archived data of the target additional write object, where the archived data is data that has been stored in the archiving storage system;

[0043] According to the data archiving records, the target additional data is stored in the archiving storage system;

[0044] After the target additional data has been stored in the archive storage system, the data archive record is updated according to the data information of the target additional data, and the target additional data in the object storage system is deleted.

[0045] In a fifth aspect, the present application further provides a computer program product, comprising a computer program, which, when executed by a processor, implements the following steps:

[0046] When it is determined that target additional data of the target additional write object exists in the object storage system, obtaining a data archiving record of the target additional write object from the object storage system according to the object identifier of the target additional write object; wherein the data archiving record is used to record data information of archived data of the target additional write object, where the archived data is data that has been stored in the archiving storage system;

[0047] According to the data archiving records, the target additional data is stored in the archiving storage system;

[0048] After the target additional data has been stored in the archive storage system, the data archive record is updated according to the data information of the target additional data, and the target additional data in the object storage system is deleted.

[0049] The above-mentioned data processing method, apparatus, and computer device, upon determining that target additional data of a target additional write object exists in an object storage system, obtains a data archiving record recording the data information of the archived data of the target additional write object from the object storage system based on the object identifier of the target additional write object, and stores the target additional data in the archive storage system based on the data archiving record. Subsequently, after the target additional data has been stored in the archive storage system, the data archiving record is updated based on the data information of the target additional data, and the target additional data in the object storage system is deleted. The above-mentioned method introduces the data archiving record, and by storing the target additional data in the archive storage system based on the data archiving record, it solves the problem in the prior art of being unable to perform additional write operations on objects that have already been archived in an external archive storage system, thereby improving the flexibility and reliability of data processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the following briefly introduces the drawings required for use in the embodiments of the present application or related technical descriptions. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying any creative work.

[0051] Figure 1A 1 is a flow chart of a data processing method in one embodiment;

[0052] Figure 1B A schematic diagram of data archiving processing in one embodiment;

[0053] Figure 2 A schematic diagram of a data archiving process in one embodiment;

[0054] Figure 3 A schematic diagram of a process for archiving data in another embodiment;

[0055] Figure 4 A schematic diagram of a process for determining data archiving reliability in one embodiment;

[0056] Figure 5A A schematic diagram of a process for creating a data archiving record in one embodiment;

[0057] Figure 5B A schematic diagram of data archiving records in one embodiment;

[0058] Figure 6 is a flow chart of a data processing method in another embodiment;

[0059] Figure 7 is a structural block diagram of a data processing device in one embodiment;

[0060] Figure 8 is a structural block diagram of a data processing device in another embodiment;

[0061] Figure 9 FIG. 1 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0062] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0063] As object storage systems become more widely used, append write operations are becoming increasingly common to ensure data upload efficiency. Existing append write operations generally split the data to be uploaded into the first uploaded data and subsequent appended data. Subsequently, after the first uploaded data is uploaded to the object storage system, each subsequent appended data is uploaded to the object storage system in sequence.

[0064] However, existing append write operations can only be performed on data objects in the object storage system, and cannot be performed on data objects that have been archived in an external archiving storage system, which reduces the flexibility of data processing.

[0065] Based on this, in an exemplary embodiment, a data processing method is provided, which is described by taking the method applied to a data processing device associated with an object storage system as an example. Figure 1A As shown, the specific steps include:

[0066] S101 , when it is determined that target additional data of a target additional writing object exists in the object storage system, obtain a data archiving record of the target additional writing object from the object storage system according to an object identifier of the target additional writing object.

[0067] Object storage systems store different types of data or data with different access frequencies on different storage media. Furthermore, different storage media are divided into different storage types based on storage performance, such as performance storage, standard storage, and archive storage. Archival storage systems generally use Blu-ray disc libraries or tape libraries as storage media. Archival storage systems offer high reliability, long storage times, and low storage costs.

[0068] An append write object is an object to which data can be appended and uploaded; a target append write object is an append write object that requires data to be appended and uploaded; target append data is data that is appended and uploaded to the object storage system by the target append write object; and an object identifier is an identifier that can identify the target append write object.

[0069] The data archive record is used to record data information of archived data of the target append-write object, where the archived data is data stored in the archive storage system; the data information of the archived data may include metadata information of the archived data.

[0070] In this embodiment, in order to improve the efficiency of data uploading, the data of the appended write object can be split into multiple data blocks and uploaded to the object storage system in sequence; then, in order to ensure the available capacity of the object storage system, the data in the object storage system can be transferred to the archive storage system and stored by the archive storage system.

[0071] It is understandable that due to the upload interval between data blocks, there may be a situation where some data in the same append-write object has been stored in the archive storage system, while subsequent appended data is still stored in the object storage system. At this time, it is necessary to continue uploading the appended data stored in the object storage system.

[0072] Optionally, when it is determined that the target additional data of the target additional write object exists in the object storage system based on the data storage situation of the object storage system, the object identifier of the target additional write object can be used as an index to retrieve the corresponding relationship between the object identifier and the data archiving record stored in the object storage system to obtain the data archiving record of the target additional write object.

[0073] S102: Store the target additional data in an archiving storage system according to the data archiving record.

[0074] In this embodiment, to ensure standardized data storage within the object storage system, the target appended data is written to the data storage pool, and the metadata of the target appended data object is written to the metadata storage pool. The metadata storage pool is typically composed of hard disks with high storage performance, such as solid-state drives (SSDs). The data storage pool is typically composed of storage media with average storage performance, such as mechanical hard disks (HDDs).

[0075] Optionally, after determining the data archiving record of the target appended write object, the data processing strategy for the target appended data can be determined based on the data archiving record; then, the target appended data is read out from the data storage pool, and after the target appended data is processed using the data processing strategy, it is uploaded to the archive storage system through the interface between the archive storage system and the object storage system in the same manner as an ordinary object upload.

[0076] For example, reference Figure 1B , users can upload additional data to the object storage system by appending upload. At this time, the additional data will be stored in the data storage pool in the object storage system; then, the PUT interface between the archive storage system and the object storage system can be used to upload the additional data to the archive storage system.

[0077] It is understandable that in this embodiment, the archival storage system may include multiple archive buckets, but in order to ensure the consistency of data storage, all data of any appended write object must be stored in the same archive bucket; further, the processing strategy of each archive bucket after receiving the appended data can be different or the same.

[0078] Optionally, after receiving the target additional data, the archive storage system can adopt the identification generation strategy of the archive bucket where the target additional data is located, and generate unique identification information of the target additional data in the archive storage system based on the object identifier of the target additional write object and the data content of the target additional data, so as to accurately locate the required data when reading the data subsequently.

[0079] S103: After the target additional data has been stored in the archive storage system, the data archive record is updated according to the data information of the target additional data, and the target additional data in the object storage system is deleted.

[0080] Optionally, after the target additional data has been stored in the archive storage system, the archive metadata in the data archive record can be updated according to the storage location of the target additional data, and the metadata of the target additional data and the corresponding archive object name can be added to the data archive record.

[0081] Furthermore, after updating the data archiving record, the location information of the target additional data in the object storage system is deleted, and a background task is submitted to delete the target additional data to complete the archiving process of the target additional data.

[0082] In this embodiment, when there are multiple copies of additional data of the target additional write object in the object storage system, for each copy of the additional data, the above steps can be used in the order of data upload to archive each copy of the additional data in sequence; then, after each copy of the additional data has been stored in the archival storage system, the data archiving record is uniformly updated according to the data information of each copy of the additional data.

[0083] In the above-mentioned data processing method, when it is determined that target additional data of a target additional write object exists in the object storage system, a data archiving record recording data information of the archived data of the target additional write object is obtained from the object storage system based on the object identifier of the target additional write object, and the target additional data is stored in the archive storage system based on the data archiving record. Subsequently, after the target additional data has been stored in the archive storage system, the data archiving record is updated based on the data information of the target additional data, and the target additional data in the object storage system is deleted. The above-mentioned method introduces the data archiving record and stores the target additional data in the archive storage system based on the data archiving record, thereby resolving the problem in the prior art of being unable to perform additional write operations on objects that have already been archived in an external archive storage system, thereby improving the flexibility and reliability of data processing.

[0084] In order to ensure the continuity of the additional data storage, based on the above embodiment, in this embodiment, the data information of the archived data includes the data storage location in the archive storage system; further, an optional method of data archiving is provided, such as Figure 2 As shown, the specific steps include:

[0085] S201 : Determine a target storage location of target additional data in an archiving storage system according to a data storage location.

[0086] The target storage location is the storage location of the target additional data in the archiving storage system.

[0087] The data storage location can be the identification information of the target archive bucket where the target appended write object is located, or it can be the previous archived data corresponding to the target appended data, the previous data storage location in the archive storage system, which is not specifically limited in this embodiment.

[0088] Optionally, if the data storage location is the identification information of the target archive bucket, the target archive bucket's free storage locations can be obtained based on the target archive bucket's identification information; then, any free storage location can be used as the target storage location. Alternatively, the free storage location closest to the storage location of the previous archived data of the target appended data can be used as the target storage location.

[0089] In a case where the data storage location is the previous data storage location, the target storage location may be calculated based on the previous data storage location based on a preset storage interval.

[0090] S202: Store the target additional data in the archiving storage system according to the target storage location.

[0091] Optionally, after the target storage location is determined, the additional data may be uploaded to the target storage location in the archive storage system by using an interface between the archive storage system and the object storage system to complete archiving of the target additional data.

[0092] In the embodiment of the present application, the target storage location of the target additional data in the archival storage system is determined by the data storage location, so that the consistency of the additional data in the archival storage system can be ensured.

[0093] In order to ensure the data access efficiency and storage cost of the object storage system, based on the above embodiment, this embodiment provides another optional method of data archiving, such as Figure 3 As shown, the specific steps include:

[0094] S301 , determining a data archiving time according to a delayed archiving period and a data uploading time of target additional data to an object storage system.

[0095] The delayed archiving period is the period for delayed archiving of the appended data. Different delayed archiving periods can be configured for different types of appended write objects. In this embodiment, the delayed archiving period can be a pre-set number of days for delayed archiving; the data upload time is the time when the object storage system receives the target appended data; and the data archiving time is the time when the target appended data is archived.

[0096] Optionally, to ensure data access efficiency and storage costs in the object storage system, a delayed archiving period for the additional data stored in the object storage system can be pre-set. The data upload time corresponding to the target additional data can then be processed based on the delayed archiving period to obtain the data archiving time. For example, the data upload time corresponding to the target additional data can be added with a number of delayed archiving days to calculate the data archiving time for the target additional data.

[0097] S302: When the data archiving time is reached, the target additional data is stored in the target storage location in the archiving storage system.

[0098] Optionally, the current time of the object storage system may be compared with the data archiving time. If the times are consistent, the target additional data may be stored in the target storage location in the archiving storage system.

[0099] It can be understood that since there are multiple copies of additional data stored in the object storage system, a first-time scanning task can be uniformly configured to scan the data archiving time corresponding to each additional data; accordingly, the current time of the object storage system can be used as an index, and the first-time scanning task can be used to query the data archiving time corresponding to each additional data to obtain the current archived data whose data archiving time is consistent with the current time; then, the current archived data can be stored in the corresponding storage location in the archive storage system by parallel uploading.

[0100] In an embodiment of the present application, by determining the data archiving time based on the delayed archiving period and the data upload time of the target additional data, and storing the target additional data in the archiving storage system at the data archiving time, the data access efficiency and storage cost of the object storage system can be guaranteed.

[0101] In the process of uploading data from the object storage system to the archive storage system, data loss or upload errors may occur. Based on this, in order to ensure the reliability of data archiving, based on the above embodiment, this embodiment provides an optional method for determining the reliability of data archiving, such as Figure 4 As shown, the specific steps include:

[0102] S401: Obtain first data summary information of stored target additional data fed back by the archiving storage system.

[0103] The first data summary information is data summary information fed back by the archiving storage system based on the stored target additional data.

[0104] Optionally, after obtaining the target additional data, the archival storage system may generate first data digest information based on the obtained target additional data. For example, the archival storage system may use the Message Digest Algorithm Version 5 (MD5) to calculate an MD5 value corresponding to the target additional data and feed the MD5 value as the first data digest information back to the object storage system.

[0105] S402: Determine second data summary information corresponding to the target additional data.

[0106] The second data summary information is data summary information generated by the object storage system based on the stored target additional data.

[0107] Optionally, before the object storage system uploads the target additional data to the archive storage system, it may determine second data digest information corresponding to the target additional data and store the second data digest information at a corresponding location in the data archive record. For example, the object storage system may use MD5 to calculate an MD5 value corresponding to the target additional data and store the MD5 value as the second data digest information at a corresponding location in the data archive record.

[0108] Correspondingly, after receiving the first data summary information, the second data summary information corresponding to the target additional data can be obtained at the corresponding position in the data archive record.

[0109] S403: Determine the data archiving reliability of the target additional data according to the consistency between the first data summary information and the second data summary information.

[0110] Optionally, the first data summary information and the second data summary information may be verified for consistency. If the consistency verification result is consistent, the data archiving reliability of the target additional data is determined to be reliable; if the consistency verification result is inconsistent, the data archiving reliability of the target additional data is determined to be unreliable.

[0111] When the data archiving reliability is unreliable, the target additional data needs to be archived again until the data archiving reliability is reliable.

[0112] For example, if the data archiving reliability is unreliable, a data purge command can be sent to the archiving storage system to delete the erroneous target additional data in the archiving storage system. Subsequently, the object storage system can re-upload the target additional data to the archiving storage system. Alternatively, a data overwrite method can be used to directly overwrite the erroneous target additional data with the newly uploaded target additional data.

[0113] In the embodiment of the present application, the reliability of data archiving can be ensured by determining the data archiving reliability of the target additional data based on the consistency between the first data summary information and the second data summary information.

[0114] To ensure the reliability of determining the data storage status, a corresponding storage identifier may be configured for the append-write object based on the data storage situation. Based on this, and in addition to the above embodiment, this embodiment provides an optional method for determining the state of the append-write object, which specifically includes the following steps:

[0115] According to the object identifier of the target append-write object, a storage identifier of the target append-write object is obtained; when the storage identifier is a hybrid storage identifier, it is determined that target append-write data of the target append-write object exists in the object storage system.

[0116] The storage identifier is used to represent the data storage status of the target append-write object; and the hybrid storage identifier is used to represent that the data of the target append-write object is stored in both the object storage system and the archive storage system.

[0117] Optionally, the object identifier of the target append write object can be used as an index to query in the object storage system to obtain the storage identifier of the target append write object; then, if it is determined that the storage identifier is a hybrid storage identifier, it proves that the data of the target append write object is stored in both the object storage system and the archive storage system, that is, the target append data of the target append write object exists in the object storage system.

[0118] If the storage identifier is determined to be an archive storage identifier, it proves that the data of the target append-write object is all stored in the archive storage system, that is, the target append-write object's target data does not exist in the object storage system. The archive storage identifier is used to indicate that the data of the target append-write object is all stored in the archive storage system.

[0119] Accordingly, to prevent the target additional data from being archived repeatedly, after the target additional data has been stored in the archival storage system, the storage identifier is changed to the archive storage identifier. Specifically, if the data archiving reliability of the target additional data is determined to be reliable, the storage identifier of the target additional write object needs to be changed from the hybrid storage identifier to the archive storage identifier.

[0120] It can be understood that configuring corresponding storage identifiers for appended write objects can not only ensure the reliability of the data storage status determination, but also directly skip the appended write objects whose storage identifier is the archive storage identifier when using the first-time scanning task to scan the data archiving moment of each appended data, thereby reducing the amount of data processing.

[0121] In the embodiment of the present application, the data storage status of the target append-write object is determined by the storage identifier of the target append-write object, which can ensure the reliability of the data storage status determination.

[0122] When the data of the appended write object is uploaded to the object storage system for the first time, there is no data archive record of the appended write object in the object storage system. Based on this, on the basis of the above embodiment, this embodiment provides an optional method for creating a data archive record, such as Figure 5A As shown, the specific steps include:

[0123] S501 : When a data archiving record of a target additional writing object is not obtained from an object storage system according to an object identifier of the target additional writing object, a data archiving record for the target additional writing object is created.

[0124] If the data archiving record of the target append-write object is not obtained from the object storage system based on the object identifier of the target append-write object, it proves that the data of the target append-write object is uploaded to the object storage system for the first time. At this time, it is necessary to create a data archiving record for the target append-write object based on the relevant information of the target append-write object.

[0125] Exemplary, reference Figure 5B , a data archiving record for the target append-write object can be created based on the metadata of the target append-write object, the access control list (object ACL) of the target append-write object, and the user metadata related to the target append-write object (object usermeta).

[0126] S502: Determine the data storage location of the target append-write object in the archival storage system according to the data storage status of the archival storage system.

[0127] The data storage status may be data storage information of each archive bucket in the archive storage system.

[0128] Optionally, after creating a data archive record for the target append-to-write object, the data storage status of the archive storage system can be obtained from the archive storage system. Subsequently, an archive bucket can be allocated to the target append-to-write object based on the remaining storage space in each archive bucket in the archive storage system. For example, the archive bucket with the largest remaining storage space can be allocated to the target append-to-write object.

[0129] Furthermore, the free storage location associated with the archive bucket to which the target append-write object is allocated may be used as the data storage location of the data of the target append-write object in the archive storage system.

[0130] S503: Add the data storage location to the data archiving record.

[0131] Optionally, after determining the data storage location of the target appended write object's data in the archive storage system, the data storage location can be added to the data archive record, and then the first upload data of the target appended write object can be uploaded to the archive storage system according to the data storage location.

[0132] In an embodiment of the present application, when there is no data archiving record for the target appended write object in the object storage system, the reliability of the data archiving record construction can be ensured by creating a data archiving record for the target appended write object and adding the determined data storage location to the data archiving record.

[0133] In order to ensure a timely response to a data read request, based on the above embodiment, in this embodiment, an optional method for reading archived data is provided. Specifically, in response to a data read request for target archived data in the archived data of a target append-write object, a data archive record of the target append-write object is obtained; according to the data archive record, the target archived data is obtained from the archive storage system, and a temporary data copy corresponding to the target archived data is constructed in the object storage system; and based on the temporary data copy, the data read request is executed.

[0134] The target archived data is the archived data of the target append-write object that the user wants to read; the temporary data copy is the temporarily obtained data copy.

[0135] Optionally, after the object storage system detects a data read request for target archived data in the archived data of the target append-write object, it may obtain the data archive record of the target append-write object based on the target append-write object identifier with reference to the above step S101.

[0136] It is understandable that since the target archived data is not stored in the object storage system, it is necessary to retrieve the target archived data from the archived storage system to the object storage system. At this time, in order to facilitate the rapid response of data read requests after the archived data is restored to the object storage system, a temporary data copy corresponding to the archived data can be constructed.

[0137] It should be noted that what is obtained from the object storage system here is temporary target archive data, and the real target archive data is still stored in the archive storage system.

[0138] Optionally, the target append-write object metadata and the archive metadata in the data archive record can be used to determine the archive bucket where the target append-write object is located and the archive object name corresponding to the target archive data; then, the interface between the archive storage system and the object storage system can be used to obtain the target archive data from the corresponding archive bucket of the archive storage system according to the archive object name. For example, refer to Figure 1B , the GET interface between the archive storage system and the object storage system can be used to obtain the target archive data from the archive storage system to the object storage system.

[0139] Furthermore, after the object storage system obtains the target archive data, it can determine the MD5 value corresponding to the target archive data, and perform consistency verification on the MD5 value and the second data summary information corresponding to the target archive data in the data archive record. If the consistency verification result is consistent, it is determined that the obtained target archive data is consistent with the target archive data in the archive storage system; if the consistency verification result is inconsistent, it is determined that the obtained target archive data is inconsistent with the target archive data in the archive storage system, and the target archive data needs to be re-acquired at this time.

[0140] If the consistency verification result is consistent, a temporary data copy corresponding to the target archive data is constructed in the object storage system. At this time, the user can obtain the target archive data from the temporary data copy in the object storage system through the GET interface.

[0141] In this embodiment, in order to avoid repeatedly constructing a temporary data copy corresponding to the same archived data, after constructing the temporary data copy corresponding to the target archived data, a restored identifier can be added to the data archiving record and the metadata of the target archived data; after the temporary data copy is deleted, the restored identifier needs to be deleted.

[0142] Accordingly, before constructing a temporary data copy of any archived data, it is necessary to read the metadata of the archived data, and if there is no restored identifier in the metadata of the archived data, then obtain the archived data from the archive storage system.

[0143] In an embodiment of the present application, by constructing a temporary data copy corresponding to the target archived data in the object storage system to execute a data read request, the data read request can be responded to quickly, thereby improving the processing efficiency of the data read request.

[0144] In order to ensure the available storage space in the object storage system, based on the above embodiment, in this embodiment, an optional method for processing data copies is provided. Specifically, according to the copy construction time of the temporary data copy, the copy deletion time is determined; when the copy deletion time is reached, it is determined whether the data read request is executed; if so, the temporary data copy in the object storage system is deleted.

[0145] The replica creation time is the time when the object storage system obtains the target archive data; the replica deletion time is the time when the temporary data copy is deleted.

[0146] Optionally, to facilitate deletion of temporary data copies, you can configure a deletion time for the temporary data copy based on the copy build time. For example, you can add a lifetime to the copy build time to calculate the deletion time for the temporary data copy.

[0147] Subsequently, the replica build time can be compared with the replica deletion time. If the times are consistent, it can be determined whether the data read request has been completed based on the request execution progress. If so, the temporary data copy in the object storage system is directly deleted; if not, the replica deletion time can be updated until the data read request is completed.

[0148] It is understandable that since multiple temporary data copies may be stored in the object storage system at the same time, a second time scanning task can be uniformly configured to scan the copy deletion time corresponding to each temporary data copy; accordingly, the current time of the object storage system can be used as an index, and the second time scanning task can be used to query the copy deletion time of each temporary data copy to obtain the current deleted copy whose data archiving time is consistent with the current time, and delete the temporary data copy.

[0149] Furthermore, after deleting the temporary data copy, if the user still wants to obtain the target archived data, it is necessary to repeat the above steps to rebuild the temporary data copy.

[0150] In an embodiment of the present application, by configuring a corresponding copy deletion time for each temporary data copy and deleting the temporary data copy in the object storage system when the copy deletion time is reached, the available capacity in the object storage system can be guaranteed.

[0151] Figure 6 FIG1 is a flow chart of a data processing method in another embodiment. Based on the above embodiment, this embodiment provides an optional example of a data processing method. Figure 6 The specific implementation process is as follows:

[0152] S601 : When a data archiving record of a target additional writing object is not obtained from an object storage system according to an object identifier of the target additional writing object, a data archiving record for the target additional writing object is created.

[0153] The data archive record is used to record data information of archived data of the target append write object, and the archived data is data that has been stored in the archive storage system.

[0154] S602: Determine the data storage location of the target appended write object in the archiving storage system according to the data storage status of the archiving storage system, and add the data storage location to the data archiving record.

[0155] S603: When it is determined that the target additional data of the target additional write object exists in the object storage system, obtain a data archiving record of the target additional write object from the object storage system.

[0156] Optionally, obtaining a storage identifier of the target append-write object based on the object identifier of the target append-write object; if the storage identifier is a hybrid storage identifier, determining that target append-write data of the target append-write object exists in the object storage system; wherein the hybrid storage identifier is used to indicate that data of the target append-write object is stored in both the object storage system and the archive storage system;

[0157] S604: Determine a target storage location of the target additional data in the archiving storage system according to the data storage location.

[0158] S605 , determining the data archiving time according to the delayed archiving period and the data uploading time of the target additional data to the object storage system.

[0159] S606: When the data archiving time is reached, the target additional data is stored in the target storage location in the archiving storage system.

[0160] Optionally, first data summary information of the stored target additional data fed back by the archiving storage system is obtained; second data summary information corresponding to the target additional data is determined; and data archiving reliability of the target additional data is determined based on consistency between the first data summary information and the second data summary information.

[0161] S607: After the target additional data has been stored in the archive storage system, the data archive record is updated according to the data information of the target additional data, and the target additional data in the object storage system is deleted.

[0162] Optionally, after the target additional data has been stored in the archival storage system, the method further includes: modifying the storage identifier to an archival storage identifier; wherein the archival storage identifier is used to indicate that the data of the target additional write object are all stored in the archival storage system.

[0163] S608 : Responding to a data read request for target archived data in the archived data of the target appended write object, obtaining a data archive record of the target appended write object.

[0164] S609: Obtain target archived data from the archive storage system according to the data archive record, and construct a temporary data copy corresponding to the target archived data in the object storage system.

[0165] S610: Execute a data read request based on the temporary data copy.

[0166] Optionally, a copy deletion time is determined based on the copy creation time of the temporary data copy. When the copy deletion time is reached, it is determined whether the data read request has been completed. If so, the temporary data copy in the object storage system is deleted.

[0167] The specific process of S601-S610 can be found in the description of the above method embodiment. The implementation principle and technical effects are similar and will not be repeated here.

[0168] It should be understood that, although the various steps in the flowcharts involved in the various embodiments described above are displayed in sequence according to the instructions of the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and these steps can be executed in other orders. Moreover, at least a portion of the steps in the flowcharts involved in the various embodiments described above can include multiple steps or multiple stages, and these steps or stages are not necessarily executed and completed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a portion of steps or stages in other steps.

[0169] Based on the same inventive concept, the present application also provides a data processing device for implementing the aforementioned data processing method. The implementation solution provided by this device is similar to the implementation solution described in the aforementioned method. Therefore, the specific limitations in one or more data processing device embodiments provided below can be found in the above-mentioned limitations on the data processing method and will not be repeated here.

[0170] In an exemplary embodiment, Figure 7As shown, a data processing device 1 is provided, comprising: a record acquisition module 10, a data archiving module 20 and a record updating module 30, wherein:

[0171] The record acquisition module 10 is configured to, upon determining that target additional data of a target additional write object exists in the object storage system, acquire a data archive record of the target additional write object from the object storage system based on the object identifier of the target additional write object; wherein the data archive record is configured to record data information of archived data of the target additional write object, where the archived data is data that has been stored in the archive storage system;

[0172] The data archiving module 20 is used to store the target additional data in the archiving storage system according to the data archiving record;

[0173] The record updating module 30 is configured to update the data archiving record according to the data information of the target additional data after the target additional data has been stored in the archiving storage system, and delete the target additional data in the object storage system.

[0174] In an exemplary embodiment, the data information of the archived data includes a data storage location in the archive storage system; Figure 8 As shown, the data archiving module 20 includes:

[0175] A location determination unit 21 is configured to determine a target storage location of the target additional data in the archiving storage system according to the data storage location;

[0176] The data archiving unit 22 is configured to store the target additional data into the archiving storage system according to the target storage location.

[0177] In an exemplary embodiment, the data archiving unit 22 is specifically configured to:

[0178] The data archiving time is determined according to the delayed archiving period and the data uploading time of the target additional data to the object storage system; when the data archiving time is reached, the target additional data is stored in the target storage location in the archiving storage system.

[0179] In an exemplary embodiment, the data processing device 1 further includes a verification module, wherein the verification module is specifically configured to:

[0180] Obtain first data summary information of the stored target additional data fed back by the archiving storage system; determine second data summary information corresponding to the target additional data; and determine data archiving reliability of the target additional data based on consistency between the first data summary information and the second data summary information.

[0181] In an exemplary embodiment, the record acquisition module 10 is specifically configured to:

[0182] Obtaining a storage identifier of the target append-write object based on the object identifier of the target append-write object; if the storage identifier is a hybrid storage identifier, determining that target append-write data of the target append-write object exists in the object storage system; wherein the hybrid storage identifier is used to indicate that data of the target append-write object is stored in both the object storage system and the archive storage system;

[0183] Accordingly, after the target additional data has been stored in the archival storage system, the record acquisition module 10 is further configured to:

[0184] The storage identifier is changed to an archive storage identifier; the archive storage identifier is used to indicate that the data of the target append-write object is stored in the archive storage system.

[0185] In an exemplary embodiment, the record acquisition module 10 is further configured to:

[0186] If the data archiving record of the target append-write object is not obtained from the object storage system based on the object identifier of the target append-write object, a data archiving record for the target append-write object is created; based on the data storage situation of the archiving storage system, the data storage location of the data of the target append-write object in the archiving storage system is determined; and the data storage location is added to the data archiving record.

[0187] In an exemplary embodiment, the data processing device 1 further includes a data reading module, wherein the data reading module is specifically configured to:

[0188] In response to a data read request for target archived data in the archived data of the target append-write object, a data archive record of the target append-write object is obtained; based on the data archive record, the target archived data is obtained from the archive storage system, and a temporary data copy corresponding to the target archived data is constructed in the object storage system; based on the temporary data copy, the data read request is executed.

[0189] In an exemplary embodiment, the data processing device 1 further includes a copy deletion module, wherein the copy deletion module is specifically configured to:

[0190] The copy deletion time is determined according to the copy construction time of the temporary data copy; when the copy deletion time is reached, it is determined whether the data read request is completed; if so, the temporary data copy in the object storage system is deleted.

[0191] Each module in the above-mentioned data processing device can be implemented in whole or in part through software, hardware, or a combination thereof. Each module can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the corresponding operations of each module.

[0192] In an exemplary embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as shown in FIG. Figure 9 As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, abbreviated as I / O) and a communication interface. The processor, memory and input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and computer program in the non-volatile storage medium. The database of the computer device is used to store appended write data. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a data processing method is implemented.

[0193] Those skilled in the art will understand that Figure 9 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0194] In one embodiment, a computer device is further provided, including a memory and a processor. The memory stores a computer program, and the processor implements the steps in the above method embodiments when executing the computer program.

[0195] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0196] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0197] It should be noted that the data involved in this application (including but not limited to appending data, etc.) are all data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0198] Those skilled in the art will understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. In particular, any reference to memory, database, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The databases involved in the various embodiments provided herein may include at least one of a relational database and a non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the various embodiments provided herein may be, but are not limited to, general-purpose processors, central processing units (CPUs), graphics processing units (GPUs), digital signal processors (DSPs), programmable logic devices (PLDs), quantum computing-based data processing logic devices, artificial intelligence (AI) processors, and the like.

[0199] The technical features of the above embodiments can be combined arbitrarily. In order to make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0200] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present application. It should be noted that a person of ordinary skill in the art may make various modifications and improvements without departing from the spirit of the present application, and these modifications and improvements fall within the scope of protection of the present application. Therefore, the scope of protection of the present application shall be determined by the appended claims.

Claims

1. A data processing method, characterized in that: The method comprises: When it is determined that target additional data of the target additional write object exists in the object storage system, obtaining a data archiving record of the target additional write object from the object storage system according to the object identifier of the target additional write object; wherein the data archiving record is used to record data information of archived data of the target additional write object, and the archived data is data that has been stored in the archiving storage system; storing the target additional data in the archiving storage system according to the data archiving record; After the target additional data has been stored in the archival storage system, the data archiving record is updated according to the data information of the target additional data, and the target additional data in the object storage system is deleted.

2. The method according to claim 1, characterized in that The data information of the archived data includes a data storage location in the archive storage system; Storing the target additional data in the archival storage system according to the data archiving record includes: Determining a target storage location of the target additional data in the archival storage system according to the data storage location; The target additional data is stored in the archival storage system according to the target storage location.

3. The method according to claim 2, characterized in that Storing the target additional data in the archival storage system according to the target storage location includes: Determining a data archiving time according to a delayed archiving period and a data uploading time of uploading the target additional data to the object storage system; When the data archiving time is reached, the target additional data is stored in the target storage location in the archival storage system.

4. The method according to claim 3, characterized in that The method further comprises: Acquire first data summary information of the stored target additional data fed back by the archiving storage system; Determining second data summary information corresponding to the target additional data; The data archiving reliability of the target additional data is determined according to the consistency between the first data summary information and the second data summary information.

5. The method according to claim 1, wherein The determining whether target additional data of the target additional write object exists in the object storage system includes: Acquire a storage identifier of the target append-write object according to the object identifier of the target append-write object; In a case where the storage identifier is a hybrid storage identifier, determining that target additional data of the target additional write object exists in the object storage system; wherein the hybrid storage identifier is used to indicate that data of the target additional write object is stored in both the object storage system and the archive storage system; Accordingly, after the target additional data has been stored in the archival storage system, the method further includes: The storage identifier is modified to an archive storage identifier; wherein the archive storage identifier is used to indicate that the data of the target append-write object are all stored in the archive storage system.

6. The method according to claim 1, wherein The acquiring, from the object storage system according to the object identifier of the target append-write object, the data archiving record of the target append-write object includes: creating a data archiving record for the target append-write object if the data archiving record for the target append-write object is not obtained from the object storage system according to the object identifier of the target append-write object; Determining a data storage location of the target append-write object in the archive storage system according to a data storage situation of the archive storage system; The data storage location is added to the data archive record.

7. The method according to claim 1, characterized in that The method further comprises: In response to a data read request for target archived data in the archived data of the target append-write object, acquiring a data archive record of the target append-write object; According to the data archiving record, the target archived data is obtained from the archive storage system, and a temporary data copy corresponding to the target archived data is constructed in the object storage system; The data read request is executed based on the temporary data copy.

8. The method according to claim 7, characterized in that The method further comprises: Determining a copy deletion time based on a copy creation time of the temporary data copy; When the copy deletion time is reached, determining whether the data read request is completed; If so, the temporary data copy in the object storage system is deleted.

9. A data processing device, characterized in that: The device comprises: a record acquisition module configured to, upon determining that target additional data of a target additional write object exists in the object storage system, acquire, based on the object identifier of the target additional write object, a data archiving record of the target additional write object from the object storage system; wherein the data archiving record is configured to record data information of archived data of the target additional write object, the archived data being data that has been stored in the archiving storage system; A data archiving module, configured to store the target additional data into the archiving storage system according to the data archiving record; A record updating module is used to update the data archiving record according to the data information of the target additional data and delete the target additional data in the object storage system after the target additional data has been stored in the archiving storage system.

10. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 8 are implemented.

Images