Data processing method and device and computer equipment

Abstract

The invention relates to a data processing method and device and computer equipment. The method comprises the steps that under the condition that it is determined that target additional data of a target additional writing object exists in an object storage system, a data archiving record of the target additional writing object is acquired from the object storage system according to an object identifier of the target additional writing object, the data archiving record is used for recording data information of archived data of the target additional writing object, and the archived data is data stored in the archiving storage system, storing the target additional data in the archiving storage system according to the data archiving record, and storing the target additional data in the archiving storage system after the target additional data is stored in the archiving storage system. And updating the data archiving record according to the data information of the target additional data, and deleting the target additional data in the object storage system. By adopting the method, the flexibility and reliability of data processing can be improved.

Description

Technical Field

[0001] The present application relates to the technical field of data storage, and in particular to a data processing method, device and computer equipment. Background Art

[0002] As the usage scenarios of object storage systems become more diverse, in order to ensure the efficiency of data upload, the use of append write operations is becoming more and more common. Existing append write operations generally split the data to be uploaded into the first uploaded data and subsequent appended data; then, after uploading the first uploaded data to the object storage system, each subsequent appended data is uploaded to the object storage system in turn.

[0003] However, existing append write operations can only be performed on data objects in the object storage system, and cannot be performed on data objects that have been archived in an external archive storage system, which reduces the flexibility of data processing. Summary of the invention

[0004] Based on this, it is necessary to provide a data processing method, device and computer equipment that can improve the flexibility of data processing in response to the above technical problems.

[0005] In a first aspect, the present application provides a data processing method, comprising:

[0006] When it is determined that the target additional data of the target additional write object exists in the object storage system, a data archiving record of the target additional write object is obtained from the object storage system according to the object identifier of the target additional write object; wherein the data archiving record is used to record data information of archived data of the target additional write object, and the archived data is data that has been stored in the archiving storage system;

[0007] According to the data archiving record, the target additional data is stored in the archiving storage system;

[0008] After the target additional data has been stored in the archive storage system, the data archive record is updated according to the data information of the target additional data, and the target additional data in the object storage system is deleted.

[0009] In an embodiment of the present application, data archiving records are introduced. By storing the target additional data in the archiving storage system according to the data archiving records, the problem in the prior art that it is impossible to perform additional write operations on objects that have been archived in an external archiving storage system is solved, thereby improving the flexibility and reliability of data processing.

[0010] In one embodiment, the data information of the archived data includes a data storage location in the archive storage system; storing the target additional data in the archive storage system according to the data archive record includes:

[0011] According to the data storage location, determine the target storage location of the target additional data in the archive storage system; according to the target storage location, store the target additional data in the archive storage system.

[0012] In the embodiment of the present application, the target storage location of the target additional data in the archival storage system is determined through the data storage location, so that the continuity of the additional data in the archival storage system can be ensured.

[0013] In one embodiment, storing the target additional data in the archive storage system according to the target storage location includes:

[0014] The data archiving time is determined according to the delayed archiving cycle and the data uploading time of the target additional data to the object storage system; when the data archiving time is reached, the target additional data is stored in the target storage location in the archiving storage system.

[0015] In an embodiment of the present application, by determining the data archiving time according to the delayed archiving period of the target additional data and the data upload time, and storing the target additional data in the archiving storage system at the data archiving time, the data access efficiency and storage cost of the object storage system can be guaranteed.

[0016] In one embodiment, the method further comprises:

[0017] Acquire first data summary information of the stored target additional data fed back by the archiving storage system; determine second data summary information corresponding to the target additional data; and determine data archiving reliability of the target additional data based on consistency between the first data summary information and the second data summary information.

[0018] In the embodiment of the present application, by determining the data archiving reliability of the target additional data based on the consistency between the first data summary information and the second data summary information, the reliability of the data archiving can be ensured.

[0019] In one embodiment, determining that target append data of a target append write object exists in the object storage system includes:

[0020] According to the object identifier of the target append-write object, a storage identifier of the target append-write object is obtained; when the storage identifier is a hybrid storage identifier, it is determined that the target append-write object has target append-write data in the object storage system; wherein the hybrid storage identifier is used to indicate that the data of the target append-write object is stored in both the object storage system and the archive storage system;

[0021] Accordingly, after the target additional data has been stored in the archival storage system, the method further includes:

[0022] The storage identifier is changed to an archive storage identifier; wherein the archive storage identifier is used to indicate that the data of the target append write object is stored in the archive storage system.

[0023] In the embodiment of the present application, the data storage state of the target additional write object is determined by the storage identifier of the target additional write object, so that the reliability of the determination of the data storage state can be ensured.

[0024] In one embodiment, obtaining a data archive record of a target append-write object from an object storage system according to an object identifier of the target append-write object includes:

[0025] If the data archiving record of the target append write object is not obtained from the object storage system based on the object identifier of the target append write object, create a data archiving record for the target append write object; determine the data storage location of the data of the target append write object in the archiving storage system based on the data storage situation of the archiving storage system; and add the data storage location to the data archiving record.

[0026] In an embodiment of the present application, when there is no data archiving record of the target append write object in the object storage system, the reliability of the data archiving record construction can be ensured by creating a data archiving record for the target append write object and adding the determined data storage location to the data archiving record.

[0027] In one embodiment, the method further comprises:

[0028] In response to a data read request for target archived data in the archived data of a target append write object, a data archive record of the target append write object is obtained; according to the data archive record, the target archived data is obtained from the archive storage system, and a temporary data copy corresponding to the target archived data is constructed in the object storage system; based on the temporary data copy, the data read request is executed.

[0029] In an embodiment of the present application, by constructing a temporary data copy corresponding to the target archived data in the object storage system to execute a data read request, the data read request can be responded to quickly, thereby improving the processing efficiency of the data read request.

[0030] In one embodiment, the method further comprises:

[0031] The copy deletion time is determined according to the copy building time of the temporary data copy; when the copy deletion time is reached, it is determined whether the data read request is executed; if so, the temporary data copy in the object storage system is deleted.

[0032] In an embodiment of the present application, by configuring a corresponding copy deletion time for each temporary data copy and deleting the temporary data copy in the object storage system when the copy deletion time is reached, the available capacity in the object storage system can be guaranteed.

[0033] In a second aspect, the present application further provides a data processing device, comprising:

[0034] A record acquisition module is used to acquire a data archiving record of the target additional write object from the object storage system according to the object identifier of the target additional write object when it is determined that the target additional write object has target additional data in the object storage system; wherein the data archiving record is used to record data information of archived data of the target additional write object, and the archived data is data that has been stored in the archiving storage system;

[0035] A data archiving module, used for storing target additional data into an archiving storage system according to data archiving records;

[0036] The record update module is used to update the data archive record and delete the target additional data in the object storage system according to the data information of the target additional data after the target additional data has been stored in the archive storage system.

[0037] In a third aspect, the present application further provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

[0038] When it is determined that the target additional data of the target additional write object exists in the object storage system, a data archiving record of the target additional write object is obtained from the object storage system according to the object identifier of the target additional write object; wherein the data archiving record is used to record data information of archived data of the target additional write object, and the archived data is data that has been stored in the archiving storage system;

[0039] According to the data archiving record, the target additional data is stored in the archiving storage system;

[0040] After the target additional data has been stored in the archive storage system, the data archive record is updated according to the data information of the target additional data, and the target additional data in the object storage system is deleted.

[0041] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the following steps are implemented:

[0042] When it is determined that the target additional data of the target additional write object exists in the object storage system, a data archiving record of the target additional write object is obtained from the object storage system according to the object identifier of the target additional write object; wherein the data archiving record is used to record data information of archived data of the target additional write object, and the archived data is data that has been stored in the archiving storage system;

[0043] According to the data archiving record, the target additional data is stored in the archiving storage system;

[0044] After the target additional data has been stored in the archive storage system, the data archive record is updated according to the data information of the target additional data, and the target additional data in the object storage system is deleted.

[0045] In a fifth aspect, the present application further provides a computer program product, including a computer program, which implements the following steps when executed by a processor:

[0046] When it is determined that the target additional data of the target additional write object exists in the object storage system, a data archiving record of the target additional write object is obtained from the object storage system according to the object identifier of the target additional write object; wherein the data archiving record is used to record data information of archived data of the target additional write object, and the archived data is data that has been stored in the archiving storage system;

[0047] According to the data archiving record, the target additional data is stored in the archiving storage system;

[0048] After the target additional data has been stored in the archive storage system, the data archive record is updated according to the data information of the target additional data, and the target additional data in the object storage system is deleted.

[0049] The above data processing method, device and computer equipment, when determining that there is target additional data of the target additional write object in the object storage system, obtains the data archiving record recording the data information of the archived data of the target additional write object from the object storage system according to the object identifier of the target additional write object, and stores the target additional data in the archive storage system according to the data archiving record; then, after the target additional data has been stored in the archive storage system, the data archiving record is updated according to the data information of the target additional data, and the target additional data in the object storage system is deleted. The above method introduces the data archiving record, and stores the target additional data in the archive storage system according to the data archiving record, which solves the problem in the prior art that it is impossible to perform additional write operations on objects that have been archived in an external archive storage system, and improves the flexibility and reliability of data processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the drawings required for use in the embodiments of the present application or related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0051] Figure 1A is a flow chart of a data processing method in one embodiment;

[0052] Figure 1B A schematic diagram of data archiving processing in one embodiment;

[0053] Figure 2 A schematic diagram of a process of data archiving in one embodiment;

[0054] Figure 3 A schematic diagram of a process of data archiving in another embodiment;

[0055] Figure 4 A schematic diagram of a process for determining data archiving reliability in one embodiment;

[0056] Figure 5A A schematic diagram of a process for creating a data archiving record in one embodiment;

[0057] Figure 5B A schematic diagram of data archiving records in one embodiment;

[0058] Figure 6 is a flow chart of a data processing method in another embodiment;

[0059] Figure 7 is a structural block diagram of a data processing device in one embodiment;

[0060] Figure 8 is a structural block diagram of a data processing device in another embodiment;

[0061] Fig. 9 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0062] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0063] As the usage scenarios of object storage systems become more diverse, in order to ensure the efficiency of data upload, the use of append write operations is becoming more and more common. Existing append write operations generally split the data to be uploaded into the first uploaded data and subsequent appended data; then, after uploading the first uploaded data to the object storage system, each subsequent appended data is uploaded to the object storage system in turn.

[0064] However, existing append write operations can only be performed on data objects in the object storage system, and cannot be performed on data objects that have been archived in an external archive storage system, which reduces the flexibility of data processing.

[0065] Based on this, in an exemplary embodiment, a data processing method is provided, and the method is applied to a data processing device associated with an object storage system as an example for explanation. Figure 1A As shown, the specific steps include:

[0066] S101, when it is determined that target additional data of a target additional write object exists in the object storage system, obtain a data archiving record of the target additional write object from the object storage system according to an object identifier of the target additional write object.

[0067] Among them, the object storage system will store different types of data or data with different access frequencies in different storage media. Furthermore, different storage media are divided into different storage types according to storage performance, such as performance storage, standard storage, archive storage, etc. The archive storage system generally refers to a storage system that uses a Blu-ray disc library or a tape library as a storage medium. The archive storage system has the advantages of high reliability, long storage time, and low storage cost.

[0068] The append write object is an object to which data can be appended and uploaded at the end of the object; the target append write object is an append write object with data append upload requirements; the target append data is the data that the target append write object appends and uploads to the object storage system; and the object identifier is an identifier that can identify the target append write object.

[0069] The data archive record is used to record data information of archived data of the target append write object, where the archived data is data that has been stored in the archive storage system; the data information of the archived data may include metadata information of the archived data.

[0070] In this embodiment, in order to improve the efficiency of data uploading, the data of the appended write object can be split into multiple data blocks and uploaded to the object storage system in sequence; then, in order to ensure the available capacity of the object storage system, the data in the object storage system can be transferred to the archive storage system and stored by the archive storage system.

[0071] It is understandable that due to the upload interval between data blocks, some data in the same append write object may have been stored in the archive storage system, while subsequent append data is still stored in the object storage system. At this time, it is necessary to continue uploading the append data stored in the object storage system.

[0072] Optionally, when it is determined that the object storage system contains target additional data of the target additional write object, based on the data storage situation of the object storage system, the object identifier of the target additional write object can be used as an index to retrieve the data archive record of the target additional write object from the correspondence between the object identifier and the data archive record stored in the object storage system.

[0073] S102: Store the target additional data in an archiving storage system according to the data archiving record.

[0074] In this embodiment, in order to ensure the standardization of data storage in the object storage system, the target additional data will be written into the data storage pool, and the metadata of the target additional data object will be written into the metadata storage pool. Among them, the metadata storage pool is generally composed of a hard disk with good storage performance, such as a solid state drive SSD. The data storage pool is generally composed of a storage medium with average storage performance, such as a mechanical hard disk HDD.

[0075] Optionally, after determining the data archiving record of the target additional write object, a data processing strategy for the target additional data can be determined based on the data archiving record; then, the target additional data is read out from the data storage pool, and after the target additional data is processed using the data processing strategy, it is uploaded to the archiving storage system through the interface between the archiving storage system and the object storage system in the same manner as an ordinary object.

[0076] For example, refer to Figure 1B , users can upload additional data to the object storage system by appending upload. At this time, the additional data will be stored in the data storage pool in the object storage system; then, the PUT interface between the archive storage system and the object storage system can be used to upload the additional data to the archive storage system.

[0077] It can be understood that in this embodiment, the archive storage system may include multiple archive buckets, but in order to ensure the consistency of data storage, all data of any appended write object must be stored in the same archive bucket; further, the processing strategy of each archive bucket after receiving the appended data can be different or the same.

[0078] Optionally, after receiving the target additional data, the archive storage system can adopt the identification generation strategy of the archive bucket where the target additional data is located, and generate unique identification information of the target additional data in the archive storage system based on the object identifier of the target additional write object and the data content of the target additional data, so as to accurately locate the required data when reading the data subsequently.

[0079] S103, after the target additional data has been stored in the archive storage system, update the data archive record according to the data information of the target additional data, and delete the target additional data in the object storage system.

[0080] Optionally, after the target additional data has been stored in the archiving storage system, the archiving metadata in the data archiving record can be updated according to the storage location of the target additional data, and the metadata of the target additional data and the corresponding archiving object name can be added to the data archiving record.

[0081] Furthermore, after updating the data archiving record, the location information of the target additional data in the object storage system is deleted, and a background task is submitted to delete the target additional data to complete the archiving process of the target additional data.

[0082] In this embodiment, when there are multiple copies of additional data of the target additional write object in the object storage system, for each copy of the additional data, the above steps can be used in the order of data upload to archive each copy of the additional data in turn; then, after each copy of the additional data has been stored in the archive storage system, the data archive record is uniformly updated according to the data information of each copy of the additional data.

[0083] In the above data processing method, when it is determined that there is target additional data of the target additional write object in the object storage system, the data archiving record recording the data information of the archived data of the target additional write object is obtained from the object storage system according to the object identifier of the target additional write object, and the target additional data is stored in the archive storage system according to the data archiving record; then, after the target additional data has been stored in the archive storage system, the data archiving record is updated according to the data information of the target additional data, and the target additional data in the object storage system is deleted. The above method introduces the data archiving record, and stores the target additional data in the archive storage system according to the data archiving record, which solves the problem in the prior art that it is impossible to perform additional write operations on objects that have been archived in an external archive storage system, and improves the flexibility and reliability of data processing.

[0084] In order to ensure the continuity of the additional data storage, based on the above embodiment, in this embodiment, the data information of the archived data includes the data storage location in the archive storage system; further, an optional method of data archiving is provided, such as Figure 2 As shown, the specific steps include:

[0085] S201, determining a target storage location of target additional data in an archiving storage system according to a data storage location.

[0086] The target storage location is the storage location of the target additional data in the archive storage system.

[0087] The data storage location may be the identification information of the target archive bucket where the target append write object is located, or it may be the last archived data corresponding to the target append data, the last data storage location in the archive storage system, which is not specifically limited in this embodiment.

[0088] Optionally, when the data storage location is the identification information of the target archive bucket, the free storage location of the target archive bucket can be obtained according to the identification information of the target archive bucket; then, any free storage location can be used as the target storage location. Alternatively, the free storage location closest to the storage location where the last archived data of the target additional data is located can be used as the target storage location.

[0089] In the case where the data storage location is the previous data storage location, the target storage location may be calculated based on the previous data storage location based on a preset storage interval.

[0090] S202: Store the target additional data in the archiving storage system according to the target storage location.

[0091] Optionally, after determining the target storage location, the additional data may be uploaded to the target storage location in the archive storage system by using an interface between the archive storage system and the object storage system to complete archiving of the target additional data.

[0092] In the embodiment of the present application, the target storage location of the target additional data in the archival storage system is determined through the data storage location, so that the continuity of the additional data in the archival storage system can be ensured.

[0093] In order to ensure the data access efficiency and storage cost of the object storage system, based on the above embodiment, another optional method of data archiving is provided in this embodiment, such as Figure 3 As shown, the specific steps include:

[0094] S301, determining a data archiving time according to a delayed archiving period and a data uploading time of target additional data to an object storage system.

[0095] The delayed archiving period is a period for delayed archiving the additional data, and different delayed archiving periods can be configured for different types of additional write objects. In this embodiment, the delayed archiving period can be a preset number of delayed archiving days; the data upload time is the time when the object storage system receives the target additional data; and the data archiving time is the time when the target additional data is archived.

[0096] Optionally, in order to ensure the data access efficiency and storage cost of the object storage system, a delayed archiving period for storing the additional data in the object storage system can be pre-set; then, based on the delayed archiving period, the data upload time corresponding to the target additional data can be processed to obtain the data archiving time. For example, based on the data upload time corresponding to the target additional data, a delayed archiving number of days is accumulated to calculate the data archiving time of the target additional data.

[0097] S302: When the data archiving time is reached, the target additional data is stored in the target storage location in the archiving storage system.

[0098] Optionally, the current time of the object storage system may be compared with the data archiving time. If the times are consistent, the target additional data may be stored in the target storage location in the archiving storage system.

[0099] It can be understood that since there are multiple copies of additional data stored in the object storage system, a first-time scanning task can be uniformly configured to scan the data archiving time corresponding to each additional data; accordingly, the current time of the object storage system can be used as an index, and the first-time scanning task can be used to query the data archiving time corresponding to each additional data to obtain the current archived data whose data archiving time is consistent with the current time; then, the parallel upload method can be used to store the current archived data in the corresponding storage location in the archive storage system.

[0100] In an embodiment of the present application, by determining the data archiving time according to the delayed archiving period and the data upload time of the target additional data, and storing the target additional data in the archiving storage system at the data archiving time, the data access efficiency and storage cost of the object storage system can be guaranteed.

[0101] In the process of uploading data from the object storage system to the archive storage system, data loss or upload errors may occur. Based on this, in order to ensure the reliability of data archiving, based on the above embodiment, in this embodiment, an optional method for determining the reliability of data archiving is provided, such as Figure 4 As shown, the specific steps include:

[0102] S401: Acquire first data summary information of stored target additional data fed back by the archiving storage system.

[0103] The first data summary information is data summary information fed back by the archiving storage system based on the stored target additional data.

[0104] Optionally, after acquiring the target additional data, the archival storage system may generate first data summary information according to the acquired target additional data. For example, in the archival storage system, the message digest algorithm version 5 MD5 may be used to calculate the MD5 value corresponding to the target additional data, and the MD5 value may be fed back to the object storage system as the first data summary information.

[0105] S402: Determine second data summary information corresponding to the target additional data.

[0106] The second data summary information is data summary information generated by the object storage system based on the stored target additional data.

[0107] Optionally, before the object storage system uploads the target additional data to the archive storage system, the second data summary information corresponding to the target additional data may be determined, and the second data summary information may be stored in a corresponding position in the data archive record. For example, in the object storage system, MD5 may be used to calculate the MD5 value corresponding to the target additional data, and the MD5 value may be stored as the second data summary information in a corresponding position in the data archive record.

[0108] Correspondingly, after receiving the first data summary information, the second data summary information corresponding to the target additional data can be obtained at the corresponding position in the data archive record.

[0109] S403: Determine the data archiving reliability of the target additional data according to the consistency between the first data summary information and the second data summary information.

[0110] Optionally, the first data summary information and the second data summary information may be verified for consistency. If the consistency verification result is consistent, the data archiving reliability of the target additional data is determined to be reliable; if the consistency verification result is inconsistent, the data archiving reliability of the target additional data is determined to be unreliable.

[0111] When the data archiving reliability is unreliable, the target additional data needs to be archived again until the data archiving reliability is reliable.

[0112] For example, when the data archiving reliability is unreliable, a data clearing instruction may be sent to the archiving storage system to delete the erroneous target additional data in the archiving storage system; then, the object storage system re-uploads the target additional data to the archiving storage system. Alternatively, the newly uploaded target additional data may be directly overwritten on the erroneous target additional data by using a data overwriting method.

[0113] In the embodiment of the present application, by determining the data archiving reliability of the target additional data based on the consistency between the first data summary information and the second data summary information, the reliability of the data archiving can be ensured.

[0114] In order to ensure the reliability of determining the data storage status, a corresponding storage identifier may be configured for the append-write object according to the data storage situation. Based on this, on the basis of the above embodiment, in this embodiment, an optional method for determining the state of the append-write object is provided, which specifically includes the following steps:

[0115] According to the object identifier of the target additional write object, the storage identifier of the target additional write object is obtained; when the storage identifier is a hybrid storage identifier, it is determined that the target additional data of the target additional write object exists in the object storage system.

[0116] The storage identifier is used to represent the data storage status of the target append-write object; and the hybrid storage identifier is used to represent that the data of the target append-write object is stored in both the object storage system and the archive storage system.

[0117] Optionally, the object identifier of the target append write object can be used as an index to query in the object storage system to obtain the storage identifier of the target append write object; then, if it is determined that the storage identifier is a hybrid storage identifier, it proves that the data of the target append write object is stored in both the object storage system and the archive storage system, that is, the target append data of the target append write object exists in the object storage system.

[0118] If the storage identifier is determined to be an archive storage identifier, it proves that the data of the target additional write object are all stored in the archive storage system, that is, the target additional data of the target additional write object does not exist in the object storage system. The archive storage identifier is used to indicate that the data of the target additional write object are all stored in the archive storage system.

[0119] Accordingly, in order to prevent the target additional data from being archived repeatedly, after the target additional data has been stored in the archive storage system, the storage identifier is modified to the archive storage identifier. Specifically, when it is determined that the data archiving reliability of the target additional data is reliable, the storage identifier of the target additional write object needs to be modified from the hybrid storage identifier to the archive storage identifier.

[0120] It can be understood that configuring corresponding storage identifiers for additional write objects can not only ensure the reliability of determining the data storage status, but also directly skip the additional write objects whose storage identifier is the archive storage identifier when using the first-time scanning task to scan the data archiving moment of each additional data, thereby reducing the amount of data processing.

[0121] In the embodiment of the present application, the data storage state of the target additional write object is determined by the storage identifier of the target additional write object, so that the reliability of the determination of the data storage state can be ensured.

[0122] When the data of the appended write object is first uploaded to the object storage system, there is no data archive record of the appended write object in the object storage system. Based on this, on the basis of the above embodiment, in this embodiment, an optional method for creating a data archive record is provided, such as Figure 5A As shown, the specific steps include:

[0123] S501 : when a data archiving record of a target additional writing object is not obtained from an object storage system according to an object identifier of the target additional writing object, a data archiving record for the target additional writing object is created.

[0124] If the data archiving record of the target append write object is not obtained from the object storage system based on the object identifier of the target append write object, it proves that the data of the target append write object is uploaded to the object storage system for the first time. At this time, it is necessary to create a data archiving record for the target append write object based on the relevant information of the target append write object.

[0125] Exemplary, reference Figure 5B , a data archive record for a target append-write object can be created based on the metadata of the target append-write object, the access control list (object ACL) of the target append-write object, and the user metadata (object usermeta) related to the target append-write object.

[0126] S502: Determine a data storage location of the target additional write object in the archive storage system according to the data storage status of the archive storage system.

[0127] The data storage status may be data storage information of each archive bucket in the archive storage system.

[0128] Optionally, after creating the data archiving record for the target append-write object, the data storage status of the archive storage system can be obtained from the archive storage system; then, according to the remaining storage space of each archive bucket in the archive storage system, an archive bucket is allocated to the target append-write object. For example, the archive bucket with the largest remaining storage space can be allocated to the target append-write object.

[0129] Furthermore, the free storage location associated with the archive bucket to which the target append-write object is allocated may be used as the data storage location of the data of the target append-write object in the archive storage system.

[0130] S503, adding the data storage location to the data archiving record.

[0131] Optionally, after determining the data storage location of the target append write object's data in the archive storage system, the data storage location can be added to the data archive record, and then the first upload data of the target append write object can be uploaded to the archive storage system according to the data storage location.

[0132] In an embodiment of the present application, when there is no data archiving record of the target append write object in the object storage system, the reliability of the data archiving record construction can be ensured by creating a data archiving record for the target append write object and adding the determined data storage location to the data archiving record.

[0133] In order to ensure timely response to data read requests, based on the above embodiments, in this embodiment, an optional method for reading archived data is provided, specifically, in response to a data read request for target archived data in the archived data of the target append write object, the data archive record of the target append write object is obtained; according to the data archive record, the target archived data is obtained from the archive storage system, and a temporary data copy corresponding to the target archived data is constructed in the object storage system; based on the temporary data copy, the data read request is executed.

[0134] The target archived data is the archived data of the target append-write object that the user wants to read; the temporary data copy is the temporarily acquired data copy.

[0135] Optionally, after the object storage system detects a data read request for target archived data in archived data of the target append-write object, the data archive record of the target append-write object may be acquired based on the target append-write object identifier with reference to the above step S101.

[0136] It is understandable that since the target archived data is not stored in the object storage system, it is necessary to re-acquire the target archived data from the archived storage system to the object storage system. At this time, in order to facilitate the rapid response of data read requests after the archived data is restored to the object storage system, a temporary data copy corresponding to the archived data can be constructed.

[0137] It should be noted that what is obtained from the object storage system here is temporary target archive data, and the real target archive data is still stored in the archive storage system.

[0138] Optionally, the archive bucket where the target append-write object is located and the archive object name corresponding to the target archive data can be determined based on the target append-write object metadata and the archive metadata in the data archive record; then, the interface between the archive storage system and the object storage system can be used to obtain the target archive data from the corresponding archive bucket of the archive storage system according to the archive object name. For example, refer to Figure 1B , the GET interface between the archive storage system and the object storage system can be used to obtain the target archive data from the archive storage system to the object storage system.

[0139] Furthermore, after the object storage system obtains the target archive data, it can determine the MD5 value corresponding to the target archive data, and verify the consistency of the MD5 value with the second data summary information corresponding to the target archive data in the data archive record. If the consistency verification result is consistent, it is determined that the obtained target archive data is consistent with the target archive data in the archive storage system; if the consistency verification result is inconsistent, it is determined that the obtained target archive data is inconsistent with the target archive data in the archive storage system, and the target archive data needs to be re-acquired.

[0140] When the consistency verification result is consistent, a temporary data copy corresponding to the target archive data is constructed in the object storage system. At this time, the user can obtain the target archive data from the temporary data copy in the object storage system through the GET interface.

[0141] In this embodiment, in order to avoid repeatedly constructing a temporary data copy corresponding to the same archived data, after the temporary data copy corresponding to the target archived data is constructed, a restored identifier can be added in the data archive record and in the metadata of the target archived data; after the temporary data copy is deleted, the restored identifier needs to be deleted.

[0142] Accordingly, before constructing a temporary data copy of any archived data, it is necessary to read the metadata of the archived data, and if there is no restored mark in the metadata of the archived data, the archived data is obtained from the archive storage system.

[0143] In an embodiment of the present application, by constructing a temporary data copy corresponding to the target archived data in the object storage system to execute a data read request, the data read request can be responded to quickly, thereby improving the processing efficiency of the data read request.

[0144] In order to ensure the available storage space in the object storage system, based on the above embodiment, in this embodiment, an optional method for processing data copies is provided, specifically, according to the copy construction time of the temporary data copy, the copy deletion time is determined; when the copy deletion time is reached, it is determined whether the data read request is executed; if so, the temporary data copy in the object storage system is deleted.

[0145] The replica construction time is the time when the object storage system obtains the target archive data; the replica deletion time is the time when the temporary data copy is deleted.

[0146] Optionally, in order to facilitate the deletion of temporary data copies, when building a temporary data copy, it is necessary to configure a copy deletion time for the temporary data copy according to the copy building time. For example, a life cycle can be accumulated based on the copy building time to calculate the copy deletion time of the temporary data copy.

[0147] Subsequently, the replica build time can be compared with the replica deletion time. If the times are consistent, it can be determined whether the data read request has been completed based on the request execution progress. If so, the temporary data copy in the object storage system is directly deleted; if not, the replica deletion time can be updated until the data read request is completed.

[0148] It is understandable that since multiple temporary data copies may be stored in the object storage system at the same time, a second time scanning task for scanning the copy deletion time corresponding to each temporary data copy can be uniformly configured; accordingly, the current time of the object storage system can be used as an index, and the second time scanning task can be used to query the copy deletion time of each temporary data copy to obtain the current deleted copy whose data archiving time is consistent with the current time, and delete the temporary data copy.

[0149] Furthermore, after deleting the temporary data copy, if the user still wants to obtain the target archived data, it is necessary to repeat the above steps to rebuild the temporary data copy.

[0150] In an embodiment of the present application, by configuring a corresponding copy deletion time for each temporary data copy and deleting the temporary data copy in the object storage system when the copy deletion time is reached, the available capacity in the object storage system can be guaranteed.

[0151] Figure 6 FIG. 2 is a flow chart of a data processing method in another embodiment. Based on the above embodiment, this embodiment provides an optional example of a data processing method. Figure 6 The specific implementation process is as follows:

[0152] S601: When a data archiving record of a target additional writing object is not obtained from an object storage system according to an object identifier of the target additional writing object, a data archiving record for the target additional writing object is created.

[0153] The data archive record is used to record data information of archived data of the target append write object, and the archived data is data that has been stored in the archive storage system.

[0154] S602: Determine the data storage location of the target additional write object in the archiving storage system according to the data storage status of the archiving storage system, and add the data storage location to the data archiving record.

[0155] S603: When it is determined that the target additional data of the target additional write object exists in the object storage system, obtain a data archiving record of the target additional write object from the object storage system.

[0156] Optionally, according to the object identifier of the target additional write object, a storage identifier of the target additional write object is obtained; when the storage identifier is a hybrid storage identifier, it is determined that target additional data of the target additional write object exists in the object storage system; wherein the hybrid storage identifier is used to indicate that data of the target additional write object is stored in both the object storage system and the archive storage system;

[0157] S604: Determine a target storage location of the target additional data in the archiving storage system according to the data storage location.

[0158] S605: Determine the data archiving time according to the delayed archiving period and the data uploading time of the target additional data to the object storage system.

[0159] S606: When the data archiving time is reached, the target additional data is stored in the target storage location in the archiving storage system.

[0160] Optionally, first data summary information of the stored target additional data fed back by the archiving storage system is obtained; second data summary information corresponding to the target additional data is determined; and data archiving reliability of the target additional data is determined based on consistency between the first data summary information and the second data summary information.

[0161] S607: After the target additional data has been stored in the archive storage system, the data archive record is updated according to the data information of the target additional data, and the target additional data in the object storage system is deleted.

[0162] Optionally, after the target additional data has been stored in the archive storage system, the method further includes: modifying the storage identifier to an archive storage identifier; wherein the archive storage identifier is used to indicate that the data of the target additional write object is stored in the archive storage system.

[0163] S608 , in response to a data read request for target archived data in the archived data of the target appended write object, obtain a data archive record of the target appended write object.

[0164] S609, acquiring target archived data from the archive storage system according to the data archive record, and constructing a temporary data copy corresponding to the target archived data in the object storage system.

[0165] S610: Execute a data read request based on the temporary data copy.

[0166] Optionally, the copy deletion time is determined according to the copy construction time of the temporary data copy. When the copy deletion time is reached, it is determined whether the data read request is executed. If so, the temporary data copy in the object storage system is deleted.

[0167] The specific process of S601-S610 can refer to the description of the above method embodiment, and its implementation principle and technical effect are similar, which will not be repeated here.

[0168] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0169] Based on the same inventive concept, the embodiment of the present application also provides a data processing device for implementing the data processing method involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in the one or more data processing device embodiments provided below can refer to the limitations on the data processing method above, and will not be repeated here.

[0170] In an exemplary embodiment, Figure 7As shown, a data processing device 1 is provided, comprising: a record acquisition module 10, a data archiving module 20 and a record updating module 30, wherein:

[0171] The record acquisition module 10 is used to acquire the data archiving record of the target additional write object from the object storage system according to the object identifier of the target additional write object when it is determined that the target additional write object has target additional data in the object storage system; wherein the data archiving record is used to record data information of the archived data of the target additional write object, and the archived data is data that has been stored in the archiving storage system;

[0172] A data archiving module 20, for storing target additional data in an archiving storage system according to the data archiving record;

[0173] The record updating module 30 is used to update the data archiving record according to the data information of the target additional data after the target additional data has been stored in the archiving storage system, and delete the target additional data in the object storage system.

[0174] In an exemplary embodiment, the data information of the archived data includes a data storage location in the archive storage system; Figure 8 As shown, the data archiving module 20 includes:

[0175] A location determination unit 21, configured to determine a target storage location of the target additional data in the archiving storage system according to the data storage location;

[0176] The data archiving unit 22 is used to store the target additional data into the archiving storage system according to the target storage location.

[0177] In an exemplary embodiment, the data archiving unit 22 is specifically used for:

[0178] The data archiving time is determined according to the delayed archiving cycle and the data uploading time of the target additional data to the object storage system; when the data archiving time is reached, the target additional data is stored in the target storage location in the archiving storage system.

[0179] In an exemplary embodiment, the data processing device 1 further includes a verification module, wherein the verification module is specifically used to:

[0180] Acquire first data summary information of the stored target additional data fed back by the archiving storage system; determine second data summary information corresponding to the target additional data; and determine data archiving reliability of the target additional data based on consistency between the first data summary information and the second data summary information.

[0181] In an exemplary embodiment, the record acquisition module 10 is specifically used for:

[0182] According to the object identifier of the target append-write object, a storage identifier of the target append-write object is obtained; when the storage identifier is a hybrid storage identifier, it is determined that the target append-write object has target append-write data in the object storage system; wherein the hybrid storage identifier is used to indicate that the data of the target append-write object is stored in both the object storage system and the archive storage system;

[0183] Accordingly, after the target additional data has been stored in the archiving storage system, the record acquisition module 10 is further used to:

[0184] The storage identifier is changed to an archive storage identifier; wherein the archive storage identifier is used to indicate that the data of the target append write object is stored in the archive storage system.

[0185] In an exemplary embodiment, the record acquisition module 10 is further configured to:

[0186] If the data archiving record of the target append write object is not obtained from the object storage system based on the object identifier of the target append write object, create a data archiving record for the target append write object; determine the data storage location of the data of the target append write object in the archiving storage system based on the data storage situation of the archiving storage system; and add the data storage location to the data archiving record.

[0187] In an exemplary embodiment, the data processing device 1 further includes a data reading module, wherein the data reading module is specifically used for:

[0188] In response to a data read request for target archived data in the archived data of a target append write object, a data archive record of the target append write object is obtained; according to the data archive record, the target archived data is obtained from the archive storage system, and a temporary data copy corresponding to the target archived data is constructed in the object storage system; based on the temporary data copy, the data read request is executed.

[0189] In an exemplary embodiment, the data processing device 1 further includes a copy deletion module, wherein the copy deletion module is specifically used to:

[0190] The copy deletion time is determined according to the copy building time of the temporary data copy; when the copy deletion time is reached, it is determined whether the data read request is executed; if so, the temporary data copy in the object storage system is deleted.

[0191] Each module in the above data processing device can be implemented in whole or in part by software, hardware or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to each module.

[0192] In an exemplary embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as shown in FIG. Fig. 9 As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. Among them, the processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store additional write data. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a data processing method is implemented.

[0193] Those skilled in the art will understand that Fig. 9 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0194] In one embodiment, a computer device is further provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above method embodiments when executing the computer program.

[0195] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0196] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0197] It should be noted that the data involved in this application (including but not limited to additional write data, etc.) are all data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0198] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., but are not limited to this.

[0199] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0200] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A data processing method, characterized in that: The method comprises: In the case of determining that the target additional data of the target additional write object exists in the object storage system, obtaining a data archiving record of the target additional write object from the object storage system according to the object identifier of the target additional write object; wherein the data archiving record is used to record data information of the archived data of the target additional write object, and the archived data is data that has been stored in the archiving storage system; According to the data archiving record, storing the target additional data in the archiving storage system; After the target additional data has been stored in the archival storage system, the data archiving record is updated according to the data information of the target additional data, and the target additional data in the object storage system is deleted.

2. The method according to claim 1, characterized in that The data information of the archived data includes a data storage location in the archive storage system; The storing the target additional data into the archiving storage system according to the data archiving record includes: Determine, according to the data storage location, a target storage location of the target additional data in the archival storage system; According to the target storage location, the target additional data is stored in the archival storage system.

3. The method according to claim 2, characterized in that The storing the target additional data in the archival storage system according to the target storage location includes: Determining the data archiving time according to the delayed archiving period and the data uploading time of uploading the target additional data to the object storage system; When the data archiving time is reached, the target additional data is stored in the target storage location in the archiving storage system.

4. The method according to claim 3, characterized in that The method further comprises: Acquire first data summary information of the stored target additional data fed back by the archiving storage system; Determine second data summary information corresponding to the target additional data; The data archiving reliability of the target additional data is determined according to the consistency between the first data summary information and the second data summary information.

5. The method according to claim 1, characterized in that The step of determining that target additional data of a target additional write object exists in the object storage system includes: According to the object identifier of the target additional writing object, obtaining the storage identifier of the target additional writing object; In the case where the storage identifier is a hybrid storage identifier, determining that the target additional data of the target additional write object exists in the object storage system; wherein the hybrid storage identifier is used to indicate that the data of the target additional write object is stored in both the object storage system and the archive storage system; Accordingly, after the target additional data has been stored in the archival storage system, the method further includes: The storage identifier is modified to an archive storage identifier; wherein the archive storage identifier is used to indicate that the data of the target append-write object are all stored in the archive storage system.

6. The method according to claim 1, characterized in that The acquiring, from the object storage system, the data archiving record of the target append-write object according to the object identifier of the target append-write object comprises: If the data archiving record of the target additional writing object is not obtained from the object storage system according to the object identifier of the target additional writing object, creating a data archiving record for the target additional writing object; Determine the data storage location of the target additional write object in the archive storage system according to the data storage status of the archive storage system; The data storage location is added to the data archive record.

7. The method according to claim 1, characterized in that The method further comprises: In response to a data read request for target archived data in the archived data of the target append-write object, acquiring a data archive record of the target append-write object; According to the data archiving record, the target archived data is obtained from the archive storage system, and a temporary data copy corresponding to the target archived data is constructed in the object storage system; Based on the temporary data copy, the data read request is executed.

8. The method according to claim 7, characterized in that The method further comprises: Determining a copy deletion time according to a copy construction time of the temporary data copy; When the copy deletion time is reached, determining whether the data read request has been executed; If so, the temporary data copy in the object storage system is deleted.

9. A data processing device, characterized in that: The device comprises: A record acquisition module is used to acquire a data archiving record of the target additional write object from the object storage system according to the object identifier of the target additional write object when it is determined that the target additional write object has target additional data in the object storage system; wherein the data archiving record is used to record data information of archived data of the target additional write object, and the archived data is data that has been stored in the archiving storage system; A data archiving module, used for storing the target additional data into the archiving storage system according to the data archiving record; A record updating module is used to update the data archiving record and delete the target additional data in the object storage system according to the data information of the target additional data after the target additional data has been stored in the archiving storage system.

10. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 8 are implemented.

Images