Object storage multi-version data loading and consistency guaranteeing method

Abstract

The invention discloses an object storage multi-version data loading and consistency guaranteeing method, and relates to the technical field of information, and the method comprises the following steps: in object storage, distributing a unique version number for a data object, and when the data object is deleted or modified, generating a new data version and retaining data of a historical version. According to the method, the unique version number is allocated to the data object in the object storage, and the version number and the metadata of the data object are recorded and associated in the data loading process, so that the data object of the corresponding version can be accurately accessed by the subsequent data read-write operation; the mechanism effectively avoids the problem of intersection of new and old data caused by data change, and can significantly improve the accuracy and reliability of the data especially in application scenarios with strict requirements on data consistency, such as financial transactions and medical information management.

Description

Technical Field

[0001] The present invention relates to the field of information technology, and in particular to a method for loading and ensuring consistency of multi-version data in object storage. Background Art

[0002] With the rapid development of information technology, object storage, as an efficient and scalable data storage method, has been widely used in various business scenarios. Object storage systems store and manage data by encapsulating it in the form of objects. Each object contains the data itself and its related metadata. This storage method not only improves the efficiency of data access, but also facilitates the distributed storage and backup of data.

[0003] However, as business needs become increasingly complex, data management issues in object storage have become increasingly prominent. Especially in business scenarios where data changes frequently, how to ensure data consistency and integrity has become an urgent problem to be solved. Traditional object storage systems often use a simple data overwrite method, that is, when a data object is modified, the old version of the data is directly replaced. Although this method is simple and easy to implement, it has obvious disadvantages: once an error occurs in the modification operation or the data is accidentally deleted, it cannot be restored to the previous state, resulting in data loss or inconsistency.

[0004] To solve the above problems, the industry has begun to explore data management solutions based on object multi-version. This solution assigns a unique version number to each data object in the object storage. When the data object is modified or deleted, a new data version is generated and the historical version of the data is retained. In this way, even if the data changes, it can be rolled back to the previous version at any time, thus ensuring the consistency and integrity of the data.

[0005] However, existing multi-version data management solutions still have some shortcomings in data loading and consistency assurance. For example, during the data loading process, how to accurately record and transmit data version number information to ensure that subsequent data read and write operations can correctly access the corresponding version of the data object is a problem that needs to be solved; in addition, when the data objects in the object storage change, how to promptly update the version number information in the local cache to prevent data inconsistency caused by version mismatch is also a challenge that the current multi-version data management solution needs to face. Summary of the invention

[0006] Based on this, the purpose of the present invention is to provide an object storage multi-version data loading and consistency assurance method to solve the technical problems of inaccurate version number information recording and insufficient data consistency assurance during data loading in the prior art.

[0007] The first aspect of the present application provides an object storage multi-version data loading and consistency assurance method, comprising the following steps: Step A: In the object storage, a unique version number is assigned to the data object. When the data object is deleted or modified, a new data version is generated and the data of the historical version is retained. The version number of the data object is automatically generated by the object storage system and increases with each modification or deletion operation of the data object; Step B: During the data loading process, the data version number corresponding to the data loading time is recorded, and the version number is associated with the metadata of the data object and stored. The data version number at the data loading time is written into the file metadata as a key basis for subsequent data reading and writing operations; Step C: When performing data reading and writing operations, the corresponding version of the data object is obtained from the object storage according to the version number information in the metadata. The system accurately matches and accesses the corresponding version of the data object in the object storage according to the version number information in the metadata, ensuring Data consistency; Step D: If the data required for data reading and writing is not in the local cache, the version number information is transmitted to the object storage to obtain and load the corresponding version of the data object to the local cache; Step E: After the read and write operations are completed, the operation results are returned to the client, and the data status in the local cache is updated; Step F: Monitor the changes of data objects in the object storage. When it is detected that the data object is modified or deleted, update the version number information of the corresponding data object in the local cache, and mark the old version data as invalid. The monitoring process is implemented through a subscription mechanism. The subscription mechanism checks the changes of data objects in the object storage within a set time interval and triggers the corresponding update operation; Step G: In subsequent data reading and writing operations, the updated version number information is automatically used to obtain data from the object storage. Step H: When the data loading operation is completed, the current loading progress is recorded and saved in the management service so that the current progress can be continued the next time the data is loaded.

[0008] The second aspect of the present application provides an object storage multi-version data loading and consistency assurance system, including an object storage module, a data loading module, a metadata management module, a data reading and writing module and a version management module, wherein: the object storage module is used to store data objects and their version numbers; the data loading module is used to record the data version number during the data loading process, and store the version number in association with the metadata of the data object; the metadata management module is used to manage the metadata of the data object, including version number information; the data reading and writing module is used to obtain the corresponding version of the data object from the object storage according to the version number information in the metadata; the version management module is used to monitor the changes of the data objects in the object storage, update the version number information of the corresponding data objects in the local cache, and manage the version history of the data, and also includes a subscription mechanism module, which is used to check the changes of the data objects in the object storage within a set time interval, and trigger the version management module to perform the corresponding update operation.

[0009] In summary, the present invention mainly has the following beneficial effects:

[0010] The present invention assigns a unique version number to the data object in the object storage, and records and associates the version number with the metadata of the data object during the data loading process, thereby ensuring that subsequent data read and write operations can accurately access the corresponding version of the data object. This mechanism effectively avoids the problem of new and old data intersection caused by data changes, especially in application scenarios such as financial transactions and medical information management that have strict requirements on data consistency, and can significantly improve the accuracy and reliability of data;

[0011] During the data loading process, the present invention avoids unnecessary repeated reading and loading operations by recording the data version number at the time of data loading and obtaining data from the object storage according to the version number in subsequent read and write operations. This improvement not only saves valuable bandwidth resources, but also reduces network load and improves the efficiency of data loading and processing. This advantage is particularly obvious in scenarios of large-scale data processing and high concurrent access.

[0012] The present invention effectively reduces the impact of data fluctuations on the system through version management and data synchronization mechanisms. When the data object in the object storage changes, the system will promptly update the version number information in the local cache and mark the old version data as invalid, thereby ensuring the consistency and stability of subsequent data read and write operations. This feature is of great significance for maintaining the normal operation of the system and avoiding potential business risks.

[0013] In traditional solutions, data changes often require re-reading of data, which not only consumes a lot of time and resources, but also increases the complexity of data management. The present invention avoids repeated reading and tedious data update operations through version specification and asynchronous synchronization mechanism, simplifies the data management process, and improves the maintainability and ease of use of the system.

[0014] Since the present invention retains the historical versions of data objects, data traceability and auditing can be easily achieved, which is of great significance for meeting compliance requirements, resolving data disputes, and optimizing business processes. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 A flow chart of a method for loading and ensuring consistency of multi-version data stored in an object of the present invention;

[0016] Figure 2 This is an architecture diagram of the object storage multi-version data loading and consistency assurance system of the present invention. DETAILED DESCRIPTION

[0017] The following will be combined with the accompanying drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and cannot be understood as limiting the present invention.

[0018] The following describes an embodiment of the present invention based on its overall structure.

[0019] Example 1

[0020] A method for loading and ensuring consistency of multi-version data in object storage includes the following steps:

[0021] Step A: In object storage, a unique version number is assigned to a data object. When a data object is deleted or modified, a new data version is generated and the historical version of the data is retained. The version number of the data object is automatically generated by the object storage system and increases with each modification or deletion operation of the data object.

[0022] In object storage systems, in order to ensure the integrity, consistency and traceability of data, an innovative data management method is adopted, that is, assigning unique version numbers to data objects. The core of this mechanism is that whenever a data object undergoes a deletion or modification operation, the original data is not directly overwritten or deleted, but a new data version is generated on this basis, while retaining the historical version of the data.

[0023] Specifically, this process is automatically completed by the object storage system. The system assigns an initial version number to each newly created or imported data object. This version number is unique in the object storage and is used to identify the current state of the data object. As the business progresses, when a data object needs to be updated or modified, the user performs the corresponding operation. The object storage system will not directly replace the old version of the data, but will create a new data version and associate the new version of the data with a new, incremented version number. In this way, the old version of the data is retained, forming a historical version of the data.

[0024] Similarly, if a data object needs to be deleted, the object storage system will not immediately remove the data object from the storage. Instead, the system will generate a new version representing the deletion operation. This version may be a special mark or pointer to indicate that the data object has been deleted, but the original data still remains in the object storage, but is marked as invalid or inaccessible.

[0025] This data version management mechanism brings several significant advantages. First, it ensures the integrity and consistency of the data. Since the historical version of the data is retained, users can access the previous data status at any time, which is crucial for scenarios such as data auditing, compliance checking, or error recovery. Secondly, it improves the traceability of the data. Through the version number, users can easily track the history of data changes and understand how the data has evolved over time. Finally, this mechanism also provides greater flexibility for the business. For example, when you need to roll back to a previous data version or restore accidentally deleted data, the user can simply specify an older version number and then get the corresponding version of the data from the object storage.

[0026] Step B: During the data loading process, the data version number corresponding to the data loading time is recorded, and the version number is associated with the metadata of the data object and stored. The data version number at the data loading time is written into the file metadata as the key basis for subsequent data reading and writing operations.

[0027] First, when the data loading task is started, the system will immediately establish a connection with the object storage and prepare to load the specified data object. In this process, a key step is to obtain the current version number of the data object in the object storage. This version number is automatically generated by the object storage system. It increases with each modification or deletion operation of the data object, ensuring the uniqueness and timeliness of the version number.

[0028] Next, the system will associate and store this data version number with the metadata of the data object. Metadata is information that describes the basic properties of the data object, such as the structure, size, storage location, etc. of the data. By associating the version number with the metadata, we can quickly locate the correct data version in subsequent data reading and writing operations.

[0029] In particular, this data version number will be written into the file metadata. File metadata is information that describes the properties of the file itself. It is usually stored separately from the file content, but it is an indispensable part of the file. Writing the version number into the file metadata means that we record the corresponding data version at the file level, which greatly facilitates subsequent data management and access.

[0030] Step C: When performing data read and write operations, the corresponding version of the data object is obtained from the object storage according to the version number information in the metadata. The system accurately matches and accesses the corresponding version of the data object in the object storage according to the version number information in the metadata to ensure data consistency.

[0031] This step is the key to ensuring data consistency. The system can ensure that the correct version of the data object is accessed by accurately matching the version number information in the metadata and the data version in the object storage. In this way, even if the data object in the object storage is changed, the business side can obtain accurate data by accessing the corresponding version of the data object.

[0032] In addition, this mechanism also brings an important benefit, which is to avoid unnecessary repeated reading and loading operations. When reading and writing data, if the required data is already in the local cache, the cached data can be read directly; if it is not in the cache, the version number information can be transmitted to the object storage to obtain and load the corresponding version of the data object into the local cache. This saves valuable bandwidth resources and improves the efficiency of data loading and processing.

[0033] Step D: If the data required for data reading and writing is not in the local cache, the version number information is transmitted to the object storage to obtain and load the corresponding version of the data object into the local cache.

[0034] During data read and write operations, if the system finds that the required data is not stored in the local cache, this means that the data needs to be retrieved from the object storage.

[0035] To ensure that the data obtained is a correct and consistent version, the system will transmit the version number information required for the current operation to the object storage. This version number information is recorded during the data loading process and is associated with the metadata of the data object.

[0036] Object storage accurately matches and returns the data object of the corresponding version based on the received version number information. Then, the system loads this data object into the local cache so that subsequent operations can read data directly from the cache, improving access efficiency.

[0037] Through this process, the system can ensure that when reading and writing data, the correct and consistent version can be obtained regardless of whether the data is stored in the local cache, thereby ensuring the consistency and integrity of the data.

[0038] Step E: After the read and write operations are completed, the operation results are returned to the client and the data status in the local cache is updated.

[0039] After the read and write operations are completed, the system will return the results of the operations (such as the content of the data read, whether the write operation is successful, etc.) to the client that initiated the request. This is a key step to ensure that the client can correctly perceive the changes in data status.

[0040] In addition to returning the operation results, the system also updates the data status in the local cache based on the results of the operation. This means that if a write operation is performed, the newly written data will be stored in the local cache; if a read operation is performed and the read data is not in the local cache, the read data will be loaded into the local cache for subsequent quick access. At the same time, the data version number in the local cache will also be updated according to the results of the operation to ensure that subsequent data read and write operations can be performed based on the latest data version.

[0041] Step F: Monitor changes in data objects in the object storage. When a modification or deletion of a data object is detected, update the version number information of the corresponding data object in the local cache and mark the old version data as invalid. The monitoring process is implemented through a subscription mechanism, which checks changes in data objects in the object storage within a set time interval and triggers corresponding update operations.

[0042] The subscription mechanism is a way to actively obtain notifications of data changes. It will regularly check the status of data objects in the object storage at a set time interval. This time interval is usually short to ensure that you can be notified quickly after the data changes. When the subscription mechanism detects that the data object has been modified or deleted, it will trigger a series of update operations. First, the system will update the version number information of the corresponding data object in the local cache, which means that the version number in the local cache will be synchronized with the latest version number in the object storage. Then, the system will mark the old version of the data as invalid. This is to avoid using outdated data in subsequent data read and write operations.

[0043] Step G: In subsequent data read and write operations, the updated version number information is automatically used to obtain data from the object storage.

[0044] Specifically, when the system needs to read or write data, it will first check whether the required data is in the local cache. If the data is in the cache, it will directly read or write the data in the cache; if the data is not in the cache, the system will obtain the corresponding version of the data object from the object storage based on the version number information in the metadata, and load it into the local cache. Since the system has updated the version number information in the local cache, it can ensure that the latest version of the data object is obtained from the object storage.

[0045] Step H: When the data loading operation is completed, the current loading progress is recorded and saved in the management service so that the current progress can be continued the next time the data is loaded.

[0046] During the data loading process, the system will monitor and record the amount of data that has been loaded or the specific location to which it has been loaded in real time or in stages. This progress information usually includes the identifier, offset or some form of index of the loaded data to accurately indicate which step the loading has reached. The recorded loading progress information will not only remain locally or in the loading process, but will be securely saved to a dedicated management service. This management service may be a centralized database or a distributed storage system. Its core function is to ensure that the loading progress information will not be lost even if the loading process is unexpectedly interrupted (such as system crash, network disconnection, etc.).

[0047] When data needs to be reloaded (perhaps because the loading process is interrupted or the user wants to continue from where it was last loaded), the system will directly locate the location where the last load stopped based on the progress information saved in the management service and continue loading from there. This not only saves time and resources for repeatedly loading processed data, but also significantly improves loading efficiency and user experience.

[0048] Example 2

[0049] The object storage multi-version data loading and consistency guarantee system includes an object storage module, a data loading module, a metadata management module, a data reading and writing module, and a version management module, among which:

[0050] The object storage module is used to store data objects and their version numbers;

[0051] The data loading module is used to record the data version number during the data loading process and store the version number in association with the metadata of the data object;

[0052] The metadata management module is used to manage the metadata of data objects, including version number information;

[0053] The data reading and writing module is used to obtain the corresponding version of the data object from the object storage according to the version number information in the metadata;

[0054] The version management module is used to monitor changes in data objects in the object storage, update the version number information of the corresponding data objects in the local cache, and manage the version history of the data.

[0055] It also includes a subscription mechanism module, which is used to check the changes of data objects in the object storage within a set time interval and trigger the version management module to perform corresponding update operations.

[0056] In summary, the system achieves the goal of object storage multi-version data loading and consistency assurance by integrating multiple functional modules such as object storage module, data loading module, metadata management module, data reading and writing module, version management module and subscription mechanism module.

[0057] Although an embodiment of the present invention has been shown and described, this specific embodiment is merely an explanation of the present invention and is not a limitation of the invention. The specific features, structures, materials or characteristics described may be combined in a suitable manner in any one or more embodiments or examples. After reading this specification, those skilled in the art may make modifications, substitutions and variations to the embodiments without creative contributions as needed without departing from the principles and purpose of the present invention. However, as long as they are within the scope of the claims of the present invention, they are protected by patent law.

Claims

1. A method for loading and ensuring consistency of multi-version data in object storage, characterized by: The following steps are involved: Step A: In object storage, a unique version number is assigned to the data object. When the data object is deleted or modified, a new data version is generated and the historical version of the data is retained. Step B: During the data loading process, the data version number corresponding to the data loading moment is recorded, and the version number is associated with the metadata of the data object and stored; Step C: When performing data read and write operations, obtain the corresponding version of the data object from the object storage according to the version number information in the metadata; Step D: If the data required for data reading and writing is not in the local cache, the version number information is transmitted to the object storage to obtain and load the corresponding version of the data object into the local cache; Step E: After the read and write operations are completed, the operation results are returned to the client and the data status in the local cache is updated.

2. The object storage multi-version data loading and consistency assurance method according to claim 1 is characterized by: In step A, the version number of the data object is automatically generated by the object storage system and increases with each modification or deletion operation of the data object.

3. The object storage multi-version data loading and consistency assurance method according to claim 1 is characterized by: In step B, the data version number at the time of data loading is written into the file metadata as a key basis for subsequent data reading and writing operations.

4. The object storage multi-version data loading and consistency assurance method according to claim 1 is characterized by: In step C, the system accurately matches and accesses the data object of the corresponding version in the object storage based on the version number information in the metadata to ensure data consistency.

5. The object storage multi-version data loading and consistency assurance method according to claim 1 is characterized by: The following steps are also included: Step F: Monitor changes in data objects in the object storage. When a data object is detected to be modified or deleted, update the version number information of the corresponding data object in the local cache and mark the old version data as invalid. Step G: In subsequent data read and write operations, the updated version number information is automatically used to obtain data from the object storage.

6. The object storage multi-version data loading and consistency assurance method according to claim 5 is characterized by: The monitoring process in step F is implemented through a subscription mechanism, which checks the changes of data objects in the object storage within a set time interval and triggers corresponding update operations.

7. The object storage multi-version data loading and consistency assurance method according to claim 1 is characterized by: The following steps are also included: Step H: When the data loading operation is completed, the current loading progress is recorded and saved in the management service so that the current progress can be continued the next time the data is loaded.

8. An object storage multi-version data loading and consistency assurance system using the method according to any one of claims 1 to 8, characterized in that: It includes object storage module, data loading module, metadata management module, data reading and writing module and version management module, among which: The object storage module is used to store data objects and their version numbers; The data loading module is used to record the data version number during the data loading process and store the version number in association with the metadata of the data object; The metadata management module is used to manage the metadata of data objects, including version number information; The data reading and writing module is used to obtain the corresponding version of the data object from the object storage according to the version number information in the metadata; The version management module is used to monitor changes in data objects in the object storage, update the version number information of the corresponding data objects in the local cache, and manage the version history of the data.

9. The system method according to claim 8, characterized in that: It also includes a subscription mechanism module, which is used to check the changes of data objects in the object storage within a set time interval and trigger the version management module to perform corresponding update operations.

Images