A data loading implementation method based on object storage

Abstract

This invention discloses a data loading method based on object storage. Data loading is established by directory, and each data loading directory can map to an object bucket. After establishing a connection, data can be loaded using different loading strategies: preheating or read / write-triggered loading. After each loading, a marker for the current progress is recorded and stored in the management service. The next loading will continue from the current progress. File storage can detect changes in data in object storage and can specify different strategies to update changed data. When the lifecycle of data in a directory is completed, the data can be exported to object storage. This invention's data loading based on object storage can automatically synchronize data between object storage and file storage. Data can be imported from object storage to file storage and vice versa, always maintaining centralized data storage in object storage.

Description

Technical Field

[0001] This invention relates to the field of data loading, and in particular to a data loading implementation method based on object storage. Background Technology

[0002] With the rise of emerging businesses, the volume of unstructured data is growing exponentially year by year, originating from different businesses or teams. This has resulted in data silos or a lack of connectivity between data points. Data lakes emerged in this context to solve the data silo problem. Object storage is an excellent choice, offering both affordability and reliability while supporting massive amounts of data.

[0003] Data lakes need to support a variety of computing engines to meet diverse business needs. However, file storage and object storage are not interoperable; data cannot be automatically loaded from object storage to file storage, and different storage type interfaces such as POSIX and S3API cannot be satisfied, hindering the flexible flow of data between different storage systems and different clouds. The current common solution is manual (script-based) copying, where users manually copy data from object storage to file storage before use. This process has low automation, requires full human intervention, increases operational risks and implementation costs, and necessitates manual copying back to object storage after use.

[0004] To address this, we propose a data loading method based on object storage. Summary of the Invention

[0005] The technical problem to be solved by the present invention is to overcome the defects of the prior art and provide a data loading implementation method based on object storage.

[0006] To solve the above-mentioned technical problems, the present invention provides the following technical solution:

[0007] This invention provides a data loading method based on object storage, comprising the following steps:

[0008] Step 1: After receiving the import message, the Agent service adds the relevant information of the task, such as agent id, bucket information, and perload attribute, to the task list of the management node, and the management node then assigns the task.

[0009] Step 2: The management node receives the creation command, creates a data loading task, adds it to the task list of the management service, the newly created task status is new, and the management service assigns the corresponding agent ID when creating it, and persists it on the management service;

[0010] Step 3: The Agent service periodically retrieves data loading tasks corresponding to its own node's agnet ID from the management service, obtains agentid, marker, perload, and status, executes the task according to the corresponding command, and marks the task status as import / export according to the type.

[0011] Step 4: The proxy service retrieves the object list, retrieving 1000 object records each time, distinguishing between files and directories, and sending a message to the file storage metadata service to create files and directories. If it is creating a file, it needs to return the corresponding pattern and marker from the file storage metadata service. If the retrieval fails, the task status is marked as failed and the process exits. If the list is empty, it means that the loading is complete, and the task status is marked as finished and the process exits.

[0012] Step 5: The management service writes the marker into the bucket information and persists it to the management service. If the retrieval fails, the task status is marked as failed and the process exits.

[0013] Step 6: If Perload=true, the proxy service sends a GetRange request to the file storage data service corresponding to the pattern to the object storage, and pulls the data from the object storage to write it to the file storage. If Perload=false, it will not read data from the object storage. If the retrieval fails, the task status will be marked as failed and the process will exit.

[0014] Step 7: The proxy service sends an update dentry to the metadata service where the file is located, updating the file's location information to the file storage layer. If the retrieval fails, the task is marked as failed and the process exits.

[0015] Step 8: Once a batch of tasks has been loaded, return to step 4.

[0016] As a preferred embodiment of the present invention, the key terms in this method are explained as follows:

[0017] S3: Object storage;

[0018] Bucket: Object storage bucket;

[0019] Object: an object;

[0020] Agent: Agency services;

[0021] Agent ID: Multiple agent services, each with its own unique identifier.

[0022] Mgr: Management service, the management node of the file storage service, which persists the necessary information for file storage;

[0023] Marker: Indicates the starting position of the next listobjects. Each loop will start from the position corresponding to the mark in s3.

[0024] Perload: Indicates whether data needs to be imported at the same time. True means import all data at the same time, while false means only the corresponding metadata is created during import, and the data will be read from s3 when it is actually read.

[0025] Status: Indicates whether the current status is import, import means importing, export means exporting, finish means completion, and failed means failure.

[0026] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0027] 1. This invention allows business access to be completed after only the metadata has been loaded. The data will be read locally during the business process, which will greatly shorten the business access time.

[0028] 2: This invention is based on an object loading scheme that only loads metadata. It can load on demand when the actual business is running, only loading the locations that have been read and written, saving file storage space. Object storage has the advantages of low price and large capacity, while local high-performance file storage has the advantages of high price and high performance. The data loading scheme can balance performance and capacity, and the loading speed can also be balanced.

[0029] 3: This invention is based on object storage for data loading and can automatically synchronize data between object storage and file storage. Data can be imported from object storage to file storage and vice versa, always keeping data centrally stored in object storage. It also has flexible storage interface adaptability, requires no human intervention, and ensures data consistency and reliability.

[0030] 4. This invention enables data loading to migrate quickly between clouds, easily achieving cross-cloud and hybrid cloud operations. Since the data is centrally stored in object storage, the files are deleted after use, achieving cost savings. Attached Figure Description

[0031] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings:

[0032] Figure 1 This is a flowchart of the data loading process of the present invention;

[0033] Figure 2 This is a flowchart illustrating the data loading process of the present invention;

[0034] Figure 3 This is a comparison chart of the loading time of the object storage BOS-bcecmd download tool in the embodiments; Detailed Implementation

[0035] The preferred embodiments of the present invention will be described below with reference to the accompanying drawings. It should be understood that the preferred embodiments described herein are for illustration and explanation only and are not intended to limit the present invention.

[0036] Example 1

[0037] like Figure 1-3 As shown, this invention provides a data loading implementation method based on object storage, including the following steps:

[0038] Step 1: After receiving the import message, the Agent service adds the relevant information of the task, such as agent id, bucket information, and perload attribute, to the task list of the management node, and the management node then assigns the task.

[0039] Step 2: The management node receives the creation command, creates a data loading task, adds it to the task list of the management service, the newly created task status is new, and the management service assigns the corresponding agent ID when creating it, and persists it on the management service;

[0040] Step 3: The Agent service periodically retrieves data loading tasks corresponding to its own node's agnet ID from the management service, obtains agentid, marker, perload, and status, executes the task according to the corresponding command, and marks the task status as import / export according to the type.

[0041] Step 4: The proxy service retrieves the object list, retrieving 1000 object records each time, distinguishing between files and directories, and sending a message to the file storage metadata service to create files and directories. If it is creating a file, it needs to return the corresponding pattern and marker from the file storage metadata service. If the retrieval fails, the task status is marked as failed and the process exits. If the list is empty, it means that the loading is complete, and the task status is marked as finished and the process exits.

[0042] Step 5: The management service writes the marker into the bucket information and persists it to the management service. If the retrieval fails, the task status is marked as failed and the process exits.

[0043] Step 6: If Perload=true, the proxy service sends a GetRange request to the file storage data service corresponding to the pattern to the object storage, and pulls the data from the object storage to write it to the file storage. If Perload=false, it will not read data from the object storage. If the retrieval fails, the task status will be marked as failed and the process will exit.

[0044] Step 7: The proxy service sends an update dentry to the metadata service where the file is located, updating the file's location information to the file storage layer. If the retrieval fails, the task is marked as failed and the process exits.

[0045] Step 8: Once a batch of tasks has been loaded, return to step 4.

[0046] The key terms used in the above method are explained as follows:

[0047] S3: Object storage;

[0048] Bucket: Object storage bucket;

[0049] Object: an object;

[0050] Agent: Agency services;

[0051] Agent ID: Multiple agent services, each with its own unique identifier.

[0052] Mgr: Management service, the management node of the file storage service, which persists the necessary information for file storage;

[0053] Marker: Indicates the starting position of the next listobjects. Each loop will start from the position corresponding to the mark in s3.

[0054] Perload: Indicates whether data needs to be imported at the same time. True means import all data at the same time, while false means only the corresponding metadata is created during import, and the data will be read from s3 when it is actually read.

[0055] Status: Indicates whether the current status is import, import means import, export means export, finish means finished, and failed means failed;

[0056] Import: Establish corresponding metadata in file storage that corresponds to that in object storage. Data is then loaded from object storage to file storage using the metadata information.

[0057] Export: Create metadata on the object storage that corresponds to that on the file storage. Data is loaded from the file storage to the object storage through the metadata information.

[0058] Specifically, data loading is organized by directory, with each data loading directory mapping to an object bucket. After establishing a connection, data can be loaded using different strategies: preheating or read / write-triggered loading. After each load, a progress marker is recorded and stored in the management service, allowing the next load to continue from the current progress. File storage can detect changes to data in object storage and can specify different strategies to update changed data. When data in a directory completes its lifecycle, it can be exported to object storage.

[0059] Object storage cannot support high-performance storage requirements, so when using data, it is necessary to load the entire data locally first. With large amounts of data, a complete load can take a very long time. However, this method allows for business integration after only loading the metadata; the data is read locally during the business process, significantly reducing the business integration time. Figure 3 (Compare this to the object storage BOS-bcecmd download tool).

[0060] The object loading scheme based on loading only metadata can perform on-demand loading when the actual business is running, loading only the locations that have been read and written, saving file storage space. Object storage has the advantages of low price and large capacity, while local high-performance file storage has the advantages of high price and high performance. The data loading scheme can balance performance and capacity, and loading speed can also be balanced.

[0061] Data loading based on object storage can automatically synchronize data between object storage and file storage. Data can be imported from object storage to file storage, and vice versa. Data is always centrally stored in object storage. It also has flexible storage interface adaptability, requires no human intervention, and ensures data consistency and reliability.

[0062] Data loading can be quickly migrated between clouds, easily achieving cross-cloud and hybrid cloud operations. Since the data is centrally stored in object storage (data lake), the files are deleted after use, achieving cost savings.

[0063] Finally, it should be noted that the above descriptions are merely preferred embodiments of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A data loading implementation method based on object storage, characterized in that, Includes the following steps: Step 1: After receiving the import message, the Agent service adds the relevant information of the task, such as agent id, bucket information, and perload attribute, to the task list of the management node, and the management node then assigns the task. Step 2: The management node receives the creation command, creates a data loading task, adds it to the task list of the management service, the newly created task status is new, and the management service assigns the corresponding agent ID when creating it, and persists it on the management service; Step 3: The Agent service periodically retrieves data loading tasks corresponding to its own node's agnet ID from the management service, obtains agentid, marker, perload, and status, executes the task according to the corresponding command, and marks the task status as import / export according to the type. Step 4: The proxy service retrieves the object list, retrieving 1000 object records each time, distinguishing between files and directories, and sending a message to the file storage metadata service to create files and directories. If it is creating a file, it needs to return the corresponding pattern and marker from the file storage metadata service. If the retrieval fails, the task status is marked as failed and the process exits. If the list is empty, it means that the loading is complete, and the task status is marked as finished and the process exits. Step 5: The management service writes the marker into the bucket information and persists it to the management service. If the retrieval fails, the task status is marked as failed and the process exits. Step 6: If Perload=true, the proxy service sends a GetRange request to the file storage data service corresponding to the pattern to the object storage, and pulls the data from the object storage to write it to the file storage. If Perload=false, it will not read data from the object storage. If the retrieval fails, the task status will be marked as failed and the process will exit. Step 7: The proxy service sends an update dentry to the metadata service where the file is located, updating the file's location information to the file storage layer. If the retrieval fails, the task is marked as failed and the process exits. Step 8: Once a batch of tasks has been loaded, return to step 4.

2. The data loading method based on object storage according to claim 1, characterized in that, The key terms used in this method are explained as follows: S3: Object storage; Bucket: Object storage bucket; Object: an object; Agent: Agency services; Agent ID: Multiple agent services, each with its own unique identifier. Mgr: Management service, the management node of the file storage service, which persists the necessary information for file storage; Marker: Indicates the starting position of the next listobjects. Each loop will start from the position corresponding to the mark in s3. Perload: Indicates whether data needs to be imported at the same time. True means import all data at the same time, while false means only the corresponding metadata is created during import, and the data will be read from s3 when it is actually read. Status: Indicates whether the current status is import, import means importing, export means exporting, finish means completion, and failed means failure.

Images