Customized information and data synchronization methods, devices, and storage media for object storage

Abstract

This invention discloses a method, device, and storage medium for customizing object storage information and synchronizing data. The method includes: adding customized information to the object storage service; a user using distributed storage software to issue a task to a distributed storage software service; the distributed storage software service executing the corresponding task according to the task type and returning a response status to the user; when the response status is successful and the RGW request type of the data body is add, delete, or modify, an NSQ message is issued; the NSQ real-time messaging platform receives the NSQ message and pushes it to the distributed storage software NSQ message processing service; the distributed storage software NSQ message processing service calling the corresponding method to execute the task according to the request type of the NSQ message. This invention also enables data synchronization between the underlying layer and the database, ensuring synchronized storage of basic object information.

Description

Technical Field

[0001] This invention relates to distributed storage technology, specifically to a method, device, and storage medium for object storage customized information and data synchronization. Background Technology

[0002] Object storage is an object-centric storage method that stores data as objects rather than files. It has advantages such as high reliability, high scalability, and high performance.

[0003] A distributed storage appliance is an integrated device that combines distributed storage software and hardware, primarily used to address the needs of large-scale data storage and access. This type of device typically has the following characteristics:

[0004] High availability: Through data redundancy and failover mechanisms, we ensure high data availability and continuous system operation.

[0005] High performance: Utilizing parallel processing and high-speed networks, it provides fast data read and write speeds, making it suitable for large-scale data read and write operations.

[0006] Scalability: Supports horizontal scaling, meaning that storage capacity and performance can be increased by adding nodes.

[0007] Unified Management: Provides a unified management and monitoring interface, simplifying the management and maintenance of storage resources.

[0008] Multi-protocol support: Supports multiple data access protocols, such as NFS, CIFS, FTP, S3, etc., to meet the needs of different applications.

[0009] Data protection: Technologies such as RAID, data replication, and snapshots protect data from damage or loss. Distributed network storage involves distributing data across multiple independent devices. Traditional network storage systems use a centralized storage server to store all data, making the storage server a bottleneck for system performance and a focal point for reliability and security, failing to meet the needs of large-scale storage applications. Distributed network storage systems employ a scalable system architecture, utilizing multiple storage servers to share the storage load and location servers to locate stored information. This not only improves system reliability, availability, and access efficiency but also facilitates expansion.

[0010] Distributed object storage is a technology that distributes data across multiple physical locations (nodes). It is primarily used to handle large-scale data storage needs, especially in cloud computing, big data processing, and web applications. Unlike traditional file systems or database storage, object storage stores data in a flat structure, where each data item is treated as an "object," containing the data itself and associated metadata.

[0011] Ceph object storage provides a RESTful API interface, compatible with S3 and Swift. RGW (RADOS Gateway) is essentially a semantic translation layer; taking the S3 interface as an example, it translates S3 data read / write operations into read / write operations of objects within the RADOS cluster. For instance, when uploading a file, RGW primarily splits the file object into multiple RADOS cluster objects and saves them (though it's not that simple). When downloading the file, it reads these multiple RADOS cluster objects and returns them to the user. Therefore, a user might see a file mapped to a RADOS cluster in a 1:N relationship.

[0012] NSQ is a distributed real-time messaging platform based on the Go programming language. It features a distributed, decentralized topology and supports unlimited horizontal scaling. It boasts characteristics such as no single point of failure, fault tolerance, high availability, and guaranteed reliable message delivery. Furthermore, NSQ is very easy to configure and deploy, and supports numerous messaging protocols. It supports various clients and has a simple protocol.

[0013] When using distributed storage software, object storage's massive data storage capabilities lead to slow object data retrieval, and it lacks pagination and search functionality. Furthermore, if users create, modify, or delete object storage data using commands or third-party software, the distributed storage software cannot synchronize the data, resulting in data loss. Additionally, the fixed format of object data limits its scalability. Summary of the Invention

[0014] This invention addresses the shortcomings of existing technologies by providing a method, device, and storage medium for object storage customized information and data synchronization. It uses a database to store basic object storage information and distributed storage software customized information, while simultaneously enabling data synchronization between the underlying layer and the database, ensuring synchronized storage of basic object information.

[0015] To achieve the above objectives, the present invention adopts the following technical solution:

[0016] A method for customizing information and synchronizing data in object storage includes the following steps:

[0017] Add customized information to the object storage service;

[0018] Users send tasks to the distributed storage software service using distributed storage software. The distributed storage software service executes the corresponding tasks according to the task type and returns the response status to the user.

[0019] When the response status is successful and the RGW request type of the data body is add, delete or modify, an NSQ message is sent. The NSQ real-time messaging platform receives the NSQ message and pushes it to the distributed storage software NSQ message processing service.

[0020] The distributed storage software NSQ message processing service calls the corresponding method to execute tasks based on the request type of the NSQ message.

[0021] To optimize the above technical solution, the specific measures also include:

[0022] Furthermore, the specific steps for adding customized information to the object storage service are as follows:

[0023] The object user has added the fields des, userType, and createName for the created account, and the storage bucket has added the fields des and createName for the created account; the userType includes management software users, RGW underlying users, and virtualization users.

[0024] Furthermore, the distributed storage software service executes corresponding tasks according to the task type, wherein the task type is a new task, and the distributed storage software service executes the new task specifically as follows:

[0025] Upon receiving the parameters for a new task, the system first adds the object user, object sub-user, and bucket in the distributed storage software database. Then, based on the parameters of the new task, it executes the corresponding command for Ceph Object Storage RGW to create the corresponding object user, object sub-user, and bucket. After obtaining the successfully created underlying data, it updates the corresponding data in the distributed storage software database and returns a success status to the user. If Ceph Object Storage RGW fails to create the underlying data, it deletes the corresponding data in the distributed storage software database and finally returns a failure status to the user.

[0026] Furthermore, the distributed storage software service executes corresponding tasks according to the task type. In the case of an update task, the distributed storage software service executes the update task specifically as follows:

[0027] After receiving the update task parameters, the system first updates the object user, object sub-user, and storage bucket in the distributed storage software database. Then, it executes the corresponding command for Ceph object storage RGW according to the update task parameters to update the corresponding object user, object sub-user, and storage bucket. After obtaining the successfully updated data, it updates the corresponding data in the distributed storage software database and finally returns a success status to the user.

[0028] Furthermore, the distributed storage software service executes a corresponding task based on the task type. In this case, the task type is a deletion task, and the distributed storage software service executes the deletion task specifically as follows:

[0029] Upon receiving the parameters for the deletion task, the system first queries the distributed storage software database to check if the object user, object sub-user, and bucket exist. If they do not exist, the corresponding data in the distributed storage software database is deleted directly. If they exist, the system deletes the corresponding data from the corresponding table in the distributed storage software database according to the parameters of the deletion task. Then, the system executes the corresponding command at the Ceph object storage RGW underlying layer to delete the corresponding object user, object sub-user, and bucket. If the deletion is successful, a success status is returned to the user. If the deletion fails, the deleted data in the distributed storage software database is restored, and a failure status is returned to the user.

[0030] Furthermore, the request type of the NSQ message is "add data," and the specific steps for calling the corresponding method to execute the task are as follows:

[0031] After receiving the parameter to add data, the system first queries the distributed storage software database to see if the corresponding object user, object sub-user, and storage bucket already exist. If they exist, the system exits; otherwise, it creates the corresponding data in the distributed storage software database. Then, based on the parameter to add data, the system queries the underlying data in Ceph object storage RGW. After obtaining the corresponding data, it updates the newly created data in the distributed storage software database.

[0032] Furthermore, the NSQ message request type is "delete data," and the specific steps for calling the corresponding method to execute the task are as follows:

[0033] Upon receiving the data deletion parameters, the system first queries the Ceph object storage RGW to see if the object user, object sub-users, and bucket exist. If they exist, the system exits; otherwise, it deletes the corresponding data from the corresponding table in the distributed storage software database based on the data deletion parameters.

[0034] Furthermore, the request type of the NSQ message is "modify data," and the specific steps for calling the corresponding method to execute the task are as follows:

[0035] After receiving the parameters for modifying the data, the system first checks whether the corresponding object user, object sub-user, and storage bucket already exist in the distributed storage software database. If they do not exist, the system creates the corresponding data. Then, based on the parameters for modifying the data, the system queries the underlying data in Ceph object storage RGW. After obtaining the corresponding data, the system updates the corresponding data in the distributed storage software database.

[0036] The present invention also proposes an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, it implements the object storage customized information and data synchronization method described above.

[0037] The present invention also proposes a computer-readable storage medium storing a computer program that enables a computer to execute the object storage customized information and data synchronization method described above.

[0038] The beneficial effects of this invention are: this invention uses a database to store basic information of objects and distributed storage software to customize information, while simultaneously synchronizing data between the underlying layer and the database, which can ensure that the basic information of objects is stored synchronously. Attached Figure Description

[0039] Figure 1 This is an overall flowchart of the object storage customized information and data synchronization method proposed in this invention.

[0040] Figure 2 Customize information flow diagrams for distributed storage software.

[0041] Figure 3 This is a flowchart of the overall data synchronization process.

[0042] Figure 4 This is a diagram of the internal logic processing hierarchy of RGW.

[0043] Figure 5 This is a flowchart of the message processing service for the distributed storage software NSQ. Detailed Implementation

[0044] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0045] Example 1

[0046] This invention proposes a method for customizing information and synchronizing data in object storage. The overall process of this method is as follows: Figure 1 As shown, it mainly includes two parts: customized information for distributed storage software and data synchronization for distributed storage software object storage.

[0047] The distributed storage software used in this embodiment is CloudPan Distributed Storage Software, a fully distributed software-defined storage product for the enterprise market with large-scale horizontal scaling capabilities. It abstracts and pools existing physical storage resources through innovative software-defined storage technology, and uses algorithms to distribute data across multiple independent devices. CloudPan Distributed Storage Software provides a unified and concise user interface for managing storage resources, offering enterprises highly scalable, convenient, flexible, and cost-effective storage services, effectively meeting the on-demand storage needs of applications.

[0048] I. Customization Information for Cloudpan Distributed Storage Software:

[0049] like Figure 2 As shown, customized information has been added to the object storage service. For example, the object user has added fields such as des, userType (0 represents management software user, 1 represents RGW underlying user, 2 represents virtualization user, which can be extended), and createName of the created account. The storage bucket has added information such as des and createName of the created account.

[0050] When users submit new, update, or delete tasks on the Yunpan Distributed Storage Software website, the Yunpan Distributed Storage Software service will execute different methods depending on the task type.

[0051] 1. Add method:

[0052] The CreatUser (add object user), CreatSubuser (add object subuser), and CreatBucket (add bucket) implementation logic is as follows: Upon receiving parameters, the system first adds the object user, object subuser, and bucket to the Yunpan distributed storage software database. Then, based on the parameters, it executes the corresponding RGW (Ceph Object Storage) commands to create the corresponding object user, object subuser, and bucket. After successfully creating the data, it updates the corresponding database data. If the RGW fails to create the data, it deletes the corresponding database data. Finally, it returns a success or failure status to the user.

[0053] 2. Deletion method:

[0054] The `DeleteUser` (delete object user), `DeleteSubuser` (delete object subuser), and `DeleteBucket` (delete bucket) functions work as follows: Upon receiving parameters, the system first checks the database to see if the object user, object subuser, or bucket exists. If it doesn't exist, the corresponding data is directly deleted from the database. If it exists, the system deletes the corresponding data from the relevant table in the Cloudpan distributed storage software database based on the parameters. Then, it executes the corresponding command at the RGW underlying layer to delete the object user, object subuser, and bucket. If the deletion is successful, a success status is returned to the user. If the deletion fails, the deleted data is restored to the database, and a failure status is returned to the user.

[0055] 3. Modified method:

[0056] The ModifyUser (updates the object user), ModifySubuser (updates the object sub-user), and ModifyBucket (updates the bucket) functions work as follows: Upon receiving parameters, they first update the object user, object sub-user, and bucket in the Yunpan distributed storage software database. Then, based on the parameters, they execute the corresponding RGW (Ceph object storage) commands to update the corresponding object user, object sub-user, and bucket. After retrieving the successfully updated data, they update the corresponding database data. Finally, a success status is returned to the user.

[0057] II. Cloudpan Distributed Storage Software Object Storage Data Synchronization:

[0058] The overall data synchronization process is as follows: Figure 3 .

[0059] 1. RGW (Ceph Object Storage) NSQ task issuance:

[0060] like Figure 4 As shown, a conditional statement has been added to the RGW HTTP response layer code. When the response status is successful and the RGW request type in the data body is CRUD (Create, Delete, Update), the NSQ message delivery method `getNSQ` is triggered. The parameters of the `getNSQ` method are: `category` (task type: user object, subuser, bucket), `action` (request type: create, delete, modify), `user_name` (object user name), `subuser_name` (object user subuser name), `bucket_name` (bucket name), and `change_time` (current timestamp).

[0061] 2. NSQ Real-time Messaging Platform: Receives messages pushed from RGW and pushes them to the Cloudpan Distributed Storage Software in real time.

[0062] 3. NSQ message processing service of Yunpan distributed storage software:

[0063] like Figure 5 As shown, after receiving an NSQ message, the corresponding add, delete, and modify methods in userSouth, subuserSouth, and bucketSouth are called according to the message category (task type) and action (request type).

[0064] Add method:

[0065] The CreatUserSouth (add object user), CreatSubuserSouth (add object subuser), and CreatBucketSouth (add bucket) functions work as follows: Upon receiving parameters, they first query the CloudPan distributed storage software database to see if the corresponding object user, object subuser, and bucket already exist. If they do, the process exits; otherwise, they create the corresponding data. Then, based on the parameters, they query the underlying data in RGW (Ceph object storage), retrieve the corresponding data, and update the database data created in the previous step.

[0066] Deletion method:

[0067] The `DeleteUserSouth` (delete object user), `DeleteSubuserSouth` (delete object subuser), and `DeleteBucketSouth` (delete bucket) functions work as follows: Upon receiving parameters, they first query the RGW (Ceph object storage) to see if the object user, object subuser, and bucket exist. If they do, the process exits. If they do not exist, the corresponding data is deleted from the relevant table in the Yunpan distributed storage software database based on the parameters.

[0068] Modified method:

[0069] The `ModifyUserSouth` (update object user), `ModifySubuserSouth` (update object sub-user), and `ModifyBucketSouth` (update bucket) functions work as follows: Upon receiving parameters, they first query the Yunpan distributed storage software database to see if the corresponding object user, object sub-user, and bucket already exist. If not, they create the corresponding data. Then, based on the parameters, they query the underlying data in RGW (Ceph object storage), retrieve the corresponding data, and then update the corresponding database data.

[0070] Example 2

[0071] This invention proposes an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the object storage customized information and data synchronization method as described in Embodiment 1.

[0072] Example 3

[0073] This invention proposes a computer-readable storage medium storing a computer program that enables a computer to execute the object storage customized information and data synchronization method as described in Embodiment 1.

[0074] In the embodiments disclosed in this application, a computer storage medium may be a tangible medium that may contain or store programs for use by or in conjunction with an instruction execution system, apparatus, or device. The computer storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of computer storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0075] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed in this application can be implemented in electronic hardware or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0076] The above are merely preferred embodiments of the present invention. The scope of protection of the present invention is not limited to the above embodiments. All technical solutions falling within the scope of the present invention's concept are within the scope of protection of the present invention. It should be noted that for those skilled in the art, any improvements and modifications made without departing from the principles of the present invention should be considered within the scope of protection of the present invention.

Claims

1. A method for customizing information and synchronizing data in object storage, characterized in that, Includes the following steps: Add customization information to the object storage service; specifically, adding customization information to the object storage service includes: The object user has added the fields des, userType, and createName for the created account; the storage bucket has added the fields des and createName for the created account. The userType includes management software users, RGW underlying users, and virtualization users. Users submit tasks to the distributed storage software service using distributed storage software. The distributed storage software service executes the corresponding task based on the task type and returns a response status to the user. Specifically, when the task type is a new task, the distributed storage software service executes the new task as follows: Upon receiving the parameters for a new task, the system first adds an object user, object sub-user, and storage bucket in the distributed storage software database. Then, based on the parameters of the new task, it executes the corresponding command for Ceph Object Storage RGW to create the corresponding object user, object sub-user, and storage bucket. After obtaining the successfully created underlying data, it updates the corresponding data in the distributed storage software database and returns a success status to the user. If Ceph Object Storage RGW fails to create the underlying data, it deletes the corresponding data in the distributed storage software database and finally returns a failure status to the user. When the response status is successful and the RGW request type of the data body is add, delete or modify, an NSQ message is sent. The NSQ real-time messaging platform receives the NSQ message and pushes it to the distributed storage software NSQ message processing service. The distributed storage software NSQ message processing service calls the corresponding method to execute tasks based on the request type of the NSQ message.

2. The object storage customized information and data synchronization method as described in claim 1, characterized in that, The distributed storage software service executes corresponding tasks according to task type. In the case of an update task, the distributed storage software service executes the update task as follows: After receiving the update task parameters, the system first updates the object user, object sub-user, and storage bucket in the distributed storage software database. Then, it executes the corresponding command for Ceph object storage RGW according to the update task parameters to update the corresponding object user, object sub-user, and storage bucket. After obtaining the successfully updated data, it updates the corresponding data in the distributed storage software database and finally returns a success status to the user.

3. The object storage customized information and data synchronization method as described in claim 1, characterized in that, The distributed storage software service executes corresponding tasks based on task type. In the case of a deletion task, the distributed storage software service executes the deletion task as follows: Upon receiving the parameters for the deletion task, the system first queries the distributed storage software database to check if the object user, object sub-user, and bucket exist. If they do not exist, the corresponding data in the distributed storage software database is deleted directly. If they exist, the system deletes the corresponding data from the corresponding table in the distributed storage software database according to the parameters of the deletion task. Then, the system executes the corresponding command at the Ceph object storage RGW underlying layer to delete the corresponding object user, object sub-user, and bucket. If the deletion is successful, a success status is returned to the user. If the deletion fails, the deleted data in the distributed storage software database is restored, and a failure status is returned to the user.

4. The object storage customized information and data synchronization method as described in claim 1, characterized in that, The NSQ message request type is "add data," and the specific steps for calling the corresponding method to execute the task are as follows: After receiving the parameter to add data, the system first queries the distributed storage software database to see if the corresponding object user, object sub-user, and storage bucket already exist. If they exist, the system exits; otherwise, it creates the corresponding data in the distributed storage software database. Then, based on the parameter to add data, the system queries the underlying data in Ceph object storage RGW. After obtaining the corresponding data, it updates the newly created data in the distributed storage software database.

5. The object storage customized information and data synchronization method as described in claim 1, characterized in that, The NSQ message request type is "delete data," and the specific steps for calling the corresponding method to execute the task are as follows: Upon receiving the data deletion parameters, the system first queries the Ceph object storage RGW to see if the object user, object sub-users, and bucket exist. If they exist, the system exits; otherwise, it deletes the corresponding data from the corresponding table in the distributed storage software database based on the data deletion parameters.

6. The object storage customized information and data synchronization method as described in claim 1, characterized in that, The NSQ message request type is "modify data," and the specific steps for calling the corresponding method to execute the task are as follows: After receiving the parameters for modifying the data, the system first checks whether the corresponding object user, object sub-user, and storage bucket already exist in the distributed storage software database. If they do not exist, the system creates the corresponding data. Then, based on the parameters for modifying the data, the system queries the underlying data in Ceph object storage RGW. After obtaining the corresponding data, the system updates the corresponding data in the distributed storage software database.

7. An electronic device, characterized in that, include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, it implements the object storage customized information and data synchronization method as described in any one of claims 1-6.

8. A computer-readable storage medium storing a computer program, characterized in that, The computer program causes the computer to execute the object storage customized information and data synchronization method as described in any one of claims 1-6.

Images