Bucket data management method and device, computer equipment and storage medium

Through distributed server clusters and object storage gateways collect and manage bucket data in object storage systems, the problems of poor timeliness and limited storage capacity in traditional technologies are solved, and efficient management and integrity of bucket data are achieved.

CN119065789APending Publication Date: 2024-12-03JINAN INSPUR DATA TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411127230.4
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-16
Publication Date
2024-12-03

AI Technical Summary

Technical Problem

Traditional object storage systems have poor timeliness in bucket information management, and the capacity of historical performance data is stored on the underlying data shard is limited, making it difficult to effectively manage and query.

Method used

A distributed server cluster is adopted to collect bucket data through an object storage gateway, store bucket data in shards and synchronize it to the management system module, and the master node pulls data from shards and databases to realize real-time update and verification of data.

Benefits of technology

It improves the timeliness and completeness of bucket data, solves the problems of poor timeliness and limited storage capacity in traditional technologies, and ensures the accuracy and consistency of data management.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119065789A_ABST
    Figure CN119065789A_ABST
Patent Text Reader

Abstract

The invention relates to a storage bucket data management method and device, computer equipment and a storage medium. The method is applied to a distributed server cluster, the distributed server cluster comprises a master node and a slave node, the slave node is provided with a management system module, an object storage gateway and a rados fragment, the slave node collects bucket data through the object storage gateway, the bucket data comprises bucket attribute information and historical performance data, and the bucket attribute information comprises bucket attribute information. The slave node reports bucket attribute information to the master node, the slave node stores historical performance data into a rados fragment through the object storage gateway, transmits the historical performance data to the management system module, and stores the historical performance data into a database of the slave node through the management system module; and the master node obtains the historical performance data from the database of the slave node and the rados fragment, and updates the historical performance data and the bucket attribute information to the database of the master node. By adopting the method, the timeliness and integrity of storage bucket data collection can be improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a storage bucket data management method, apparatus, computer equipment and storage medium. Background Art

[0002] Object storage service is an emerging cloud storage service that is massive, secure, low-cost and highly reliable, suitable for storing any type of files. The basic data entities of an object storage system include users, buckets and objects, and objects are directly stored in buckets. As the number of buckets increases, the interface faces great challenges in managing bucket information, including query timeliness, sorting by key fields, and bucket information data storage.

[0003] In traditional technical solutions, historical performance data is stored on the underlying data shards, which has limited storage capacity and the interface directly interacts with rados commands, resulting in poor timeliness. Summary of the invention

[0004] Based on this, it is necessary to provide a bucket data management method, device, computer equipment and storage medium that can improve the timeliness and integrity of bucket data collection in response to the above technical problems.

[0005] A storage bucket data management method is applied to a distributed server cluster, the distributed server cluster includes a master node and a slave node, the slave node is provided with a management system module, an object storage gateway and a rados shard, and the storage bucket data management method includes:

[0006] The slave node collects bucket data through the object storage gateway. The bucket data includes bucket attribute information and historical performance data.

[0007] The slave node reports the bucket attribute information to the master node;

[0008] The slave node stores the historical performance data in the rados shard through the object storage gateway, and transmits the historical performance data to the management system module, which stores the historical performance data in the database of the slave node through the management system module;

[0009] The master node obtains historical performance data from the slave node's database and rados shards, and updates the historical performance data and bucket attribute information to the master node's database.

[0010] In one embodiment, the object storage gateway of the slave node is provided with an application programming interface, and the master node obtains historical performance data from the database of the slave node and the rados shard, including:

[0011] The master node obtains historical performance data within a first preset time from a database of the slave node;

[0012] The master node calls the application programming interface of the slave node to obtain historical performance data within a second preset time from the rados shard, and the second preset time is less than the first preset time.

[0013] In one embodiment, the bucket attribute information includes the number of buckets, the bucket identifier, and the bucket modification time, and the bucket data management method further includes:

[0014] The master node obtains first-category bucket attribute information and second-category bucket attribute information respectively, wherein the first-category bucket attribute information is obtained from a database of the master node, and the second-category bucket attribute information is obtained by calling an application programming interface;

[0015] The master node determines the difference in the number of buckets according to the number of buckets in the first type of bucket attribute information and the number of buckets in the second type of bucket attribute information;

[0016] The master node compares the first type of bucket attribute information with the storage buckets with the same identifier in the second type of bucket attribute information, compares the bucket modification time, and obtains the number of buckets with different bucket modification times;

[0017] After the performance data and bucket attribute information are updated to the database of the master node, the bucket data management method further includes:

[0018] When the difference in the number of buckets and the sum of the number of buckets with different bucket modification times is greater than the preset value, the master node updates all the second-category bucket attribute information to the master node's database;

[0019] When the difference in the number of buckets and the sum of the number of buckets with different bucket modification times is less than a preset value, the master node updates the newly added data in the second type of bucket attribute information to the database of the master node.

[0020] In one embodiment, the application programming interface is a rados interface.

[0021] In one embodiment, the bucket data also includes real-time performance data, and the real-time performance data is collected from the node, including:

[0022] Start the scheduled task from the node and obtain the cluster deployment type in the preset configuration file. When the cluster deployment type is object storage, run the scheduled task to collect real-time performance data.

[0023] Bucket data management methods also include:

[0024] The master node calls the application programming interface to obtain real-time performance data.

[0025] In one embodiment, the slave node reports the bucket attribute information to the master node, including:

[0026] The object storage gateway of the slave node assembles the bucket attribute information and reports the assembled data to the management system module of the slave node. The assembled data includes the operation type.

[0027] The management system module of the slave node parses the assembled data, obtains the operation type, and maps the operation type to a pre-set operation code;

[0028] The slave node starts the sending thread and determines whether the destination node is the master node and whether it is reachable through the sending thread. If the destination node is the master node and reachable, the operation code and the destination node are encapsulated into the message header of the message queue, the bucket attribute information is encapsulated into the message body, and the message is reported to the master node through the message queue;

[0029] Bucket data management methods also include:

[0030] When the object storage gateway of the slave node fails to report, the object storage gateway of the slave node stores the bucket attribute information that failed to be reported in the cache queue so as to re-report the bucket attribute information that failed to be reported.

[0031] In one embodiment, updating the historical performance data and bucket attribute information to the database of the master node includes:

[0032] The master node parses the operation code to obtain the operation type. When the operation type includes more than one operation, each operation is encapsulated into the same database operation event, and the historical performance data and bucket attribute information are updated to the master node's database according to the database operation event.

[0033] A distributed server cluster includes a master node and a slave node. The slave node is provided with a management system module, an object storage gateway and a rados shard, wherein:

[0034] The slave node is used to collect bucket data through the object storage gateway. The bucket data includes bucket attribute information and historical performance data.

[0035] The slave node is also used to report bucket attribute information to the master node, store historical performance data in the rados shard through the object storage gateway, and pass the historical performance data to the management system module, which stores the historical performance data in the database of the slave node through the management system module;

[0036] The master node is used to obtain historical performance data from the database of the slave node and the rados shard, and update the historical performance data and bucket attribute information to the database of the master node.

[0037] A computer device comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor implements the following steps when executing the computer program:

[0038] The slave node collects bucket data through the object storage gateway. The bucket data includes bucket attribute information and historical performance data.

[0039] The slave node reports the bucket attribute information to the master node;

[0040] The slave node stores the historical performance data in the rados shard through the object storage gateway, and transmits the historical performance data to the management system module, which stores the historical performance data in the database of the slave node through the management system module;

[0041] The master node obtains historical performance data from the slave node's database and rados shards, and updates the historical performance data and bucket attribute information to the master node's database.

[0042] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the following steps:

[0043] The slave node collects bucket data through the object storage gateway. The bucket data includes bucket attribute information and historical performance data.

[0044] The slave node reports the bucket attribute information to the master node;

[0045] The slave node stores the historical performance data in the rados shard through the object storage gateway, and transmits the historical performance data to the management system module, which stores the historical performance data in the database of the slave node through the management system module;

[0046] The master node obtains historical performance data from the slave node's database and rados shards, and updates the historical performance data and bucket attribute information to the master node's database.

[0047] The above-mentioned bucket data management method, device, computer equipment and storage medium adopt different collection methods for different types of bucket data, and adopt real-time reporting method for bucket attribute information, so as to solve the problem of poor timeliness in the traditional technology that the interface directly interacts with the rados command. For historical performance data, the historical performance data is first collected from the node and stored in the rados shard, and the collected data is simultaneously passed to the management system module, which is stored in the database by the management module, so that the master node can simultaneously pull historical performance data from the underlying rados shard and the database, thus solving the problem of limited storage capacity in the traditional technology that historical performance data is stored on the underlying data shard. It can be seen that the use of this application can improve the timeliness of bucket data collection and make the collected data more complete. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 A schematic diagram of a process of a storage bucket data management method in one embodiment;

[0049] Figure 2 A schematic diagram of a process for collecting storage bucket information in one embodiment;

[0050] Figure 3 A schematic diagram of a process of executing data verification by a master node in one embodiment;

[0051] Figure 4 A block diagram of the principle of collecting bucket data from a node in one embodiment;

[0052] Figure 5 It is a block diagram of the interface management principle in one embodiment;

[0053] Figure 6 It is a structural block diagram of a distributed server cluster in another embodiment;

[0054] Figure 7 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0055] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0056] In one embodiment, Figure 1 As shown, a bucket data management method is provided, which is applied to a distributed server cluster, the distributed server cluster includes a master node and a slave node, the slave node is provided with a management system module, an object storage gateway and a rados shard, and the method includes:

[0057] Step S11, the slave node collects bucket data through the object storage gateway, and the bucket data includes bucket attribute information and historical performance data.

[0058] Among them, a bucket is a container for storing objects in distributed object storage. It provides a flat storage method based on buckets and objects, in which all objects are at the same logical level, without the multi-level tree directory structure in the file system. Each bucket has its own storage category, access rights, region and other attributes. Users can create buckets with different storage categories and access rights in different regions according to different storage requirements and access rights, and configure more advanced properties to meet different storage requirements. Among them, distributed object storage is a data storage technology in which data is stored in the form of objects.

[0059] The master node is a node that aggregates and manages the bucket information of each slave node. The slave node is a node that collects and reports the bucket data on each node. Among them, the slave nodes can include management nodes and non-management nodes. When the management node needs to switch to a non-management node (such as when the node fails), the master node can synchronize the aggregated bucket data to the newly switched non-management node to ensure the consistency of data on each node.

[0060] The management system module is a module for managing the bucket data collected from each slave node. Specifically, each node in the distributed server cluster is provided with the management system module for managing the bucket data collected by each node itself, and storing the bucket data collected by each node in the local database.

[0061] The object storage gateway (rados Gateway, RGW) is used to provide object storage access interfaces compatible with both S3 and OpenStack Swift (OpenStack's object storage component), and call the librados API (Application Programming Interface) to store and read data. Among them, OpenStack is an open source cloud computing management platform project, which is a combination of a series of software open source projects. Librados is a basic library in the Ceph distributed storage system, which provides a direct access interface to rados for programming languages ​​such as Java, Python, C and C++.

[0062] Bucket data refers to data related to a bucket, including bucket attribute information and bucket performance data. Bucket performance data may include historical performance data. Specifically, bucket attribute information may include static attribute information such as tenant, user, object policy, site, multiple versions, encryption, encryption algorithm, multi-site synchronization, and dynamic attribute information such as capacity and number of objects that do not require high real-time performance. Historical performance data may include bandwidth, latency, TPS (Transaction Per Second), return code, space increment, and bandwidth.

[0063] Specifically, bucket attribute information can be collected in the following ways:

[0064] The object storage gateway RGW monitors the changes in bucket attribute information caused by two methods: through the management interface or by monitoring the call status of the S3 interface. When the bucket attribute information is monitored to change, the bucket attribute information after the change is obtained, that is, the bucket attribute information in this application is obtained.

[0065] The bucket attribute information changes caused by the management interface refer to the bucket attribute information changes caused by the user operating the management interface to create or delete a bucket. The bucket attribute information changes caused by the S3 interface refer to the bucket attribute information changes caused by deleting or creating a bucket by calling the S3 interface. The S3 interface is an interface for accessing the object storage service, which allows users to interact with the object storage system through standard APIs and tools.

[0066] Please refer to Figure 2 , Figure 2 FIG. 1 is a flow chart of collecting storage bucket information in an embodiment. Figure 2 As shown, the slave node monitoring interface management and S interface monitor the changes in bucket attribute information. When the bucket attribute information sends a change, the changed bucket attribute information is obtained and reported to the management system module of the slave node. The management system module of the slave node decrypts the reported data and stores it in the database of the slave node. At the same time, the slave node can store the reported identified bucket attribute information in the cache queue to execute re-reporting.

[0067] Step S12: The slave node reports the bucket attribute information to the master node.

[0068] Among them, the slave node encrypts the bucket data with Base64 data, calls the reporting interface provided by RGW, assembles the data according to the interface parameters, and reports the assembled data to the master node.

[0069] Step S13, the slave node stores the historical performance data into the rados shard through the object storage gateway, and transmits the historical performance data to the management system module, and stores the historical performance data into the database of the slave node through the management system module.

[0070] Among them, Rados sharding refers to Rados data sharding, which refers to the technology of dividing data and storing it on multiple nodes to improve the scalability and performance of the system. In a distributed database system, data sharding allows data to be replicated to multiple physical databases in the network.

[0071] Specifically, a database is created in advance from the node. After the object storage gateway stores the bucket data into the rados shard, it reports the bucket attribute information to the management system module through the data reporting module, and passes the historical performance data to the management system module. The management system module stores the bucket data into the database and provides a WEB UI (User Interface) interface for dynamic display of various information, full sorting, and data switching display functions.

[0072] Step S14, the master node obtains historical performance data from the database of the slave node and the rados shard, and updates the historical performance data and bucket attribute information to the database of the master node.

[0073] Specifically, the management system module of the slave node can send the data in the database of the slave node to the master node. The master node can pull the bucket data stored in the rados shard by calling the interface provided by RGW, so that the pulled bucket data is more complete, and better management of the bucket data is achieved.

[0074] In one embodiment, the object storage gateway of the slave node is provided with an application programming interface, and the master node obtains historical performance data from the database of the slave node and the rados shard, including: the master node obtains the historical performance data within a first preset time from the database of the slave node, and the master node calls the application programming interface of the slave node to obtain the historical performance data within a second preset time from the rados shard, and the second preset time is less than the first preset time.

[0075] Among them, Application Programming Interface (API) is a set of predefined functions, the purpose of which is to provide applications and developers with the ability to access a set of routines based on certain software or hardware without accessing the source code or understanding the details of the internal working mechanism.

[0076] The first preset time and the second preset time can be set according to the actual application scenario or actual demand. For example, the first preset time can be set to one month, and the second preset time period can be set to more than one month. When it is necessary to obtain historical performance data within one month, it can be obtained from the database of the slave node. When it is necessary to obtain historical performance data for more than one month, the corresponding historical performance data can be pulled from the rados shard of the slave node.

[0077] Specifically, the underlying rados shards serve as a cache area. Due to the limitation on the number of shards, it can store one month's historical performance data. When querying data within one month, the interface directly interacts with rados for real-time query to ensure the accuracy of the data.

[0078] When historical performance data for more than one month is needed, the master node can enable a scheduled task, call the application programming interface provided by RGW, collect data at a preset time such as one hour, and store it in the database chronology.

[0079] The beneficial effect of this embodiment is that when collecting historical performance data, data can be collected from the database in combination with the rados sharding, thereby ensuring the integrity and timeliness of data collection.

[0080] In one embodiment, the bucket attribute information includes the number of buckets, the bucket identifier and the bucket modification time. The bucket data management method may further include: the master node obtains the first type of bucket attribute information and the second type of bucket attribute information respectively, wherein the first type of bucket attribute information is obtained from the database of the master node, and the second type of bucket attribute information is obtained by calling an application programming interface, and the difference in the number of buckets is determined according to the number of buckets in the first type of bucket attribute information and the number of buckets in the second type of bucket attribute information, and the first type of bucket attribute information is compared with the storage buckets with the same identifier in the second type of bucket attribute information, and the bucket modification time is compared to obtain the number of buckets with different bucket modification times. After the performance data and the bucket attribute information are updated to the database of the master node, the bucket data management method also includes: when the difference in the number of buckets and the sum of the number of buckets with different bucket modification times is greater than a preset value, the master node updates the second type of bucket attribute information in full to the database of the master node; when the difference in the number of buckets and the sum of the number of buckets with different bucket modification times is less than the preset value, the master node updates the newly added data in the second type of bucket attribute information to the database of the master node.

[0081] The master node obtains the first type of bucket attribute information in the current database, including the number of buckets, bucket identifiers (such as bucket IDs), and bucket modification time. Further, the master node obtains the full amount of bucket attribute information by calling the underlying application programming interface to obtain the second type of bucket attribute information, which also includes the number of buckets, bucket identifiers, and bucket modification time.

[0082] Furthermore, the difference between the number of buckets in the first type of bucket attribute information and the number of buckets in the second type of bucket attribute information is taken, and the absolute value of the difference is taken to obtain the difference a of the number of buckets.

[0083] Furthermore, the modification time of each bucket in the first type of bucket attribute information is compared with the modification time of the corresponding bucket in the second type of bucket attribute information to obtain the number b of buckets with different modification times. That is, the bucket modification times corresponding to two buckets with the same bucket identifier are compared, and if they are different, the comparison result is counted as the number b of different buckets.

[0084] Furthermore, the sum of a and b is calculated to obtain n, that is, n=a+b.

[0085] Furthermore, n is compared with a preset value (such as 100). If n>100, the data is fully updated. First, the bucket data of the current database is deleted, and then the acquired second-category bucket attribute information is inserted into the database. If n<=100, a partial update operation is performed according to the bucket ID, that is, the latest data is stored in the database to implement the data verification process. The data verification process can be set to be executed at a preset time every day, such as 2 a.m. every day, when the business pressure is small.

[0086] Please refer to Figure 3 , Figure 3 FIG. 1 is a flow chart of a master node performing data verification in an embodiment. Figure 3 In the process, the master node is provided with a verification thread, which can obtain the database data, i.e., the first type of bucket attribute information, and the rados data, i.e., the second type of bucket attribute information, calculate the above n according to the first type of bucket attribute information and the second type of bucket attribute information, and determine whether to perform a full update or a partial update on the bucket attribute information according to n.

[0087] The beneficial effects of this embodiment are: comparing the acquired attribute information in the database with the full data obtained from the underlying rados shard, verifying the current bucket data using partial update and full update methods respectively, to ensure the consistency of the management system module and RGW data.

[0088] In one embodiment, the application programming interface is a rados interface.

[0089] Among them, the rados interface refers to a sub-project of the distributed storage system, which aims to provide a stable, scalable, high-performance single logical object storage interface for dynamically changing and heterogeneous storage device clusters, and realize node adaptation and self-management.

[0090] In one of the embodiments, the bucket data also includes real-time performance data, and collecting the real-time performance data from the node includes: starting a scheduled task from the node and obtaining the cluster deployment type in a preset configuration file. When the cluster deployment type is object storage, running the scheduled task to collect real-time performance data, and the master node calls an application programming interface to obtain the real-time performance data.

[0091] The real-time performance data may include upload requests, download requests, delete requests, upload delay, download delay, upload IO (Input / Output) size, and download IO size.

[0092] Among them, in this application, three types of bucket data are collected from the node, namely bucket attribute information, historical performance data and real-time performance data. The collected bucket data is stored in the rados shard, and the bucket attribute information is reported to the master node.

[0093] Specifically, when collecting real-time performance data from the node, a scheduled task with a preset period (such as 1 minute) is started to determine whether the cluster deployment type is a storage object deployment type. If so, the scheduled task is run to collect real-time performance data.

[0094] Specifically, the deployment type of the cluster can generally include block storage, file storage, and object storage. This application sets the deployment type in the configuration file in advance. When real-time performance data needs to be collected, the cluster deployment type is first obtained. When the cluster deployment type is object storage, the scheduled task is started to collect real-time performance data.

[0095] Specifically, this application can obtain the ak (Access Key) and sk (Secret Access Key) of the built-in user admin (administrator account), encapsulate them into the header (message header) for authentication, and call the application programming interface for reporting after authentication. The real-time performance data collected by each slave node is the bucket data of the actual business running on the node, and these real-time performance data are stored in the local database of this node, and sent to the bucket data pool of the master node for aggregation. The master node accumulates and summarizes the real-time performance data of each node by bucket ID at the same time. After deleting the real-time table, the bucket performance data at the latest moment is inserted into the real-time table.

[0096] Among them, when collecting real-time performance data, the RESTful API interface is specifically called. This interface has the advantages of low latency and fast response, and is suitable for collecting real-time performance data.

[0097] Among them, the access key (ak / sk) is an identity authentication method, which is often used for identity authentication when calling interfaces between systems. ak is part of the key pair that users need to use when accessing services, and is used to identify users. sk corresponds to ak one by one, and is used for key authentication to ensure access security.

[0098] Please refer to Figure 4 , Figure 4 FIG. 1 is a block diagram of a principle for collecting bucket data from a node in an embodiment. Figure 4 As shown, the slave node is provided with a management system module and an object storage gateway, wherein the object storage gateway is provided with a data reporting module, a RESTful API interface, a rados interface, and rados sharding, etc., and the management system module is provided with a database, attribute information collection, historical performance data, real-time performance data, and dynamic display, etc. The object storage gateway of the slave node collects data and reports it to the management system module through the corresponding interface for aggregation and display.

[0099] The beneficial effects of this embodiment are: this application adopts different collection methods according to different types of storage bucket data, adopts the RESTful API interface call method for real-time performance data, and adopts the database combined with the underlying RADAS sharding method for historical performance data, thereby ensuring the integrity of the collection of each type of data and improving effectiveness.

[0100] In one of the embodiments, the slave node reports bucket attribute information to the master node, including: the object storage gateway of the slave node assembles data on the bucket attribute information, and reports the assembled data to the management system module of the slave node, the assembled data includes an operation type, the management system module of the slave node parses the assembled data, obtains the operation type, and maps the operation type to a preset operation code, the slave node starts a sending thread, and determines whether the destination node is the master node and whether it is reachable through the sending thread, when the destination node is the master node and is reachable, the operation code and the destination node are encapsulated into the message header of the message queue, the bucket attribute information is encapsulated into the message body, and the message is reported to the master node through the message queue, when the object storage gateway of the slave node fails to report, the object storage gateway of the slave node stores the bucket attribute information that failed to be reported in the cache queue, so as to re-report the bucket attribute information that failed to be reported.

[0101] The operation type refers to the type of database operation on the bucket data, and the operation type here may include updating bucket information, inserting bucket information, deleting bucket information, and inserting and then deleting bucket information, etc.

[0102] The operation code refers to the information pre-agreed between the slave node and the master node for indicating the operation on the bucket data. The master node can obtain the database operation of the bucket data according to the operation code.

[0103] Whether a node is reachable refers to whether a communication connection is established between the slave node and the master node.

[0104] Specifically, the reporting of the present application includes two parts. The GRW of the slave node reports the collected bucket data to the management system module, and then the management system module reports it to the master node.

[0105] Specifically, the RGW of the slave node collects the bucket attribute information, then encrypts the data using Base64, calls the interface provided by the reporting module of the RGW, assembles the data according to the interface parameters, and reports it to the management system module. The interface can be a Python script interface, which provides parameter verification and data parsing. The interface parameters here mainly include type (operation type) and data (reported data). Type supports 0 (update bucket information), 1 (insert bucket information), 2 (delete bucket information), and 3 (delete first and then insert bucket information), and reserves expansion space for easy operation expansion.

[0106] Furthermore, the management system module can obtain a list of slave nodes in the cluster, start a sending thread in each slave node, carry three parameters including node name, reporting type (i.e., operation type) and reported data, and then determine whether the destination node is the master node and whether the destination node is reachable. If the target node is the master node and is reachable, map the corresponding operation codes respectively, encapsulate the operation code, destination node, and destination module into the headers of the MQ (message queue) message attributes, check the availability of the MQ service, and send it through the MQ message channel. In order to prevent network fluctuations and other scenarios, a retry mechanism is added. After each transmission, wait for the return result, and update the sending identifier according to the result. If the sending identifier is Fail, it means that the transmission has failed, and the message is stored in the cache queue and resent. The maximum number of retries supported can be set in advance, such as 100 times.

[0107] Furthermore, the slave node RGW will open up a space in the hard disk to cache the bucket attribute information data that failed to be reported, and store it in the cache queue in the form of omap (a data structure containing key and value). The real-time monitoring thread is started, and the monitoring period can be set to 1 minute. The exception queue will be monitored. If there is bucket data with reported exceptions in the queue, it will continue to be reported until the report is successful, and the storage bucket data will be deleted from the exception queue.

[0108] The beneficial effects of this embodiment are: a real-time reporting mechanism is adopted for bucket attribute information, so that the collection of bucket attribute information is highly real-time. At the same time, a space is opened for caching the storage bucket data that failed to be reported. The RGW exception handling module monitors the reporting results in real time. When there is abnormal data in the exception queue, it is reported regularly to ensure the integrity of the reported data.

[0109] This application can adopt different information collection methods according to different types of storage bucket data, such as real-time reporting of attribute information, real-time performance data through REST API interface calls, and historical performance data through a database combined with a rados interface to improve the timeliness of information collection.

[0110] In one embodiment, historical performance data and bucket attribute information are updated to the database of the master node, including: the master node parses the operation code to obtain the operation type, when the operation type includes more than one operation, encapsulates each operation into the same database operation event, and updates the historical performance data and bucket attribute information to the database of the master node according to the database operation event.

[0111] Among them, database operation events refer to operations or changes that occur in the database, which can be a single operation or a collection of operations. Events can include operations such as inserting, updating, and deleting data, or they can trigger triggers, stored procedures, or other types of database objects in the database.

[0112] The beneficial effect of this embodiment is: when there are multiple operations in an operation, such as deleting data first and then inserting data, the multiple operations can be encapsulated into the same database event operation, that is, an atomic operation. The atomic operation can improve the overall rollback in abnormal scenarios and improve the accuracy of the operation.

[0113] In one of the embodiments, the bucket data management method also includes: the master node sends the bucket attribute information to the initiator of the query request for display in response to a query request for bucket attribute information, and the master node sends real-time performance data to the initiator of the query request for display in response to a request for display switching to switch the display of the attribute information to the display of the real-time performance data.

[0114] Specifically, please refer to Figure 5 , Figure 5 This is a block diagram of the interface display when displaying bucket data. Figure 5 As shown, the interface can display attribute information, real-time performance data and historical performance data, wherein the attribute information and real-time performance data are displayed in the form of a table, and the historical performance data is displayed in the form of a line graph. The table uses the grid component (a user interface control) of angula (a front-end framework for creating a single application interface), and the data is bound to the attribute information and real-time performance data of the background database, wherein the attribute information and real-time performance data can be switched for display. When displaying the interface, the present application can adopt a grouping strategy for the displayed fields, i.e., a fixed display group, a default display group, and a default hidden group. The advantage of grouped display is that it can facilitate the switching and expansion of the two types of data information, attribute information and real-time performance data. In addition, the present application can also be dynamically adjusted according to the configuration, and the dynamic adjustment scheme is as follows:

[0115] 1. Field maintenance of bucket data, define isFix (whether fixed), visible (whether displayed), type (type, 1 for attribute, 2 for performance), priority (priority, starting with the type and numbered from 0), and isSort (whether sorting is supported);

[0116] 2. The interface supports switching between attribute information and real-time performance data. For example, the bucket name, tenant, and user fixed display group remain unchanged, and other fields are displayed according to the switching type;

[0117] 3. When displaying bucket data, adjust the information display based on the screen width and the standard width occupied by a single field, as well as the priority of the field and whether to display it, to prevent excessive display of field information;

[0118] 4. When expanding the field information of the corresponding type, if the priority conflicts with the existing one, the existing priority will be automatically increased by 1.

[0119] In addition, the historical performance data of the bucket uses a line graph. You can set the bucket data within one month to be bound to the data on the rados shard, and the bucket data older than one month to be bound to the database.

[0120] The beneficial effects of this embodiment are: the present application can manage various types of bucket data through the management system module, dynamically adjust the bucket data display of the interface according to the configuration, facilitate the switching of bucket attribute information and real-time performance data, expand field information, and fully sort by key fields of bucket information.

[0121] In one of the embodiments, the distributed server cluster also includes a non-management node, and the bucket data management method further includes: when the slave node is switched to the non-management node, the master node synchronizes the data in the database to the non-management node.

[0122] In one embodiment, Figure 6 As shown, a distributed server cluster is provided, the distributed server cluster includes a master node and a slave node, the slave node is provided with a management system module, an object storage gateway and a rados shard, wherein:

[0123] The slave node is used to collect bucket data through the object storage gateway. The bucket data includes bucket attribute information and historical performance data.

[0124] The slave node is also used to report bucket attribute information to the master node, store historical performance data in the rados shard through the object storage gateway, and pass the historical performance data to the management system module, which stores the historical performance data in the database of the slave node through the management system module;

[0125] The master node is used to obtain historical performance data from the database of the slave node and the rados shard, and update the historical performance data and bucket attribute information to the database of the master node.

[0126] In one embodiment, the object storage gateway of the slave node is provided with an application programming interface, wherein:

[0127] The master node is used to obtain historical performance data within a first preset time from a database of the slave node;

[0128] The master node is used to call the application programming interface of the slave node to obtain historical performance data within a second preset time from the rados shard, and the second preset time is less than the first preset time.

[0129] In one embodiment, the bucket attribute information includes the bucket quantity, bucket identifier, and bucket modification time, wherein:

[0130] The master node is also used to obtain first-category bucket attribute information and second-category bucket attribute information respectively, wherein the first-category bucket attribute information is obtained from the database of the master node, and the second-category bucket attribute information is obtained by calling an application programming interface. The difference in the number of buckets is determined based on the number of buckets in the first-category bucket attribute information and the number of buckets in the second-category bucket attribute information, and the first-category bucket attribute information is compared with the storage buckets with the same identifier in the second-category bucket attribute information. The bucket modification time is compared to obtain the number of buckets with different bucket modification times. After the performance data and bucket attribute information are updated to the database of the master node, the master node is also used to update the second-category bucket attribute information in full to the database of the master node when the difference in the number of buckets and the sum of the number of buckets with different bucket modification times is greater than a preset value, and when the difference in the number of buckets and the sum of the number of buckets with different bucket modification times is less than a preset value, the master node updates the newly added data in the second-category bucket attribute information to the database of the master node.

[0131] In one embodiment, the application programming interface is a rados interface.

[0132] In one embodiment, the bucket data also includes real-time performance data, wherein:

[0133] Start the scheduled task from the node and obtain the cluster deployment type in the preset configuration file. When the cluster deployment type is object storage, run the scheduled task to collect real-time performance data. The master node calls the application programming interface to obtain real-time performance data.

[0134] In one of the embodiments, the object storage gateway of the slave node is used to assemble data of bucket attribute information and report the assembled data to the management system module of the slave node. The assembled data includes an operation type. The management system module of the slave node is used to parse the assembled data, obtain the operation type, and map the operation type to a preset operation code. The slave node is used to start a sending thread and determine whether the destination node is a master node and whether it is reachable through the sending thread. When the destination node is the master node and reachable, the operation code and the destination node are encapsulated into the message header of the message queue, and the bucket attribute information is encapsulated into the message body. The message is reported to the master node through the message queue. When the object storage gateway of the slave node fails to report, the object storage gateway of the slave node stores the bucket attribute information that failed to be reported in the cache queue so that the bucket attribute information that failed to be reported can be re-reported.

[0135] In one embodiment, the master node is used to parse the operation code to obtain the operation type. When the operation type includes more than one operation, each operation is encapsulated into the same database operation event, and the historical performance data and bucket attribute information are updated to the database of the master node according to the database operation event.

[0136] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 7 As shown. The computer device includes a processor, a memory and a network interface connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, the storage bucket data management method is implemented.

[0137] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the following steps are implemented: the slave node collects bucket data through an object storage gateway, the bucket data includes bucket attribute information and historical performance data, the slave node reports the bucket attribute information to the master node, the slave node stores the historical performance data in a rados shard through the object storage gateway, and transmits the historical performance data to a management system module, the management system module stores the historical performance data in a database of the slave node, the master node obtains the historical performance data from the database of the slave node and the rados shard, and updates the historical performance data and bucket attribute information to the database of the master node.

[0138] In one of the embodiments, the object storage gateway of the slave node is provided with an application programming interface, and the processor executes a computer program to implement the above-mentioned master node to obtain historical performance data from the database of the slave node and the rados shard, and specifically implements the following steps: the master node obtains the historical performance data within a first preset time from the database of the slave node, and the master node calls the application programming interface of the slave node to obtain the historical performance data within a second preset time from the rados shard, and the second preset time is less than the first preset time.

[0139] In one of the embodiments, the bucket attribute information includes the number of buckets, the bucket identifier and the bucket modification time. When the processor executes the computer program, the following steps are specifically implemented: the master node obtains the first type of bucket attribute information and the second type of bucket attribute information respectively, wherein the first type of bucket attribute information is obtained from the database of the master node, and the second type of bucket attribute information is obtained by calling an application programming interface; the difference in the number of buckets is determined according to the number of buckets in the first type of bucket attribute information and the number of buckets in the second type of bucket attribute information, the first type of bucket attribute information is compared with the storage buckets with the same identifier in the second type of bucket attribute information, the bucket modification time is compared, and the number of buckets with different bucket modification times is obtained; after updating the performance data and the bucket attribute information to the database of the master node, when the processor executes the computer program, the following steps are specifically implemented: when the difference in the number of buckets and the sum of the number of buckets with different bucket modification times is greater than a preset value, the master node updates the second type of bucket attribute information in full to the database of the master node; when the difference in the number of buckets and the sum of the number of buckets with different bucket modification times is less than the preset value, the master node updates the newly added data in the second type of bucket attribute information to the database of the master node.

[0140] In one embodiment, the application programming interface is a rados interface.

[0141] In one of the embodiments, the storage bucket data also includes real-time performance data. When the processor executes a computer program to implement the above-mentioned step of collecting real-time performance data from the node, the following steps are specifically implemented: starting a scheduled task from the node and obtaining the cluster deployment type in a preset configuration file. When the cluster deployment type is object storage, running the scheduled task to collect real-time performance data. When the processor executes the computer program, the following steps are also specifically implemented: the master node calls an application programming interface to obtain real-time performance data.

[0142] In one of the embodiments, when the processor executes a computer program to implement the above-mentioned step of reporting the bucket attribute information from the slave node to the master node, the following steps are specifically implemented: the object storage gateway of the slave node assembles the bucket attribute information data, and reports the assembled data to the management system module of the slave node, the assembled data includes the operation type, the management system module of the slave node parses the assembled data, obtains the operation type, and maps the operation type to a preset operation code, the slave node starts a sending thread, and determines whether the destination node is the master node and whether it is reachable through the sending thread, when the destination node is the master node and is reachable, the operation code and the destination node are encapsulated into the message header of the message queue, the bucket attribute information is encapsulated into the message body, and the message is reported to the master node through the message queue, and when the processor executes the computer program, the following steps are specifically implemented: when the object storage gateway of the slave node fails to report, the object storage gateway of the slave node stores the bucket attribute information that failed to be reported in the cache queue so that the bucket attribute information that failed to be reported can be re-reported.

[0143] In one of the embodiments, when the processor executes a computer program to implement the above-mentioned step of updating the historical performance data and bucket attribute information to the database of the master node, the following steps are specifically implemented: the master node parses the operation code to obtain the operation type, when the operation type includes more than one operation, encapsulates each operation into the same database operation event, and updates the historical performance data and bucket attribute information to the database of the master node according to the database operation event.

[0144] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the following steps are implemented: the slave node collects storage bucket data through an object storage gateway, the storage bucket data includes bucket attribute information and historical performance data, the slave node reports the bucket attribute information to the master node, the slave node stores the historical performance data in a rados shard through the object storage gateway, and passes the historical performance data to a management system module, and stores the historical performance data in a database of the slave node through the management system module, the master node obtains the historical performance data from the database of the slave node and the rados shard, and updates the historical performance data and bucket attribute information to the database of the master node.

[0145] In one of the embodiments, the object storage gateway of the slave node is provided with an application programming interface, and the computer program is executed by the processor to implement the above-mentioned master node to obtain historical performance data from the database of the slave node and the rados shard, and specifically implement the following steps: the master node obtains the historical performance data within a first preset time from the database of the slave node, and the master node calls the application programming interface of the slave node to obtain the historical performance data within a second preset time from the rados shard, and the second preset time is less than the first preset time.

[0146] In one of the embodiments, the bucket attribute information includes the number of buckets, the bucket identifier and the bucket modification time. When the computer program is executed by the processor, the following steps are specifically implemented: the master node obtains the first type of bucket attribute information and the second type of bucket attribute information respectively, wherein the first type of bucket attribute information is obtained from the database of the master node, and the second type of bucket attribute information is obtained by calling an application programming interface; the difference in the number of buckets is determined according to the number of buckets in the first type of bucket attribute information and the number of buckets in the second type of bucket attribute information, the first type of bucket attribute information is compared with the storage buckets with the same identifier in the second type of bucket attribute information, the bucket modification time is compared, and the number of buckets with different bucket modification times is obtained; after updating the performance data and the bucket attribute information to the database of the master node, when the processor executes the computer program, the following steps are specifically implemented: when the difference in the number of buckets and the sum of the number of buckets with different bucket modification times is greater than a preset value, the master node updates the second type of bucket attribute information in full to the database of the master node; when the difference in the number of buckets and the sum of the number of buckets with different bucket modification times is less than a preset value, the master node updates the newly added data in the second type of bucket attribute information to the database of the master node.

[0147] In one embodiment, the application programming interface is a rados interface.

[0148] In one of the embodiments, the storage bucket data also includes real-time performance data. When the computer program is executed by the processor to implement the above-mentioned step of collecting real-time performance data from the node, the following steps are specifically implemented: starting a scheduled task from the node, and obtaining the cluster deployment type in a preset configuration file. When the cluster deployment type is object storage, running the scheduled task to collect real-time performance data. When the computer program is executed by the processor, the following steps are also specifically implemented: the master node calls the application programming interface to obtain real-time performance data.

[0149] In one of the embodiments, when the computer program is executed by the processor to implement the above-mentioned step of reporting the bucket attribute information from the slave node to the master node, the following steps are specifically implemented: the object storage gateway of the slave node assembles the bucket attribute information data, and reports the assembled data to the management system module of the slave node, the assembled data includes the operation type, the management system module of the slave node parses the assembled data, obtains the operation type, and maps the operation type to a preset operation code, the slave node starts a sending thread, and determines whether the destination node is the master node and whether it is reachable through the sending thread, when the destination node is the master node and is reachable, the operation code and the destination node are encapsulated into the message header of the message queue, the bucket attribute information is encapsulated into the message body, and the message is reported to the master node through the message queue, and when the computer program is executed by the processor, the following steps are specifically implemented: when the object storage gateway of the slave node fails to report, the object storage gateway of the slave node stores the bucket attribute information that failed to be reported in the cache queue so that the bucket attribute information that failed to be reported can be re-reported.

[0150] In one of the embodiments, when the computer program is executed by the processor to implement the above-mentioned step of updating the historical performance data and bucket attribute information to the database of the master node, the following steps are specifically implemented: the master node parses the operation code to obtain the operation type, when the operation type includes more than one operation, encapsulates each operation into the same database operation event, and updates the historical performance data and bucket attribute information to the database of the master node according to the database operation event.

[0151] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing related hardware through a computer program, and the above-mentioned computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct RAMbus dynamic RAM (DRDRAM), and RAMbus dynamic RAM (RDRAM), etc.

[0152] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0153] The above embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.

Claims

1. A method for managing bucket data, characterized in that: The method is applied to a distributed server cluster, the distributed server cluster includes a master node and a slave node, the slave node is provided with a management system module, an object storage gateway and a rados shard, and the method includes: The slave node collects bucket data through an object storage gateway, where the bucket data includes bucket attribute information and historical performance data; The slave node reports the bucket attribute information to the master node; The slave node stores the historical performance data in the rados shard through the object storage gateway, transmits the historical performance data to the management system module, and stores the historical performance data in the database of the slave node through the management system module; The master node obtains the historical performance data from the database of the slave node and the rados shard, and updates the historical performance data and the bucket attribute information to the database of the master node.

2. The method according to claim 1, characterized in that The object storage gateway of the slave node is provided with an application programming interface, and the master node obtains the historical performance data from the database of the slave node and the rados shard, including: The master node acquires historical performance data within a first preset time from a database of the slave node; The master node calls the application programming interface of the slave node to obtain historical performance data within a second preset time from the rados shard, where the second preset time is less than the first preset time.

3. The method according to claim 2, characterized in that The bucket attribute information includes the number of buckets, bucket identifiers, and bucket modification time, and the method further includes: The master node obtains first-category bucket attribute information and second-category bucket attribute information respectively, wherein the first-category bucket attribute information is obtained from a database of the master node, and the second-category bucket attribute information is obtained by calling the application programming interface; The master node determines a difference in the number of buckets according to the number of buckets in the first type of bucket attribute information and the number of buckets in the second type of bucket attribute information; The master node compares the first type of bucket attribute information with the storage buckets with the same identifier in the second type of bucket attribute information, compares the bucket modification time, and obtains the number of buckets with different bucket modification time; After updating the performance data and the bucket attribute information to the database of the master node, the method further includes: When the difference in the number of buckets and the sum of the number of buckets with different bucket modification times is greater than a preset value, the master node updates all the second-category bucket attribute information to the database of the master node; When the difference in the number of buckets and the sum of the number of buckets with different bucket modification times is less than the preset value, the master node updates the newly added data in the second-category bucket attribute information to the database of the master node.

4. The method according to claim 2 or 3, characterized in that: The application programming interface is a rados interface.

5. The method according to claim 1, characterized in that The storage bucket data also includes real-time performance data, and the slave node collects the real-time performance data, including: The slave node starts the scheduled task and obtains the cluster deployment type in the preset configuration file. When the cluster deployment type is object storage, the scheduled task is run to collect the real-time performance data. The method further comprises: The master node calls the application programming interface to obtain the real-time performance data.

6. The method according to claim 1, characterized in that The slave node reports the bucket attribute information to the master node, including: The object storage gateway of the slave node assembles data of the bucket attribute information and reports the assembled data to the management system module of the slave node, wherein the assembled data includes an operation type; The management system module of the slave node parses the assembled data, obtains the operation type, and maps the operation type to a preset operation code; The slave node starts a sending thread, and determines whether the destination node is a master node and whether it is reachable through the sending thread. When the destination node is a master node and reachable, the operation code and the destination node are encapsulated into a message header of a message queue, the bucket attribute information is encapsulated into a message body, and the message is reported to the master node through the message queue; The method further comprises: When the object storage gateway of the slave node fails to report, the object storage gateway of the slave node stores the bucket attribute information that failed to report into a cache queue so as to re-report the bucket attribute information that failed to report.

7. The method according to claim 4, characterized in that The updating of the historical performance data and the bucket attribute information to the database of the master node includes: The master node parses the operation code to obtain the operation type. When the operation type includes more than one operation, each operation is encapsulated into the same database operation event, and the historical performance data and the bucket attribute information are updated to the database of the master node according to the database operation event.

8. A distributed server cluster, characterized in that: The distributed server cluster includes a master node and a slave node, wherein the slave node is provided with a management system module, an object storage gateway and a rados shard, wherein: The slave node is used to collect bucket data through the object storage gateway, and the bucket data includes bucket attribute information and historical performance data; The slave node is further used to report the bucket attribute information to the master node, store the historical performance data in the rados shard through the object storage gateway, and transmit the historical performance data to the management system module, and store the historical performance data in the database of the slave node through the management system module; The master node is used to obtain the historical performance data from the database of the slave node and the rados shard, and update the historical performance data and the bucket attribute information to the database of the master node.

9. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.