Object storage method and device based on storage bucket, equipment and medium

Abstract

The invention relates to the technical field of big data, provides an object storage method, device and equipment based on storage buckets and a medium, is applied to financial and medical health care service scenes, and can allocate independent distributed transaction key value database instances for each storage bucket according to service attribute information of each storage bucket. Not only is the performance bottleneck of a single database instance avoided, but also complete resource isolation among the storage buckets is realized; establishing a corresponding relation mapping table between the storage buckets and the corresponding distributed transaction key value database instances so as to ensure that the meta-information of each storage bucket can only be stored in the corresponding distributed transaction key value database instance; according to the method, the simple storage service meta-information is divided according to the data attributes, the target storage bucket corresponding to each piece of sub-information is determined, the target instance corresponding to each target storage bucket is inquired in the corresponding relation mapping table, each piece of sub-information is written into each corresponding target instance, flexible allocation of resources is achieved, and the resource utilization rate is increased.

Description

Technical Field

[0001] This invention relates to the field of big data technology, and in particular to an object storage method, apparatus, device and medium based on storage buckets. Background Technology

[0002] Currently, access logs, operational data, and other metadata in business management systems for industries such as finance, healthcare, and elderly care all need to be effectively stored to ensure normal use.

[0003] In existing technologies, mainstream object storage systems in the industry generally suffer from the following problems when handling large-scale metadata storage: (1) In traditional distributed object storage architectures, S3 (Simple Storage Service) metadata is usually stored centrally in a unified database instance. This design creates significant performance bottlenecks when dealing with massive numbers of buckets and objects. When the number of buckets increases dramatically, the read and write pressure on a single database instance increases rapidly, leading to increased response latency and potentially service unavailability. For example, many cloud storage platforms use relational database management systems or object-relational database management systems to store metadata. When the number of buckets reaches millions, query performance drops significantly, failing to meet the demands of high-concurrency access.

[0004] (2) Existing technologies lack effective horizontal scaling capabilities. In traditional centralized architectures, when storage capacity needs to be expanded, it can usually only be achieved through vertical upgrades (such as increasing server hardware resources), which is costly and has physical limitations. For scenarios that need to process a large number of small files, the scalability of this architecture is particularly limited. In practical applications, many systems improve read capabilities by adding database master-slave nodes, but this does not solve the write bottleneck problem and increases the complexity and maintenance cost of the system.

[0005] (3) Existing technologies have shortcomings in resource isolation at the bucket level. Sharing the same metadata storage instance among different buckets means that abnormal operations in one bucket may affect the stability of the entire system. For example, if a large number of invalid metadata records or malicious requests appear in a bucket, it may cause the performance of the entire database instance to degrade, thereby affecting the normal service of all buckets.

[0006] (4) Existing technologies lack flexible resource allocation mechanisms when deployed in clusters. When multiple buckets require different levels of storage performance, it is impossible to perform fine-grained resource configuration according to business needs. For example, some high-value buckets require higher read and write performance, while some low-value buckets can accept lower performance, but it is difficult to achieve such differentiated configuration under traditional architecture.

[0007] (5) Existing technologies are inefficient in handling fault recovery. When a storage node fails, the system often needs to rebalance the data distribution of all buckets, a process that is time-consuming and prone to service interruption. Especially in large-scale cluster environments, this fault recovery process may take several hours to complete, seriously affecting the availability of the system.

[0008] (6) Existing technologies do not perform well in terms of resource utilization optimization. Due to resource sharing among buckets, it is difficult for the system to accurately control the resource consumption of each bucket, which can easily lead to resource waste or resource contention. For example, some buckets may be idle for a long time, while other buckets continuously occupy a large amount of resources. This unbalanced resource allocation pattern reduces the overall operating efficiency of the system.

[0009] In view of the above problems, it is necessary to provide a more reasonable object storage solution. Summary of the Invention

[0010] In view of the above, it is necessary to provide a bucket-based object storage method, apparatus, device and medium to solve the problems of low object storage efficiency, insufficient resource isolation and poor resource allocation flexibility.

[0011] A bucket-based object storage method, the bucket-based object storage method comprising: In response to storage commands for simple storage service metadata, retrieve each bucket and its business attribute information. Each storage bucket is assigned an independent distributed transactional key-value database instance based on its business attribute information. Establish a mapping table between each bucket and its corresponding distributed transaction key-value database instance based on the distributed transaction key-value database instance allocated to each bucket; Obtain the data attributes of each metadata in the simple storage service metadata; The metadata of the simple storage service is divided according to the data attributes to obtain multiple sub-information; The storage bucket corresponding to each sub-information is determined as each target storage bucket according to the data attributes described above; The corresponding distributed transaction key-value database instance for each target bucket is queried in the mapping table to serve as each target instance. Write each piece of information into the corresponding target instance.

[0012] A bucket-based object storage device, the bucket-based object storage device comprising: The acquisition unit is used to acquire each storage bucket and its business attribute information in response to storage instructions for simple storage service metadata. The allocation unit is used to allocate an independent distributed transactional key-value database instance to each bucket based on the business attribute information of each bucket. The establishment unit is used to establish a mapping table between each bucket and the corresponding distributed transaction key-value database instance based on the distributed transaction key-value database instance allocated to each bucket; The acquisition unit is also used to acquire the data attributes of each piece of metadata in the simple storage service metadata; A partitioning unit is used to partition the simple storage service metadata according to the data attributes to obtain multiple sub-information; The determining unit is configured to determine, according to the data attributes, the storage bucket corresponding to each sub-information as each target storage bucket; The query unit is used to query the corresponding distributed transaction key-value database instance for each target bucket as each target instance in the corresponding relationship mapping table; The write unit is used to write each piece of sub-information to each corresponding target instance.

[0013] A computer device, the computer device comprising: Memory, storing at least one instruction; and The processor executes instructions stored in the memory to implement the bucket-based object storage method.

[0014] A computer-readable storage medium storing at least one instruction, which is executed by a processor in a computer device to implement the bucket-based object storage method.

[0015] As can be seen from the above technical solutions, this invention can allocate an independent distributed transaction key-value database instance to each storage bucket based on the business attribute information of each bucket. This not only avoids the performance bottleneck of a single database instance but also achieves complete resource isolation between storage buckets. A mapping table is established between each storage bucket and its corresponding distributed transaction key-value database instance based on the distributed transaction key-value database instance allocated to each bucket, ensuring that the metadata of each storage bucket can only be stored in its corresponding distributed transaction key-value database instance. The simple storage service metadata is divided according to data attributes, and the target storage bucket corresponding to each sub-information is determined. The target instance corresponding to each target storage bucket is queried in the mapping table, and each sub-information is written to its corresponding target instance, achieving flexible resource allocation and improving resource utilization. Attached Figure Description

[0016] Figure 1 This is a flowchart of a preferred embodiment of the object storage method based on storage buckets of the present invention.

[0017] Figure 2 This is a functional block diagram of a preferred embodiment of the object storage device based on a storage bucket according to the present invention.

[0018] Figure 3 This is a schematic diagram of the structure of a computer device that implements a preferred embodiment of the bucket-based object storage method of the present invention. Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0020] like Figure 1 The diagram shown is a flowchart of a preferred embodiment of the bucket-based object storage method of the present invention. The order of the steps in this flowchart can be changed, and some steps can be omitted, depending on different requirements.

[0021] The bucket-based object storage method is applied to one or more computer devices. The computer device is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions. Its hardware includes, but is not limited to, microprocessors, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), digital signal processors (DSPs), embedded devices, etc.

[0022] The computer device can be any electronic product that can interact with the user, such as a personal computer, tablet computer, smartphone, personal digital assistant (PDA), game console, interactive network television (IPTV), smart wearable device, etc.

[0023] The computer equipment may also include network equipment and / or user equipment. The network equipment includes, but is not limited to, a single network server, a server group consisting of multiple network servers, or a cloud based on cloud computing consisting of a large number of hosts or network servers.

[0024] The server can be a standalone server or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms.

[0025] Artificial intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0026] Foundational technologies for artificial intelligence generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies mainly encompass computer vision, robotics, biometrics, speech processing, natural language processing, and machine learning / deep learning.

[0027] The network in which the computer device is located includes, but is not limited to, the Internet, wide area network, metropolitan area network, local area network, and virtual private network (VPN).

[0028] S10, in response to the storage command for Simple Storage Service (S3) metadata, obtains each bucket and its business attribute information.

[0029] In this embodiment, the simple storage service metadata may include, but is not limited to, access logs and operational data from business management systems in the financial industry, healthcare, and elderly care sectors.

[0030] In this embodiment, the storage instruction can be automatically triggered when new data is generated, or it can be triggered by relevant personnel according to actual needs.

[0031] In this embodiment, the business attribute information may include, but is not limited to, one or more of the following information combinations: object storage status, access frequency, value, etc.

[0032] S11. Allocate an independent Distributed Transactional Key-Value Database (TiKV) instance to each storage bucket based on the business attribute information of each storage bucket.

[0033] In this embodiment, allocating an independent distributed transaction key-value database instance for each storage bucket based on the business attribute information of each storage bucket includes: Based on the bucket routing algorithm, an independent distributed transaction key-value database instance is allocated to each bucket according to its storage status, access frequency, and value.

[0034] This includes pre-establishing a matching rule library between bucket business attribute information and Tikv instance performance parameters, so that Tikv instances of corresponding performance levels can be allocated to buckets according to different business needs.

[0035] The performance parameters of the Tikv instance may include, but are not limited to: CPU (Central Processing Unit) processing power, memory size, disk read / write speed, storage capacity, etc.

[0036] For example, based on the value, higher-performance TiKV instances can be allocated to high-value buckets, while standard-configuration TiKV instances can be allocated to ordinary buckets; the optimal TiKV instance can be automatically selected for metadata storage based on indicators such as the bucket's storage status and access frequency.

[0037] In the above embodiments, the flexible resource configuration method ensures load balancing across Tikv instances, preventing hotspot issues. Simultaneously, it significantly improves resource utilization efficiency and simplifies system management complexity.

[0038] S12, establish a mapping table between each storage bucket and its corresponding distributed transaction key-value database instance based on the distributed transaction key-value database instance allocated to each storage bucket.

[0039] In this embodiment, the mapping table can be used to record detailed information about each bucket and the bound Tikv instance, including bucket ID, Tikv instance ID, performance parameter configuration, binding effective time, etc.

[0040] The mapping table can also support operations such as querying, modifying, and unbinding the binding relationship.

[0041] The above embodiments change the traditional centralized storage model, avoiding the performance bottleneck of a single database instance, and maintaining stable performance even when the number of buckets reaches tens of millions. Furthermore, they achieve complete resource isolation between buckets, ensuring that abnormal operations in a single bucket will not affect the normal service of other buckets.

[0042] S13, Obtain the data attributes of each metadata in the simple storage service metadata.

[0043] In this embodiment, the data attributes may include, but are not limited to, the object's storage path, size, creation time, access permissions, etc.

[0044] S14, the metadata of the simple storage service is divided according to the data attributes to obtain multiple sub-information.

[0045] In this embodiment, dividing the metadata of the simple storage service according to the data attributes makes it easier to subsequently filter out the corresponding buckets and allocate them to the Tikv instance corresponding to the bucket for storage.

[0046] S15, determine the storage bucket corresponding to each sub-information as each target storage bucket according to the data attributes.

[0047] In this embodiment, appropriate storage buckets can be assigned to each sub-information according to its data attributes.

[0048] S16, query the distributed transaction key-value database instance corresponding to each target bucket in the corresponding relationship mapping table as each target instance.

[0049] In this embodiment, effective matching of target instances can be achieved through a unified mapping table.

[0050] S17, write each sub-information to the corresponding target instance.

[0051] For example, in a financial scenario, within a financial industry business management system, each customer's access log can be viewed as a bucket. The metadata of each customer's access log (such as access records and their storage paths) is stored in a corresponding independent TiKV instance. When a customer performs frequent queries, it does not affect the access speed of other customers' metadata, ensuring the stable operation of financial services. Simultaneously, it effectively prevents the normal service of the entire business management system from being affected by abnormal data from a single customer (such as a sudden increase in data volume or abnormal query requests).

[0052] For example, in a healthcare scenario, each patient's data in a business management system for medical, health, and elderly care can be treated as a bucket, with the patient's data (i.e., metadata) stored in a corresponding independent TiKV instance. When a department performs centralized access to or updates specific patient data, it will not interfere with access to other patient data, ensuring the independent security and access stability of medical data. This prevents issues with a single patient's data (such as a large number of invalid modification records or malicious access requests) from affecting the normal operation of the entire business management system.

[0053] Through the above embodiments, metadata can be written into the corresponding target instance, enabling targeted writing of different metadata.

[0054] In this embodiment, after writing each sub-information to its corresponding target instance, the method further includes: Monitor the resource utilization of each target storage bucket in real time; When the resource utilization rate of a target storage bucket is detected to be greater than a preset threshold, the detected target storage bucket will be identified as a storage bucket to be expanded. Obtain current business requirement data, and obtain the performance data of the target instance corresponding to the storage bucket to be expanded; Calculate the capacity expansion based on the current business demand data and the performance data; Based on the expansion capacity, the target instance corresponding to the storage bucket to be expanded is expanded individually.

[0055] The resource utilization rate can be calculated using metrics such as CPU utilization, memory usage, disk space utilization, and read / write request response time.

[0056] The preset threshold can be the optimal value selected based on the experiment.

[0057] The required amount of resources to be expanded, such as the increased memory size and disk capacity, can be calculated based on the current business demand data and the performance data.

[0058] The above embodiments enable dynamic scaling of TiKV instance resources for specific buckets without requiring large-scale adjustments to the entire cluster, reducing resource waste and improving the overall elasticity and scalability of the system. Simultaneously, it supports smooth online scaling, avoiding the service interruption risks associated with traditional vertical scaling and ensuring business continuity.

[0059] In this embodiment, after writing each sub-information to its corresponding target instance, the method further includes: Monitor the running status of each target instance in real time; When an abnormal running status of a target instance is detected, the detected target instance is identified as an instance to be processed, and an abnormal alarm is issued. The sub-information corresponding to the instance to be processed is migrated to the backup instance, and the routing mapping table is updated according to the backup instance; wherein, the routing mapping table is used for the business front-end to access the corresponding sub-information; When the running status of the instance to be processed is detected to have returned to normal, the information temporarily stored on the backup instance is synchronized to the instance to be processed.

[0060] The operating status may include network connection status, service response status, data integrity, etc.

[0061] Among them, when heartbeat interruption, service unresponsiveness, data verification errors, etc. are detected, it can be determined that the operation status is abnormal.

[0062] The above embodiments achieve automated fault detection and recovery. When a TiKV instance fails, it can quickly perform failover, ensuring business continuity and significantly shortening fault recovery time. In simulated fault tests, the system can complete failover and data recovery within seconds. Simultaneously, the backup instance also ensures data consistency and integrity, preventing data loss or corruption due to faults.

[0063] In this embodiment, after writing each sub-information to its corresponding target instance, the method further includes: The system receives storage policies configured through the storage policy definition interface in real time and obtains the target instance corresponding to the storage policy as the instance to be adjusted. Convert the storage strategy into recognizable storage instructions; The storage parameters of the instance to be adjusted are adjusted according to the identifiable storage instructions.

[0064] The storage strategy may include, but is not limited to: storage capacity limit, data access frequency limit (such as the maximum number of read and write operations per unit time), data retention period (such as data retention time, automatic deletion rules upon expiration), and data backup strategy (such as backup frequency and backup storage location).

[0065] The above embodiments enhance the flexibility and adaptability of object storage, supporting customized storage strategies based on the needs of different business scenarios to meet diverse storage requirements. Furthermore, they enable real-time application and dynamic adjustment of storage strategies without requiring system restarts or business interruptions, improving system maintainability and business response speed.

[0066] In this embodiment, after writing each sub-information to its corresponding target instance, the method further includes: Configure independent operation and maintenance monitoring metrics and anomaly alarm policies for each target instance; Real-time monitoring is performed on each target instance based on its operation and maintenance monitoring metrics and anomaly alerting policies.

[0067] For example, a data collection agent can be installed on each Tikv instance to collect key metrics such as read / write performance (e.g., number of read / write requests per second, average read / write response time), storage capacity (used capacity, remaining capacity, capacity utilization), and resource utilization (CPU utilization, memory utilization, disk I / O utilization) of the corresponding bucket in real time as the operation and maintenance monitoring metrics.

[0068] The operation and maintenance monitoring metrics for each target instance can be displayed in the form of charts and other formats.

[0069] Different alarm thresholds can be configured for each monitoring metric, such as CPU utilization exceeding 80% or remaining storage capacity below 10%. When a monitoring metric reaches or exceeds the corresponding alarm threshold, alarm information can be promptly sent to operations and maintenance personnel via SMS, email, system messages, etc., while alarm logs are recorded, including alarm time, alarm metric, alarm threshold, and current metric value, to facilitate subsequent analysis and processing.

[0070] The above embodiments provide detailed and intuitive system operation status information, thereby assisting in the timely detection and resolution of potential problems and improving maintainability. The alarm mechanism also enables rapid response in the event of abnormal situations, reducing the impact of failures on business operations.

[0071] In this embodiment, after writing each sub-information to its corresponding target instance, the method further includes: Based on the distributed transaction characteristics of the distributed transaction key-value database instance, strong consistency operations are performed on the bucket-level metadata of each target instance; and / or Transaction conflicts and concurrency are handled for each target instance based on locking mechanisms and concurrency control strategies; and / or Role-based access control (RBAC) performs security controls on each target instance.

[0072] Among them, Tikv's distributed transaction features can be utilized to implement distributed transaction processing of bucket-level metadata operations based on the Raft protocol (distributed consensus algorithm), ensuring that multiple metadata operations (such as updating the storage path and access permissions of an object at the same time) either all succeed or all fail, thereby guaranteeing the atomicity of the transaction.

[0073] Row-level locks (locks for individual metadata records) can be used to reduce the scope of lock conflicts. When multiple concurrent requests operate on different metadata records, they will not block each other, thus improving the concurrent processing capability.

[0074] One approach is to use an optimistic concurrency control (OCC) mechanism, which identifies the status of metadata records by version numbers. When multiple requests modify the same metadata record at the same time, only the first request will succeed. Other requests will determine the conflict based on the version number and retry, thus avoiding the performance overhead caused by pessimistic locking.

[0075] Role-based access control provides security control for each target instance and can assign different permissions to different roles.

[0076] The above embodiments ensure strong consistency in bucket-level metadata operations, meeting the needs of business scenarios with high data consistency requirements. Simultaneously, they minimize transaction conflicts, enhancing the system's concurrent processing capabilities, making them particularly suitable for scenarios requiring high-concurrency read / write operations. RBAC (Random Access Control) further strengthens data security, preventing unauthorized access and operations.

[0077] In this embodiment, all user operations on the bucket and Tikv instance can also be recorded, including the operator, operation time, operation type (such as creating a bucket, modifying metadata, allocating resources), operation result, etc. The audit log is tamper-proof, which facilitates subsequent security audits and problem tracing.

[0078] This embodiment is based on bucket sharding and TiKV instance isolation. Each tenant is allocated an independent bucket set and a corresponding TiKV instance cluster. The bucket data of different tenants are stored in their own independent TiKV instance clusters, which physically achieves data isolation and ensures that the data between different tenants will not access or interfere with each other, thus meeting the strict requirements of enterprise-level applications for data security.

[0079] This embodiment also boasts excellent compatibility and scalability, allowing for seamless integration into existing S3-compatible storage systems without requiring major modifications to existing business logic. Furthermore, it supports the introduction of new storage engines as technology advances, providing ample room for future technological evolution.

[0080] This embodiment also supports metadata migration across TiKV instances. When resource allocation needs to be adjusted, it can be done through an asynchronous background migration mechanism without affecting normal access to front-end services.

[0081] As can be seen from the above technical solutions, this invention can allocate an independent distributed transaction key-value database instance to each storage bucket based on the business attribute information of each bucket. This not only avoids the performance bottleneck of a single database instance but also achieves complete resource isolation between storage buckets. A mapping table is established between each storage bucket and its corresponding distributed transaction key-value database instance based on the distributed transaction key-value database instance allocated to each bucket, ensuring that the metadata of each storage bucket can only be stored in its corresponding distributed transaction key-value database instance. The simple storage service metadata is divided according to data attributes, and the target storage bucket corresponding to each sub-information is determined. The target instance corresponding to each target storage bucket is queried in the mapping table, and each sub-information is written to its corresponding target instance, achieving flexible resource allocation and improving resource utilization.

[0082] like Figure 2 The diagram shown is a functional block diagram of a preferred embodiment of the bucket-based object storage device of the present invention. The bucket-based object storage device 11 includes an acquisition unit 110, an allocation unit 111, an establishment unit 112, a partitioning unit 113, a determination unit 114, a query unit 115, and a writing unit 116. The module / unit referred to in this invention refers to a series of computer program segments that can be executed by a processor and perform a fixed function, and which are stored in memory. In this embodiment, the functions of each module / unit will be described in detail in subsequent embodiments.

[0083] The acquisition unit 110 is used to acquire each bucket and its business attribute information in response to a storage instruction for Simple Storage Service (S3) metadata.

[0084] In this embodiment, the simple storage service metadata may include, but is not limited to, access logs and operational data from business management systems in the financial industry, healthcare, and elderly care sectors.

[0085] In this embodiment, the storage instruction can be automatically triggered when new data is generated, or it can be triggered by relevant personnel according to actual needs.

[0086] In this embodiment, the business attribute information may include, but is not limited to, one or more of the following information combinations: object storage status, access frequency, value, etc.

[0087] The allocation unit 111 is used to allocate an independent Distributed Transactional Key-Value Database (TiKV) instance to each storage bucket according to the business attribute information of each storage bucket.

[0088] In this embodiment, the allocation unit 111 allocates an independent distributed transaction key-value database instance to each storage bucket based on the business attribute information of each storage bucket, including: Based on the bucket routing algorithm, an independent distributed transaction key-value database instance is allocated to each bucket according to its storage status, access frequency, and value.

[0089] This includes pre-establishing a matching rule library between bucket business attribute information and Tikv instance performance parameters, so that Tikv instances of corresponding performance levels can be allocated to buckets according to different business needs.

[0090] The performance parameters of the Tikv instance may include, but are not limited to: CPU (Central Processing Unit) processing power, memory size, disk read / write speed, storage capacity, etc.

[0091] For example, based on the value, higher-performance TiKV instances can be allocated to high-value buckets, while standard-configuration TiKV instances can be allocated to ordinary buckets; the optimal TiKV instance can be automatically selected for metadata storage based on indicators such as the bucket's storage status and access frequency.

[0092] In the above embodiments, the flexible resource configuration method ensures load balancing across Tikv instances, preventing hotspot issues. Simultaneously, it significantly improves resource utilization efficiency and simplifies system management complexity.

[0093] The establishment unit 112 is used to establish a mapping table between each storage bucket and the corresponding distributed transaction key-value database instance based on the distributed transaction key-value database instance allocated to each storage bucket.

[0094] In this embodiment, the mapping table can be used to record detailed information about each bucket and the bound Tikv instance, including bucket ID, Tikv instance ID, performance parameter configuration, binding effective time, etc.

[0095] The mapping table can also support operations such as querying, modifying, and unbinding the binding relationship.

[0096] The above embodiments change the traditional centralized storage model, avoiding the performance bottleneck of a single database instance, and maintaining stable performance even when the number of buckets reaches tens of millions. Furthermore, they achieve complete resource isolation between buckets, ensuring that abnormal operations in a single bucket will not affect the normal service of other buckets.

[0097] The acquisition unit 110 is also used to acquire the data attributes of each piece of metadata in the simple storage service metadata.

[0098] In this embodiment, the data attributes may include, but are not limited to, the object's storage path, size, creation time, access permissions, etc.

[0099] The partitioning unit 113 is used to partition the simple storage service metadata according to the data attributes to obtain multiple sub-information.

[0100] In this embodiment, dividing the metadata of the simple storage service according to the data attributes makes it easier to subsequently filter out the corresponding buckets and allocate them to the Tikv instance corresponding to the bucket for storage.

[0101] The determining unit 114 is used to determine the storage bucket corresponding to each sub-information as each target storage bucket according to the data attributes.

[0102] In this embodiment, appropriate storage buckets can be assigned to each sub-information according to its data attributes.

[0103] The query unit 115 is used to query the corresponding distributed transaction key-value database instance for each target storage bucket in the corresponding relationship mapping table as each target instance.

[0104] In this embodiment, effective matching of target instances can be achieved through a unified mapping table.

[0105] The writing unit 116 is used to write each sub-information to each corresponding target instance.

[0106] For example, in a financial scenario, within a financial industry business management system, each customer's access log can be viewed as a bucket. The metadata of each customer's access log (such as access records and their storage paths) is stored in a corresponding independent TiKV instance. When a customer performs frequent queries, it does not affect the access speed of other customers' metadata, ensuring the stable operation of financial services. Simultaneously, it effectively prevents the normal service of the entire business management system from being affected by abnormal data from a single customer (such as a sudden increase in data volume or abnormal query requests).

[0107] For example, in a healthcare scenario, each patient's data in a business management system for medical, health, and elderly care can be treated as a bucket, with the patient's data (i.e., metadata) stored in a corresponding independent TiKV instance. When a department performs centralized access to or updates specific patient data, it will not interfere with access to other patient data, ensuring the independent security and access stability of medical data. This prevents issues with a single patient's data (such as a large number of invalid modification records or malicious access requests) from affecting the normal operation of the entire business management system.

[0108] Through the above embodiments, metadata can be written into the corresponding target instance, enabling targeted writing of different metadata.

[0109] In this embodiment, after each sub-information is written to each corresponding target instance, the resource utilization rate of each target storage bucket is monitored in real time. When the resource utilization rate of a target storage bucket is detected to be greater than a preset threshold, the detected target storage bucket will be identified as a storage bucket to be expanded. Obtain current business requirement data, and obtain the performance data of the target instance corresponding to the storage bucket to be expanded; Calculate the capacity expansion based on the current business demand data and the performance data; Based on the expansion capacity, the target instance corresponding to the storage bucket to be expanded is expanded individually.

[0110] The resource utilization rate can be calculated using metrics such as CPU utilization, memory usage, disk space utilization, and read / write request response time.

[0111] The preset threshold can be the optimal value selected based on the experiment.

[0112] The required amount of resources to be expanded, such as the increased memory size and disk capacity, can be calculated based on the current business demand data and the performance data.

[0113] The above embodiments enable dynamic scaling of TiKV instance resources for specific buckets without requiring large-scale adjustments to the entire cluster, reducing resource waste and improving the overall elasticity and scalability of the system. Simultaneously, it supports smooth online scaling, avoiding the service interruption risks associated with traditional vertical scaling and ensuring business continuity.

[0114] In this embodiment, after each sub-information is written into each corresponding target instance, the running status of each target instance is monitored in real time; When an abnormal running status of a target instance is detected, the detected target instance is identified as an instance to be processed, and an abnormal alarm is issued. The sub-information corresponding to the instance to be processed is migrated to the backup instance, and the routing mapping table is updated according to the backup instance; wherein, the routing mapping table is used for the business front-end to access the corresponding sub-information; When the running status of the instance to be processed is detected to have returned to normal, the information temporarily stored on the backup instance is synchronized to the instance to be processed.

[0115] The operating status may include network connection status, service response status, data integrity, etc.

[0116] Among them, when heartbeat interruption, service unresponsiveness, data verification errors, etc. are detected, it can be determined that the operation status is abnormal.

[0117] The above embodiments achieve automated fault detection and recovery. When a TiKV instance fails, it can quickly perform failover, ensuring business continuity and significantly shortening fault recovery time. In simulated fault tests, the system can complete failover and data recovery within seconds. Simultaneously, the backup instance also ensures data consistency and integrity, preventing data loss or corruption due to faults.

[0118] In this embodiment, after writing each sub-information into each corresponding target instance, the storage policy configured through the storage policy definition interface is received in real time, and the target instance corresponding to the storage policy is obtained as the instance to be adjusted. Convert the storage strategy into recognizable storage instructions; The storage parameters of the instance to be adjusted are adjusted according to the identifiable storage instructions.

[0119] The storage strategy may include, but is not limited to: storage capacity limit, data access frequency limit (such as the maximum number of read and write operations per unit time), data retention period (such as data retention time, automatic deletion rules upon expiration), and data backup strategy (such as backup frequency and backup storage location).

[0120] The above embodiments enhance the flexibility and adaptability of object storage, supporting customized storage strategies based on the needs of different business scenarios to meet diverse storage requirements. Furthermore, they enable real-time application and dynamic adjustment of storage strategies without requiring system restarts or business interruptions, improving system maintainability and business response speed.

[0121] In this embodiment, after writing each sub-information into each corresponding target instance, an independent operation and maintenance monitoring indicator and anomaly alarm strategy are configured for each target instance; Real-time monitoring is performed on each target instance based on its operation and maintenance monitoring metrics and anomaly alerting policies.

[0122] For example, a data collection agent can be installed on each Tikv instance to collect key metrics such as read / write performance (e.g., number of read / write requests per second, average read / write response time), storage capacity (used capacity, remaining capacity, capacity utilization), and resource utilization (CPU utilization, memory utilization, disk I / O utilization) of the corresponding bucket in real time as the operation and maintenance monitoring metrics.

[0123] The operation and maintenance monitoring metrics for each target instance can be displayed in the form of charts and other formats.

[0124] Different alarm thresholds can be configured for each monitoring metric, such as CPU utilization exceeding 80% or remaining storage capacity below 10%. When a monitoring metric reaches or exceeds the corresponding alarm threshold, alarm information can be promptly sent to operations and maintenance personnel via SMS, email, system messages, etc., while alarm logs are recorded, including alarm time, alarm metric, alarm threshold, and current metric value, to facilitate subsequent analysis and processing.

[0125] The above embodiments provide detailed and intuitive system operation status information, thereby assisting in the timely detection and resolution of potential problems and improving maintainability. The alarm mechanism also enables rapid response in the event of abnormal situations, reducing the impact of failures on business operations.

[0126] In this embodiment, after writing each piece of sub-information to its corresponding target instance, a strong consistency operation of the storage bucket-level metadata is performed on each target instance based on the distributed transaction characteristics of the distributed transaction key-value database instance; and / or Transaction conflicts and concurrency are handled for each target instance based on locking mechanisms and concurrency control strategies; and / or Role-based access control (RBAC) performs security controls on each target instance.

[0127] Among them, Tikv's distributed transaction features can be utilized to implement distributed transaction processing of bucket-level metadata operations based on the Raft protocol (distributed consensus algorithm), ensuring that multiple metadata operations (such as updating the storage path and access permissions of an object at the same time) either all succeed or all fail, thereby guaranteeing the atomicity of the transaction.

[0128] Row-level locks (locks for individual metadata records) can be used to reduce the scope of lock conflicts. When multiple concurrent requests operate on different metadata records, they will not block each other, thus improving the concurrent processing capability.

[0129] One approach is to use an optimistic concurrency control (OCC) mechanism, which identifies the status of metadata records by version numbers. When multiple requests modify the same metadata record at the same time, only the first request will succeed. Other requests will determine the conflict based on the version number and retry, thus avoiding the performance overhead caused by pessimistic locking.

[0130] Role-based access control provides security control for each target instance and can assign different permissions to different roles.

[0131] The above embodiments ensure strong consistency in bucket-level metadata operations, meeting the needs of business scenarios with high data consistency requirements. Simultaneously, they minimize transaction conflicts, enhancing the system's concurrent processing capabilities, making them particularly suitable for scenarios requiring high-concurrency read / write operations. RBAC (Random Access Control) further strengthens data security, preventing unauthorized access and operations.

[0132] In this embodiment, all user operations on the bucket and Tikv instance can also be recorded, including the operator, operation time, operation type (such as creating a bucket, modifying metadata, allocating resources), operation result, etc. The audit log is tamper-proof, which facilitates subsequent security audits and problem tracing.

[0133] This embodiment is based on bucket sharding and TiKV instance isolation. Each tenant is allocated an independent bucket set and a corresponding TiKV instance cluster. The bucket data of different tenants are stored in their own independent TiKV instance clusters, which physically achieves data isolation and ensures that the data between different tenants will not access or interfere with each other, thus meeting the strict requirements of enterprise-level applications for data security.

[0134] This embodiment also boasts excellent compatibility and scalability, allowing for seamless integration into existing S3-compatible storage systems without requiring major modifications to existing business logic. Furthermore, it supports the introduction of new storage engines as technology advances, providing ample room for future technological evolution.

[0135] This embodiment also supports metadata migration across TiKV instances. When resource allocation needs to be adjusted, it can be done through an asynchronous background migration mechanism without affecting normal access to front-end services.

[0136] As can be seen from the above technical solutions, this invention can allocate an independent distributed transaction key-value database instance to each storage bucket based on the business attribute information of each bucket. This not only avoids the performance bottleneck of a single database instance but also achieves complete resource isolation between storage buckets. A mapping table is established between each storage bucket and its corresponding distributed transaction key-value database instance based on the distributed transaction key-value database instance allocated to each bucket, ensuring that the metadata of each storage bucket can only be stored in its corresponding distributed transaction key-value database instance. The simple storage service metadata is divided according to data attributes, and the target storage bucket corresponding to each sub-information is determined. The target instance corresponding to each target storage bucket is queried in the mapping table, and each sub-information is written to its corresponding target instance, achieving flexible resource allocation and improving resource utilization.

[0137] like Figure 3 The diagram shown is a structural schematic of a computer device that implements a preferred embodiment of the bucket-based object storage method of the present invention.

[0138] The computer device 1 may include a memory 12, a processor 13, and a bus (the arrow in the figure represents the bus), and may also include a computer program stored in the memory 12 and executable on the processor 13, such as a bucket-based object storage program.

[0139] Those skilled in the art will understand that the schematic diagram is merely an example of computer device 1 and does not constitute a limitation on computer device 1. Computer device 1 can be either a bus topology or a star topology. Computer device 1 may also include more or fewer other hardware or software than shown in the diagram, or different component arrangements. For example, computer device 1 may also include input / output devices, network access devices, etc.

[0140] It should be noted that the computer device 1 described is merely an example. Other existing or future electronic products that are adaptable to this invention should also be included within the scope of protection of this invention and are incorporated herein by reference.

[0141] The memory 12 includes at least one type of readable storage medium, such as flash memory, portable hard drive, multimedia card, card-type memory (e.g., SD or DX memory), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory 12 can be an internal storage unit of the computer device 1, such as a portable hard drive of the computer device 1. In other embodiments, the memory 12 can be an external storage device of the computer device 1, such as a plug-in portable hard drive, Smart Media Card (SMC), Secure Digital (SD) card, Flash Card, etc., equipped on the computer device 1. Furthermore, the memory 12 can include both internal storage units and external storage devices of the computer device 1. The memory 12 can be used not only to store application software and various types of data installed on the computer device 1, such as the code of bucket-based object storage programs, but also to temporarily store data that has been output or will be output.

[0142] In some embodiments, the processor 13 may be composed of integrated circuits, such as a single packaged integrated circuit or multiple integrated circuits packaged with the same or different functions, including combinations of one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and various control chips. The processor 13 is the control unit of the computer device 1, connecting various components of the computer device 1 via various interfaces and lines. It performs various functions of the computer device 1 and processes data by running or executing programs or modules stored in the memory 12 (e.g., executing bucket-based object storage programs) and calling data stored in the memory 12.

[0143] The processor 13 executes the operating system of the computer device 1 and various installed applications. The processor 13 executes the applications to implement the steps in the various bucket-based object storage method embodiments described above, for example... Figure 1 The steps are shown.

[0144] For example, the computer program may be divided into one or more modules / units, which are stored in the memory 12 and executed by the processor 13 to complete the present invention. The one or more modules / units may be a series of computer-readable instruction segments capable of performing specific functions, which describe the execution process of the computer program in the computer device 1. For example, the computer program may be divided into an acquisition unit 110, an allocation unit 111, an establishment unit 112, a partitioning unit 113, a determination unit 114, a query unit 115, and a writing unit 116.

[0145] The integrated unit implemented as a software functional module described above can be stored in a computer-readable storage medium. This software functional module, stored in a storage medium, includes several instructions to cause a computer device (which may be a personal computer, computer equipment, or network device, etc.) or processor to execute portions of the bucket-based object storage method described in the various embodiments of the present invention.

[0146] If the modules / units integrated in the computer device 1 are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments of the present invention can also be implemented by a computer program instructing related hardware devices. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above.

[0147] The computer program includes computer program code, which may be in the form of source code, object code, executable file, or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording media, USB flash drive, portable hard drive, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory, etc.

[0148] Furthermore, the computer-readable storage medium may primarily include a stored program area and a stored data area, wherein the stored program area may store the operating system, an application program required for at least one function, etc.; and the stored data area may store data created based on the use of blockchain nodes, etc.

[0149] The blockchain referred to in this invention is a novel application model of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanisms, and encryption algorithms. Essentially, a blockchain is a decentralized database, a chain of data blocks linked together using cryptographic methods. Each data block contains information about a batch of network transactions, used to verify the validity of the information (anti-counterfeiting) and generate the next block. A blockchain can include an underlying blockchain platform, a platform product service layer, and an application service layer.

[0150] The bus can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This bus can be divided into address bus, data bus, control bus, etc. For ease of representation, in... Figure 3 The bus is represented by only one straight line, but this does not mean that there is only one bus or one type of bus. The bus is configured to enable communication between the memory 12 and at least one processor 13, etc.

[0151] Although not shown, the computer device 1 may also include a power supply (such as a battery) to power various components. Preferably, the power supply can be logically connected to the at least one processor 13 through a power management device, thereby enabling functions such as charging management, discharging management, and power consumption management. The power supply may also include one or more DC or AC power supplies, recharging devices, power fault detection circuits, power converters or inverters, power status indicators, and other arbitrary components. The computer device 1 may also include various sensors, Bluetooth modules, Wi-Fi modules, etc., which will not be described in detail here.

[0152] Furthermore, the computer device 1 may also include a network interface. Optionally, the network interface may include a wired interface and / or a wireless interface (such as a Wi-Fi interface, a Bluetooth interface, etc.), which is typically used to establish a communication connection between the computer device 1 and other computer devices.

[0153] Optionally, the computer device 1 may further include a user interface, which may be a display, an input unit (such as a keyboard), and optionally, a standard wired interface or a wireless interface. Optionally, in some embodiments, the display may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, or an OLED (Organic Light-Emitting Diode) touchscreen, etc. The display may also be appropriately referred to as a screen or display unit, used to display information processed in the computer device 1 and to display a visual user interface.

[0154] It should be understood that the embodiments described are for illustrative purposes only and are not limited to this structure in the scope of the patent application.

[0155] It will be understood by those skilled in the art that Figure 3 The structure shown does not constitute a limitation on the computer device 1, and may include fewer or more components than shown, or combine certain components, or have different component arrangements.

[0156] Combination Figure 1 The memory 12 in the computer device 1 stores multiple instructions to implement a bucket-based object storage method, and the processor 13 can execute the multiple instructions to achieve: The acquisition unit is used to acquire each storage bucket and its business attribute information in response to storage instructions for simple storage service metadata. The allocation unit is used to allocate an independent distributed transactional key-value database instance to each bucket based on the business attribute information of each bucket. The establishment unit is used to establish a mapping table between each bucket and the corresponding distributed transaction key-value database instance based on the distributed transaction key-value database instance allocated to each bucket; The acquisition unit is also used to acquire the data attributes of each piece of metadata in the simple storage service metadata; A partitioning unit is used to partition the simple storage service metadata according to the data attributes to obtain multiple sub-information; The determining unit is configured to determine, according to the data attributes, the storage bucket corresponding to each sub-information as each target storage bucket; The query unit is used to query the corresponding distributed transaction key-value database instance for each target bucket as each target instance in the corresponding relationship mapping table; The write unit is used to write each piece of sub-information to each corresponding target instance.

[0157] Specifically, the processor 13's implementation method for the above instructions can be found in [reference needed]. Figure 1 The descriptions of the relevant steps in the corresponding embodiments are not repeated here.

[0158] It should be noted that all data involved in this case was legally obtained. Software tools or components not belonging to this company that appear in the embodiments of this application are merely illustrative examples and do not represent actual use.

[0159] In the several embodiments provided by this invention, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and other division methods may be used in actual implementation.

[0160] This invention can be used in a wide variety of general-purpose or special-purpose computer system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices. This invention can be described in the general context of computer-executable instructions, such as program modules, that are executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. This invention can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0161] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0162] Furthermore, the functional modules in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or in the form of hardware plus software functional modules.

[0163] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the present invention can be implemented in other specific forms without departing from the spirit or essential characteristics of the present invention.

[0164] Therefore, the embodiments should be considered exemplary and non-limiting in all respects, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be embraced within the invention. No appended diagram markings in the claims should be construed as limiting the scope of the claims.

[0165] Furthermore, it is clear that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. Multiple units or devices described in this invention can also be implemented by a single unit or device through software or hardware. Terms such as "first," "second," etc., are used to indicate names and do not indicate any specific order.

[0166] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention.

Claims

1. A bucket-based object storage method, characterized in that, The bucket-based object storage method includes: In response to storage commands for simple storage service metadata, retrieve each bucket and its business attribute information. Each storage bucket is assigned an independent distributed transactional key-value database instance based on its business attribute information. Establish a mapping table between each bucket and its corresponding distributed transaction key-value database instance based on the distributed transaction key-value database instance allocated to each bucket; Obtain the data attributes of each metadata in the simple storage service metadata; The metadata of the simple storage service is divided according to the data attributes to obtain multiple sub-information; The storage bucket corresponding to each sub-information is determined as each target storage bucket according to the data attributes described above; The corresponding distributed transaction key-value database instance for each target bucket is queried in the mapping table to serve as each target instance. Write each piece of information into the corresponding target instance.

2. The bucket-based object storage method as described in claim 1, characterized in that, The process of allocating an independent distributed transactional key-value database instance for each storage bucket based on the business attribute information of each storage bucket includes: Based on the bucket routing algorithm, an independent distributed transaction key-value database instance is allocated to each bucket according to its storage status, access frequency, and value.

3. The bucket-based object storage method as described in claim 1, characterized in that, After writing each sub-information to its corresponding target instance, the method further includes: Monitor the resource utilization of each target storage bucket in real time; When the resource utilization rate of a target storage bucket is detected to be greater than a preset threshold, the detected target storage bucket will be identified as a storage bucket to be expanded. Obtain current business requirement data, and obtain the performance data of the target instance corresponding to the storage bucket to be expanded; Calculate the capacity expansion based on the current business demand data and the performance data; Based on the expansion capacity, the target instance corresponding to the storage bucket to be expanded is expanded individually.

4. The bucket-based object storage method as described in claim 1, characterized in that, After writing each sub-information to its corresponding target instance, the method further includes: Monitor the running status of each target instance in real time; When an abnormal running status of a target instance is detected, the detected target instance is identified as an instance to be processed, and an abnormal alarm is issued. The sub-information corresponding to the instance to be processed is migrated to the backup instance, and the routing mapping table is updated according to the backup instance; wherein, the routing mapping table is used for the business front-end to access the corresponding sub-information; When the running status of the instance to be processed is detected to have returned to normal, the information temporarily stored on the backup instance is synchronized to the instance to be processed.

5. The bucket-based object storage method as described in claim 1, characterized in that, After writing each sub-information to its corresponding target instance, the method further includes: The system receives storage policies configured through the storage policy definition interface in real time and obtains the target instance corresponding to the storage policy as the instance to be adjusted. Convert the storage strategy into recognizable storage instructions; The storage parameters of the instance to be adjusted are adjusted according to the identifiable storage instructions.

6. The bucket-based object storage method as described in claim 1, characterized in that, After writing each sub-information to its corresponding target instance, the method further includes: Configure independent operation and maintenance monitoring metrics and anomaly alarm policies for each target instance; Real-time monitoring is performed on each target instance based on its operation and maintenance monitoring metrics and anomaly alerting policies.

7. The bucket-based object storage method as described in claim 1, characterized in that, After writing each sub-information to its corresponding target instance, the method further includes: Based on the distributed transaction characteristics of the distributed transaction key-value database instance, strong consistency operations are performed on the bucket-level metadata of each target instance; and / or Transaction conflicts and concurrency are handled for each target instance based on locking mechanisms and concurrency control strategies; and / or Role-based access control provides security controls for each target instance.

8. A bucket-based object storage device, characterized in that, The bucket-based object storage device includes: The acquisition unit is used to acquire each storage bucket and its business attribute information in response to storage instructions for simple storage service metadata. The allocation unit is used to allocate an independent distributed transactional key-value database instance to each bucket based on the business attribute information of each bucket. The establishment unit is used to establish a mapping table between each bucket and the corresponding distributed transaction key-value database instance based on the distributed transaction key-value database instance allocated to each bucket; The acquisition unit is also used to acquire the data attributes of each piece of metadata in the simple storage service metadata; A partitioning unit is used to partition the simple storage service metadata according to the data attributes to obtain multiple sub-information; The determining unit is configured to determine, according to the data attributes, the storage bucket corresponding to each sub-information as each target storage bucket; The query unit is used to query the corresponding distributed transaction key-value database instance for each target bucket as each target instance in the corresponding relationship mapping table; The write unit is used to write each piece of sub-information to each corresponding target instance.

9. A computer device, characterized in that, The computer device includes: Memory, storing at least one instruction; and The processor executes instructions stored in the memory to implement the bucket-based object storage method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores at least one instruction, which is executed by a processor in a computer device to implement the bucket-based object storage method as described in any one of claims 1 to 7.

Images