A cloud storage-based database service allocation method, device and medium

Abstract

The application discloses a cloud storage-based database service distribution method and device and a medium, relates to the technical field of database service distribution, and comprises the following steps: dividing a cloud storage database table into continuous key range fragments according to effective key columns, establishing dispersed copies and row group address indexes for the continuous key range fragments, and generating a fragment copy service directory; parsing a database service request by using the fragment copy service directory, screening hit row groups in a target fragment, merging continuous file fragments, generating a request fragment queue, matching a common access path based on the request fragment queue, merging the continuous file fragments into a common access cache package, and constructing a double-mapping candidate result in combination with the fragment copy service directory. Through buffer object reservation and cyclic layout coding rebalancing, the application improves the service continuity, query response efficiency and storage resource utilization rate of the cloud storage database under the scenes of high-concurrency query, fragment copy migration and dynamic change of buffer objects.

Description

Technical Field

[0001] This invention relates to the field of database service allocation technology, and in particular to a database service allocation method, device and medium based on cloud storage. Background Technology

[0002] With the development of cloud computing and distributed storage technologies, database services are gradually shifting from centralized deployment on a single node to a cloud database architecture that separates computing and storage. Existing cloud databases typically employ columnar storage, sharded storage, replica disaster recovery, and caching acceleration to improve data access performance. Columnar storage organizes data through row groups, stripes, and file fragments; the distributed storage layer ensures data availability through sharded replicas; and the database compute nodes access the corresponding data shards based on query requests and complete the query processing using local caching, distributed caching, or cloud storage object read mechanisms.

[0003] Existing cloud storage-based database service allocation typically focuses on the running status of database service instances, the number of connections, the location of primary replicas, or the availability of storage nodes. However, it lacks a unified and coordinated assessment of the fragment replicas containing the target data, the physical continuity of row groups, the location of cached objects, and the common access relationships between request fragments. When a database request involves multiple consecutive file fragments or multiple replica nodes, service instance allocation and cache hit status can easily become disconnected. This can lead to requests being allocated to instances that do not hold the target cached object, resulting in issues such as duplicate cloud storage reads, cross-node data access, and insufficient cached object reuse. This negatively impacts the stability and execution efficiency of database service allocation in industrial cloud storage environments. Summary of the Invention

[0004] In view of the aforementioned existing problems, the present invention is proposed.

[0005] Therefore, this invention provides a cloud storage-based database service allocation method to solve the problem of the separation between the replica location and the cache object location in cloud storage database service allocation.

[0006] To solve the above-mentioned technical problems, the present invention provides the following technical solution: In a first aspect, the present invention provides a database service allocation method based on cloud storage, comprising, The cloud storage database table is divided into continuous key range fragments according to the valid key columns, and distributed replicas and row group address indexes are created for the continuous key range fragments to generate a fragment replica service catalog. The database service requests are parsed using the fragmented replica service directory, the matching row groups in the target fragments are filtered and continuous file fragments are merged to generate a request fragment queue; Based on the request fragment queue matching common access paths, consecutive file fragments are merged into common access cache packages, and combined with the fragment replica service directory to construct dual mapping candidate results; Based on the dual mapping candidate results, select the database service instance that simultaneously holds a valid copy and the target cache object, and merge request fragments of the same instance to generate a database service allocation plan; The database service allocation plan performs row group data filtering, cache object retention, and circular layout encoding rebalancing, updates the fragment replica service directory and the separate cache directory, and generates the database fragment service execution results.

[0007] As a preferred embodiment of the cloud storage-based database service allocation method of the present invention, the generation of the fragmented replica service catalog specifically includes: Extract the valid key columns from the cloud storage database table, divide the continuous key range according to the key value order of the valid key columns and set fragment identifiers, deploy distributed replicas on different storage nodes and mark the primary replicas and secondary replicas to obtain the fragment replica location table; Locate the stripes, target columns, row groups, and file fragments of a columnar storage file within a contiguous key range fragment to obtain the row group address index; The root fragment record is established by using fragment identifier, consecutive key range, fragment replica location table, primary replica marker, secondary replica marker, and row group address index. The search items in the root fragment record are then extracted to set the directory search entry, thus obtaining the fragment replica service directory.

[0008] As a preferred embodiment of the cloud storage-based database service allocation method of the present invention, the generation of the request fragment queue specifically includes: Parse the database service request and retrieve the continuous key range in the fragment replica service directory to determine the target fragment sequence; Extract the row group statistical range of the target column within the target fragment sequence, remove the row groups that do not intersect with the data filtering conditions, and splice adjacent file fragments in physical address order to obtain the set of hit row groups and the file fragment reading range. The cache block group identifier is set according to the file fragment reading range, and the target fragment sequence, the hit line group set, the file fragment reading range, and the cache block group identifier are arranged into a request fragment queue.

[0009] As a preferred embodiment of the cloud storage-based database service allocation method of the present invention, the step of merging consecutive file fragments into a common access cache package based on matching common access paths according to request fragment queues specifically includes: Extract the file fragment reading range and cache block group identifier from the request fragment queue to obtain the request cache sequence. Then, check the common access path of the request cache sequence through the frequent pattern tree entry in the root fragment record to obtain the common access path matching result. When the common access path matching result contains multiple consecutive cache block groups, the corresponding file segment reading range is encapsulated to obtain the common access cache package; When the common access path matching result does not contain a group of consecutively occurring cache blocks, the file fragment reading range in the requested cache sequence is retained to obtain the cache object to be allocated. The common access cache package and the cache object to be allocated are identified as the target cache objects, and the target cache objects are matched with the locations of the existing cache objects in the database service instance to obtain a list of target cache objects and a separate cache directory.

[0010] As a preferred embodiment of the cloud storage-based database service allocation method of the present invention, the construction of dual-mapping candidate results specifically includes: Embed the list of target cached objects into the request fragment queue, and extract the target fragment identifier, cache block group identifier and common access cache package identifier from it to obtain the allocation index key set; By assigning target fragment identifiers in the index key set, the database service instance where the valid replica resides is retrieved in the fragment replica service directory, thus obtaining a set of candidate replica instances; By assigning cache block group identifiers and common access cache package identifiers in the index key set, the database service instance where the target cache object is located is retrieved in the separate cache directory to obtain a cache candidate instance set; Cross-validate the database service instances in the candidate replica instance set and the candidate cache instance set, and mark the instances that simultaneously hold a valid replica and the target cache object to obtain the overlapping candidate instance set; When the set of overlapping candidate instances is not empty, the database service instance in the set of overlapping candidate instances is determined as the preferred candidate instance, and the first candidate branch is obtained. When the overlapping candidate instance set is empty and the replica candidate instance set contains the database service instance corresponding to the replica tag, the database service instance with the latest replica version is determined as the replacement candidate instance, and the second candidate branch is obtained. When the overlapping candidate instance set is empty and the replica candidate instance set contains the database service instance corresponding to the primary replica mark, the database service instance corresponding to the primary replica mark is determined as the source candidate instance, and the third candidate branch is obtained. Based on the first candidate branch, the second candidate branch, and the third candidate branch, a service instance selection chain corresponding to the request fragment is determined, forming a dual-mapping candidate result.

[0011] As a preferred embodiment of the cloud storage-based database service allocation method of the present invention, the generation of the database service allocation plan specifically includes: Select the database service instance corresponding to the first candidate branch from the dual mapping candidate results as the target service instance; When the first candidate branch is missing from the dual mapping candidate results, the database service instance corresponding to the second candidate branch is selected as the target service instance. When the first and second candidate branches are missing from the dual mapping candidate results, the database service instance corresponding to the third candidate branch is selected as the target service instance. The request fragments in the request fragment queue are merged according to the target service instance to form a service execution unit. The service execution unit, the file fragment reading range and the target cache object identifier are then encapsulated to obtain the database service allocation plan.

[0012] As a preferred embodiment of the cloud storage-based database service allocation method of the present invention, the row group data filtering and cache object retention specifically include: According to the database service allocation plan, the target cache object is located and the compressed file fragment is parsed to obtain the target row group data. The target row group data is then processed using data filtering conditions and query operators to obtain the row group filtering results. When the target cached object is hit, the corresponding cached object is marked as an accessed object, and the cache access record is obtained. When the target cache object is not found, the continuous compressed file segments are obtained according to the file segment reading range, and a new cache block group is established to obtain the new cache object; When the separate cache directory lacks storage space for newly added cached objects, the cached objects in the cache queue are inspected, the access flags of accessed objects are cleared, and cached objects not referenced by the database service allocation plan are removed, thus obtaining the cached object retention result; The location of newly added cached objects is recorded by retaining the results of the cached objects, resulting in the updated separate cache directory.

[0013] As a preferred embodiment of the cloud storage-based database service allocation method of the present invention, the step of generating database fragment service execution results specifically includes: The row group filtering results are combined according to the query operators in the database service request to obtain the database service results. The common access path is corrected by using cached access records and newly added cached objects to obtain the service execution record. Based on the service execution records, the number of fragmented copies of the same database table stored on each storage node is counted, and non-primary copies that are not referenced by the service execution unit are selected to obtain the copies to be migrated; The replicas to be migrated are split according to the circular layout segment identifier to obtain the replica data segment. The available storage nodes that save the adjacent circular layout segments are called to process the replica data segment to obtain the encoded transmission segment. The target storage node restores the replica data segment through the encoded transmission segment, and combines the restored replica data segment to obtain a new fragmented replica; After verifying the new fragment replicas, delete the replicas to be migrated from the original storage nodes, and revise the storage node where the replicas are located, the replica version, the replica reordering number, and the circular layout segment identifier to obtain the updated fragment replica service catalog. The updated fragment replica service directory and the updated separate cache directory are used to record database service results, service execution records and new fragment replica locations, and database fragment service execution results are generated.

[0014] In a second aspect, the present invention provides a computer device including a memory and a processor, wherein the memory stores a computer program, wherein: when the computer program is executed by the processor, it implements any step of the cloud storage-based database service allocation method described in the first aspect of the present invention.

[0015] Thirdly, the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein: when the computer program is executed by a processor, it implements any step of the cloud storage-based database service allocation method described in the first aspect of the present invention.

[0016] The beneficial effects of this invention are as follows: By incorporating fragment replica location, row group address index, target cache object location, and database service instance selection results into the same service allocation chain, database service requests can be allocated based on the actual storage location of the target data and the cache availability status. Compared to scheduling based solely on instance load or primary replica location, this reduces the probability of requests falling into non-target cache instances, and decreases cloud storage origin access and cross-node file fragment reading. Through the coordinated limitation of consecutive key range fragments and row group address indexes, the target data range can be narrowed before service allocation, improving the positioning accuracy of target row groups in columnar storage files and preventing irrelevant row groups and file fragments from participating in subsequent processing. By setting common access paths and common access cache packages, caching between multiple consecutive file fragments can be enhanced. Reuse relationships enable frequently accessed data fragments to be uniformly identified as cache objects, thereby improving cache object hit rate and compressed file fragment reuse rate. Through dual-mapping candidate results, database service instances that simultaneously hold valid copies and target cache objects can be prioritized, ensuring that service allocation is consistent with the data copy status and cache object status, thus improving the determinism and stability of database service allocation. Through cache object retention and circular layout coding rebalancing, fragmented copy service directories and separate cache directories can continue to be corrected after service execution, reducing the impact of copy migration, cache replacement, and directory lag on subsequent request allocation, thereby improving the service continuity, query response efficiency, and storage resource utilization of cloud storage databases in scenarios with high-concurrency queries, sharded copy migration, and dynamic changes in cache objects. Attached Figure Description

[0017] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 A flowchart illustrating the allocation method for cloud storage-based database services.

[0019] Figure 2 A diagram illustrating the construction of the service catalog for fragment replicas and the generation of request fragment queues.

[0020] Figure 3 A schematic diagram is constructed for matching common access paths and constructing dual mapping candidate results.

[0021] Figure 4 A diagram illustrating the allocation of database service plans for execution and catalog updates. Detailed Implementation

[0022] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0023] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and those skilled in the art can make similar extensions without departing from the spirit of the invention. Therefore, the invention is not limited to the specific embodiments disclosed below.

[0024] Secondly, the term "one embodiment" or "embodiment" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The phrase "in one embodiment" appearing in different places in this specification does not necessarily refer to the same embodiment, nor is it a single or selective embodiment that is mutually exclusive with other embodiments.

[0025] Reference Figures 1-4 This is one embodiment of the present invention, which provides a database service allocation method based on cloud storage, including the following steps: S1. Divide the cloud storage database table into consecutive key range fragments according to the valid key columns, and create distributed replicas and row group address indexes for the consecutive key range fragments, generating a fragment replica service catalog.

[0026] S1.1. Obtain the database table identifier to be allocated and the columnar storage file path from the metadata directory of the cloud storage database instance. Determine the cloud storage database table according to the database table identifier to be allocated, and obtain the table structure metadata of the cloud storage database table. The table structure metadata includes the database table identifier, column identifier, column type, primary key constraint, sort key declaration, columnar storage file path, columnar storage file footer position, and row group statistical index position.

[0027] Extract valid key columns from the table structure metadata. Valid key columns are the primary key column and sort key column used in the cloud storage database table to maintain data sequential access. When both a primary key column and a sort key column exist in the cloud storage database table, the primary key column is selected as the valid key column. When the cloud storage database table does not have a primary key column, the sort key column recorded in the columnar storage file metadata is selected as the valid key column, thus obtaining the valid key column record.

[0028] The data records in the cloud storage database table are arranged in the order of key values ​​in the valid key column records. Combined with the page footer position of the columnar storage file and the row group statistical index position, the minimum and maximum key values ​​of the valid key columns in each row group are obtained. The minimum key value of the first row group is used as the starting key value of the first continuous key range, and the minimum key value of the adjacent row group is used as the ending key value of the previous continuous key range, so that the continuous key range adopts a half-open interval that includes the starting key value and excludes the ending key value. When the same key value crosses adjacent row groups, the ending key value of the continuous key range is moved to the end of the last data record with the same key value. The continuous key range containing the maximum key value is closed with the maximum key value as the ending key value, resulting in continuous key range fragments.

[0029] S1.2. Set fragment identifiers for consecutive key range fragments and deploy distributed replicas corresponding to consecutive key range fragments on different storage nodes; the distributed replicas include one primary replica and two secondary replicas, which are located on three different storage nodes. When a storage node is in an unavailable state, it does not participate in the distributed replica deployment. After the distributed replica deployment is completed, record the fragment identifier, consecutive key range, storage node identifier, primary replica marker, and secondary replica marker to obtain the fragment replica location table.

[0030] Within a contiguous key range of fragments, locate the stripes, target columns, row groups, and file fragments of the columnar storage file. The target column is saved as a column identifier during the fragment copy service directory generation stage. The target column location is completed when the target column in the database service request matches the column identifier. The row group uses the stripe identifier, column identifier, and row group identifier to form a logical address. The file fragment uses the columnar storage file path, starting offset, and fragment length to form a physical address. The logical address and physical address are saved according to the row group identifier to obtain the row group address index.

[0031] S1.3. Establish a root fragment record using fragment identifier, consecutive key range, fragment replica location table, primary replica marker, secondary replica marker, and row group address index; the root fragment record stores the key range location, scattered replica location, and row group access location of consecutive key range fragments. The key range location in the root fragment record forms a key range lookup item, the scattered replica location forms a replica lookup item, and the row group access location forms a row group lookup item.

[0032] The directory lookup entry is set up according to the fragment identifier as the key range lookup item, replica lookup item, and row group lookup item. After receiving the key range conditions, target columns, and data filtering conditions obtained from subsequent database service requests, the directory lookup entry can locate the storage nodes where the target fragment, primary replica, and secondary replica are located, as well as the logical and physical addresses of the target row group, and generate the fragment replica service directory.

[0033] S2. Use the fragmented replica service directory to parse database service requests, filter the hit row groups in the target fragments and merge consecutive file fragments to generate a request fragment queue.

[0034] S2.1. The query interface of the database compute node receives database service requests. The database service request includes the target database table, target column, key range conditions, and data filtering conditions. The target database table is used to match the database table identifier in the fragment replica service directory, the target column is used to match the column identifier in the fragment replica service directory, the key range conditions are used to match the key range lookup items in the fragment replica service directory, and the data filtering conditions are used to match the row group statistical range corresponding to the target column, thus obtaining the request parsing result.

[0035] The target database table and key range conditions in the request parsing results are used to retrieve the directory lookup entry in the fragment replica service directory, and the continuous key ranges that intersect with the key range conditions are searched in the key range lookup items; when the key range conditions fall into a continuous key range, the corresponding continuous key range fragment is determined as the target fragment; when the key range conditions span multiple continuous key ranges, multiple target fragments are determined in sequence according to the key value order of the continuous key ranges to obtain the target fragment sequence.

[0036] S2.2. Retrieve the row group lookup item corresponding to the target fragment sequence through the target column in the request parsing result, and obtain the row group statistical range of the target column in each target fragment; the row group statistical range includes the minimum column value, maximum column value, row group identifier, logical address and physical address of the target column in the corresponding row group. When there is no intersection between the data filtering condition and the minimum column value to the maximum column value, the corresponding row group is removed. When there is an intersection between the data filtering condition and the minimum column value to the maximum column value, the corresponding row group is retained, thus obtaining the set of hit row groups.

[0037] The hit row groups are arranged according to their physical addresses, which include the columnar storage file path, the starting offset, and the fragment length. When two adjacent hit row groups belong to the same columnar storage file path, and the sum of the starting offset and the fragment length of the previous hit row group equals the starting offset of the next hit row group, the file fragments corresponding to the two adjacent hit row groups are concatenated into a single continuous file fragment. When two adjacent hit row groups have different columnar storage file paths, or the sum of the starting offset and the fragment length of the previous hit row group does not equal the starting offset of the next hit row group, the corresponding file fragments are retained to obtain the file fragment reading range.

[0038] The cache block group identifier is set according to the file fragment reading range. The cache block group identifier corresponds to a file fragment reading range under the same columnar storage file path. The target fragment sequence, the hit row group set, the file fragment reading range, the cache block group identifier, the target copy location, the row group logical address, and the row group physical address are arranged into a request fragment queue according to the execution order of the database service request.

[0039] S3. Based on the request fragment queue, match the common access path, merge consecutive file fragments into a common access cache package, and construct a dual mapping candidate result by combining the fragment replica service directory.

[0040] S3.1. Obtain the file fragment read range, cache block group identifier, target fragment sequence, target replica location, row group logical address, and row group physical address from the request fragment queue; the cache block group identifier corresponds to a file fragment read range under the same columnar storage file path, and the file fragment read range corresponds to the hit row group in the target fragment. Arrange the file fragment read ranges according to the execution order of the database service requests to obtain the request cache sequence.

[0041] The common access path of the request cache sequence is examined by checking the frequent pattern tree entry in the root fragment record; the frequent pattern tree entry is the tree index entry in the root fragment record that stores the common occurrence order of cache block groups, and the common access path is the access path of the cache block group identifier appearing consecutively under the frequent pattern tree entry.

[0042] When multiple cache block group identifiers in the request cache sequence match consecutive nodes of the same common access path, a common access path matching result containing consecutively occurring cache block groups is obtained; when cache block group identifiers in the request cache sequence do not fall into the same common access path, a common access path matching result not containing consecutively occurring cache block groups is obtained.

[0043] When the common access path matching result contains multiple consecutive cache block groups, the file segment reading range corresponding to the consecutive cache block groups is encapsulated into a common access cache package in the order of columnar storage of file path and physical address, and the corresponding cache block group identifier, file segment reading range, row group logical address and row group physical address are retained.

[0044] When the common access path matching result does not contain consecutive cache block groups, the file fragment reading range in the request cache sequence is retained, and the cache block group identifier corresponding to the retained file fragment reading range is used as the cache object identifier to be allocated, thus obtaining the cache object to be allocated.

[0045] S3.2. Determine the common access cache package and the cache object to be allocated as the target cache object, and match the target cache object with the location of the existing cache object in the database service instance; the location of the existing cache object includes the database service instance identifier, storage node identifier, cache node identifier, cache block group identifier, common access cache package identifier, and cache object storage address.

[0046] When the target cache object is a cache object to be allocated, the cache block group identifier of the cache object to be allocated is compared with the cache block group identifier in the location of the existing cache object. The file segment reading range of the cache object to be allocated is also checked to ensure that it matches the file segment reading range pointed to by the cache object storage address corresponding to the location of the existing cache object. If the cache block group identifier and the file segment reading range are consistent, the cache object to be allocated is determined to be a successful match. The corresponding database service instance identifier, storage node identifier, cache node identifier, and cache object storage address are added to the target cache object list. If the cache block group identifier is inconsistent or the file segment reading range is inconsistent, the cache object to be allocated is determined to be a failed match. The cache object to be allocated is marked as a cache miss, and the cache object identifier and file segment reading range are retained in the target cache object list.

[0047] When the target cached object is a common access cache package, the common access cache package identifier is compared with the common access cache package identifier in the existing cached object location, and it is verified whether all cache block group identifiers contained in the common access cache package exist in the existing cached object location corresponding to the same database service instance identifier. If the common access cache package identifier is consistent and all cache block group identifiers exist in the existing cached object location corresponding to the same database service instance identifier, the common access cache package is determined to be a successful match, and the corresponding database service instance identifier, storage node identifier, cache node identifier, and cached object storage address are added to the target cached object list. If the common access cache package identifier is inconsistent or any cache block group identifier does not exist in the existing cached object location corresponding to the same database service instance identifier, the common access cache package is determined to be a failed match, the common access cache package is marked as a cached object miss, and the common access cache package identifier and the file fragment reading range contained in the common access cache package are retained in the target cached object list.

[0048] The successfully matched target cache objects, the cache objects that did not match, the database service instance identifier, the storage node identifier, the cache node identifier, the cache object storage address, and the file fragment reading range are combined into a separate cache directory.

[0049] The target cache object list is embedded into the request fragment queue, and the target fragment identifier, cache block group identifier, and common access cache package identifier are obtained from the request fragment queue after embedding the target cache object list. The target fragment identifier comes from the target fragment sequence, the cache block group identifier comes from the file fragment reading range, and the common access cache package identifier comes from the common access cache package. The target fragment identifier, cache block group identifier, and common access cache package identifier are combined into an allocation index key set.

[0050] S3.3. By assigning the target fragment identifier in the index key set, retrieve the replica lookup item in the fragment replica service directory, determine the distributed replicas whose replica version is consistent with the fragment replica service directory and whose storage node is not in an unavailable state as valid replicas, and determine the database service instance where the valid replica is located as the replica candidate instance, thus obtaining the replica candidate instance set.

[0051] By retrieving the separate cache directory using the cache block group identifier and common access cache package identifier in the assigned index key set, the database service instance that has saved the target cache object is identified as a cache candidate instance, thus obtaining a cache candidate instance set.

[0052] S3.4. Cross-validate the database service instances in the replica candidate instance set and the cache candidate instance set; when a database service instance exists in both the replica candidate instance set and the cache candidate instance set, mark that the database service instance holds both a valid replica and the target cache object, and obtain the overlapping candidate instance set.

[0053] When the set of overlapping candidate instances is not empty, the database service instance in the set of overlapping candidate instances is determined as the preferred candidate instance, resulting in the first candidate branch; when the set of overlapping candidate instances is empty and the set of replica candidate instances contains the database service instance corresponding to the replica tag, the database service instance with the latest replica version is determined as the replacement candidate instance, resulting in the second candidate branch; when the set of overlapping candidate instances is empty and the set of replica candidate instances contains the database service instance corresponding to the primary replica tag, the database service instance corresponding to the primary replica tag is determined as the back-to-origin candidate instance, resulting in the third candidate branch.

[0054] The service instance selection chain corresponding to the request fragment is determined according to the order of the first, second, and third candidate branches. Specifically: when a database service instance exists in the first candidate branch, the preferred candidate instance in the first candidate branch is placed at the beginning of the service instance selection chain; when the first candidate branch is empty and a database service instance exists in the second candidate branch, the alternative candidate instance in the second candidate branch is placed at the beginning of the service instance selection chain; when both the first and second candidate branches are empty and a database service instance exists in the third candidate branch, the origin-following candidate instance in the third candidate branch is placed at the beginning of the service instance selection chain; when the same candidate branch contains multiple database service instances, they are sorted from newest to oldest replica versions, and if the replica versions are the same, they are sorted in ascending order of database service instance identifiers, and the database service instance at the top of the sorted list is selected as the first database service instance of the corresponding candidate branch. The service instance selection chain retains the request fragment identifier, target fragment identifier, target cache object identifier, first candidate branch, second candidate branch, third candidate branch, and first database service instance, forming a dual-mapping candidate result.

[0055] S4. Select a database service instance that simultaneously holds a valid copy and the target cache object based on the dual mapping candidate results, and merge request fragments of the same instance to generate a database service allocation plan.

[0056] S4.1. Obtain the service instance selection chain from the dual mapping candidate results; the service instance selection chain includes the request fragment identifier, the target fragment identifier, the target cache object identifier, the first candidate branch, the second candidate branch, the third candidate branch, and the first and second database service instances. The first candidate branch corresponds to the database service instance that holds both a valid copy and the target cache object. The second candidate branch corresponds to the database service instance that holds a valid copy and has the latest copy version. The third candidate branch corresponds to the database service instance that holds the primary copy marker and needs to obtain the file fragment reading range from the cloud storage.

[0057] The target service instance is selected according to the order of the first candidate branch, the second candidate branch, and the third candidate branch. When the first candidate branch has a database service instance, the first database service instance in the first candidate branch is taken as the target service instance. When the first candidate branch does not have a database service instance but the second candidate branch has a database service instance, the first database service instance in the second candidate branch is taken as the target service instance. When neither the first nor the second candidate branch has a database service instance but the third candidate branch has a database service instance, the first database service instance in the third candidate branch is taken as the target service instance, thus obtaining the correspondence between the request fragment and the target service instance.

[0058] S4.2. Merge request fragments in the request fragment queue according to the correspondence between request fragments and target service instances; request fragments belonging to the same target service instance are arranged according to the execution order of database service requests, and request fragments belonging to different target service instances are arranged separately; when multiple request fragments corresponding to the same target service instance have the same target fragment identifier and adjacent file fragment reading range, merge multiple request fragments into one continuous reading task; when multiple request fragments corresponding to the same target service instance have the same common access cache package identifier, merge multiple request fragments into one cache package access task to obtain a service execution unit.

[0059] The service execution unit is supplemented with the file fragment reading range and the target cache object identifier; the continuous reading task retains the starting file fragment reading range and the ending file fragment reading range, the cache package access task retains the common access cache package identifier, and the request fragment corresponding to the cache object that was not hit retains the cache object identifier to be allocated and the file fragment reading range, thus obtaining the service execution unit record.

[0060] The service execution unit records are grouped according to the target service instances, and for each target service instance, the request fragment identifier, target fragment identifier, target cache object identifier, file fragment reading range, common access cache package identifier, cache object identifier to be allocated, and service execution order are recorded to obtain the database service allocation plan.

[0061] S5. Perform row group data filtering, cache object retention, and circular layout coding rebalancing on the database service allocation plan, update the fragment replica service directory and the separate cache directory, and generate the database fragment service execution results.

[0062] S5.1. Locate the target cache object according to the target service instance, service execution unit, target cache object identifier, and file fragment reading range in the database service allocation plan; when the target cache object identifier exists in the separate cache directory, obtain the cache object storage address, columnar storage file path, starting offset, and fragment length according to the target cache object identifier, and extract the byte data corresponding to the starting offset and fragment length from the cache object storage address to obtain the compressed file fragment; when the target cache object identifier does not exist in the separate cache directory, obtain the continuous compressed file fragments from the cloud storage according to the columnar storage file path, starting offset, and fragment length in the file fragment reading range, and establish the continuous compressed file fragments into a new cache block group to obtain the new cache object.

[0063] Based on the row group logical address and row group physical address in the database service allocation plan, locate the compressed byte stream of the target row group in the compressed file fragment or continuous compressed file fragment, decompress the compressed byte stream of the target row group according to the compression encoding type recorded in the columnar storage file footer, and restore the target row group data according to the column identifier corresponding to the target column.

[0064] S5.2. Perform row-level filtering on the target row group data using the data filtering conditions in the database service request. Specifically: parse the column values ​​in the target row group data according to the column type of the target column; apply the comparison operators, interval boundaries, and equality lists in the data filtering conditions to the corresponding column values ​​one by one; retain the row records that meet the data filtering conditions and remove the row records that do not meet the data filtering conditions to obtain the filtered row group data; retain the corresponding column values ​​from the filtered row group data according to the projection field in the query operator; sort the retained row records by key value according to the sorting field in the query operator; divide the row record set according to the grouping field in the query operator; and calculate the grouping results according to the aggregate function in the query operator to obtain the row group filtering results; the query operators are the projection field, sorting field, grouping field, and aggregate function declared in the database service request.

[0065] When the target cached object hits the separate cache directory, the corresponding cached object is marked as an accessed object, and a cache access record is obtained; when the target cached object does not hit the separate cache directory, the cache block group identifier, file fragment reading range, and cached object storage address of the newly added cached object are added to the cache access record, resulting in a cache access record containing both the hit status and the newly added status.

[0066] S5.3. When the separate cache directory lacks storage space for newly added cache objects, the cache objects in the separate cache directory are inspected according to the cache queue order; when an accessed object is encountered, the access mark of the accessed object is cleared and the accessed object is retained; when a cache object not referenced by the database service allocation plan is encountered, the cache object not referenced by the database service allocation plan is removed and the cache object storage address is released; when the released cache object storage address meets the storage requirements of the newly added cache object, the inspection of the cache queue is stopped, the newly added cache object is registered to the released cache object storage address, and the cache object retention result and the updated separate cache directory are obtained.

[0067] S5.4. The database compute node receives the row group filtering results returned by each target service instance and completes the result synthesis according to the query operators in the database service request. The query operators include projection fields, sorting fields, grouping fields, and aggregation functions. The database compute node first unifies the column structure of each row group filtering result according to the projection fields, then merges the sorted row records according to the sorting fields, and merges the aggregation results of the same group according to the grouping fields and aggregation functions to obtain the database service result. At the same time, it uses the hit status, new status, cache block group identifier, and new cache object in the cache access record to correct the common access path in the root fragment record to obtain the service execution record.

[0068] Based on the service execution records, the number of fragmented replicas of the same database table stored on each storage node is counted. Then, the scattered replicas that are not referenced by the service execution unit and have the slave replica mark are selected from the storage node with the most replicas to obtain the replicas to be migrated. When multiple scattered replicas meet the condition of not being referenced by the service execution unit and having the slave replica mark, the replicas to be migrated are selected in ascending order of replica reorder number to ensure that the replicas to be migrated have a definite source.

[0069] S5.5. Split the replicas to be migrated according to the circular layout segment identifier to obtain replica data segments; the circular layout segment identifier is the segment number in the root fragment record used to indicate the sequence number of the replica data segments in the circular layout, and the replica data segments maintain a fixed order according to the circular layout segment identifier.

[0070] Based on the circular layout segment identifier of the replica to be migrated, the adjacent circular layout segments are found, and the storage nodes that store the adjacent circular layout segments and are not in an unavailable state are selected as available storage nodes. The available storage nodes extract the adjacent replica data segments with the same segment number as the replica data segments from the distributed replicas corresponding to the adjacent circular layout segments, and align the replica data segments and adjacent replica data segments by byte number. When the lengths of the replica data segments and adjacent replica data segments are inconsistent, the shorter replica data segment is padded with zeros to the end to the same length and the original length is recorded. The available storage nodes perform XOR encoding on bytes with the same byte number to obtain the encoded transmission segment.

[0071] S5.6. The target storage node restores the replica data segment through the encoded transmission segment, and combines the restored replica data segment according to the circular layout segment identifier to obtain a new fragment replica. The target storage node verifies the fragment identifier, consecutive key range, primary replica marker, secondary replica marker, and replica version of the new fragment replica. After verification, the replica to be migrated in the original storage node is deleted, and the storage node where the replica is located, the replica version, the replica reorder number, and the circular layout segment identifier are revised to obtain the updated fragment replica service catalog.

[0072] The updated fragment replica service directory and the updated separate cache directory are used to record database service results, service execution records, the location of newly added cached objects, and the location of new fragment replicas, and to generate database fragment service execution results.

[0073] This embodiment also provides a computer device applicable to the cloud storage-based database service allocation method, comprising: a memory and a processor; the memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions to implement the cloud storage-based database service allocation method proposed in the above embodiment.

[0074] The computer device can be a terminal, comprising a processor, memory, communication interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, carrier networks, NFC (Near Field Communication), or other technologies. The display screen can be an LCD screen or an e-ink screen. The input devices can be a touch layer covering the display screen, buttons, a trackball, or a touchpad on the computer device's casing, or an external keyboard, touchpad, or mouse.

[0075] This embodiment also provides a storage medium storing a computer program, which, when executed by a processor, implements the cloud storage-based database service allocation method proposed in the above embodiments. The storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as Static Random Access Memory (SRAM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Erasable Programmable Read Only Memory (EPROM), Programmable Red-Only Memory (PROM), Read-Only Memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0076] In summary, this invention incorporates fragment replica location, row group address index, target cache object location, and database service instance selection results into the same service allocation chain. This allows database service requests to be allocated based on the actual storage location of the target data and the cache availability status. Compared to scheduling based solely on instance load or primary replica location, this reduces the probability of requests falling into non-target cache instances, decreasing cloud storage origin access and cross-node file fragment reading. Through the coordinated limitation of consecutive key range fragments and row group address indexes, the target data range can be narrowed before service allocation, improving the positioning accuracy of target row groups in columnar storage files and preventing irrelevant row groups and file fragments from participating in subsequent processing. Furthermore, by setting common access paths and common access cache packages, the cache replication between multiple consecutive file fragments can be enhanced. By using relationships, frequently accessed data fragments are uniformly identified as cache objects, thereby improving cache object hit rate and compressed file fragment reuse rate. Through dual-mapping candidate results, database service instances that simultaneously hold valid copies and target cache objects can be prioritized, ensuring that service allocation is consistent with the data copy status and cache object status, thus improving the determinism and stability of database service allocation. Through cache object retention and circular layout coding rebalancing, fragmented copy service directories and separate cache directories can continue to be corrected after service execution, reducing the impact of copy migration, cache replacement, and directory lag on subsequent request allocation. This improves the service continuity, query response efficiency, and storage resource utilization of cloud storage databases in scenarios with high-concurrency queries, sharded copy migration, and dynamic changes in cache objects.

[0077] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A database service allocation method based on cloud storage, characterized in that: include, The cloud storage database table is divided into continuous key range fragments according to the valid key columns, and distributed replicas and row group address indexes are created for the continuous key range fragments to generate a fragment replica service catalog. The database service requests are parsed using the fragmented replica service directory, the matching row groups in the target fragments are filtered and continuous file fragments are merged to generate a request fragment queue; Based on the request fragment queue matching common access paths, consecutive file fragments are merged into common access cache packages, and combined with the fragment replica service directory to construct dual mapping candidate results; Based on the dual mapping candidate results, select the database service instance that simultaneously holds a valid copy and the target cache object, and merge request fragments of the same instance to generate a database service allocation plan; The database service allocation plan performs row group data filtering, cache object retention, and cyclic layout encoding rebalancing, updates the fragment replica service directory and the separate cache directory, and generates the database fragment service execution results.

2. The database service allocation method based on cloud storage as described in claim 1, characterized in that: The service catalog for generating fragment copies specifically includes: Extract the valid key columns from the cloud storage database table, divide the continuous key range according to the key value order of the valid key columns and set fragment identifiers, deploy distributed replicas on different storage nodes and mark the primary replicas and secondary replicas to obtain the fragment replica location table; Locate the stripes, target columns, row groups, and file fragments of a columnar storage file within a contiguous key range fragment to obtain the row group address index; The root fragment record is established by using fragment identifier, consecutive key range, fragment replica location table, primary replica marker, secondary replica marker, and row group address index. The search items in the root fragment record are then extracted to set the directory search entry, thus obtaining the fragment replica service directory.

3. The database service allocation method based on cloud storage as described in claim 2, characterized in that: The generated request fragment queue specifically includes: Parse the database service request and retrieve the continuous key range in the fragment replica service directory to determine the target fragment sequence; Extract the row group statistical range of the target column within the target fragment sequence, remove the row groups that do not intersect with the data filtering conditions, and splice adjacent file fragments in physical address order to obtain the set of hit row groups and the file fragment reading range. The cache block group identifier is set according to the file fragment reading range, and the target fragment sequence, the hit line group set, the file fragment reading range, and the cache block group identifier are arranged into a request fragment queue.

4. The database service allocation method based on cloud storage as described in claim 3, characterized in that: The step of matching common access paths based on request fragment queues and merging consecutive file fragments into a common access cache package specifically includes: Extract the file fragment reading range and cache block group identifier from the request fragment queue to obtain the request cache sequence. Then, check the common access path of the request cache sequence through the frequent pattern tree entry in the root fragment record to obtain the common access path matching result. When the common access path matching result contains multiple consecutive cache block groups, the corresponding file segment reading range is encapsulated to obtain the common access cache package; When the common access path matching result does not contain a group of consecutively occurring cache blocks, the file fragment reading range in the requested cache sequence is retained to obtain the cache object to be allocated. The common access cache package and the cache object to be allocated are identified as the target cache objects, and the target cache objects are matched with the locations of the existing cache objects in the database service instance to obtain a list of target cache objects and a separate cache directory.

5. The database service allocation method based on cloud storage as described in claim 4, characterized in that: The construction of the dual-mapping candidate results specifically includes: Embed the list of target cached objects into the request fragment queue, and extract the target fragment identifier, cache block group identifier and common access cache package identifier from it to obtain the allocation index key set; By assigning target fragment identifiers in the index key set, the database service instance where the valid replica resides is retrieved in the fragment replica service directory, thus obtaining a set of candidate replica instances; By assigning cache block group identifiers and common access cache package identifiers in the index key set, the database service instance where the target cache object is located is retrieved in the separate cache directory to obtain a cache candidate instance set; Cross-validate the database service instances in the candidate replica instance set and the candidate cache instance set, and mark the instances that simultaneously hold a valid replica and the target cache object to obtain the overlapping candidate instance set; When the set of overlapping candidate instances is not empty, the database service instance in the set of overlapping candidate instances is determined as the preferred candidate instance, and the first candidate branch is obtained. When the overlapping candidate instance set is empty and the replica candidate instance set contains the database service instance corresponding to the replica tag, the database service instance with the latest replica version is determined as the replacement candidate instance, and the second candidate branch is obtained. When the overlapping candidate instance set is empty and the replica candidate instance set contains the database service instance corresponding to the primary replica mark, the database service instance corresponding to the primary replica mark is determined as the source candidate instance, and the third candidate branch is obtained. Based on the first candidate branch, the second candidate branch, and the third candidate branch, a service instance selection chain corresponding to the request fragment is determined, forming a dual-mapping candidate result.

6. The database service allocation method based on cloud storage as described in claim 5, characterized in that: The generated database service allocation plan specifically includes: Select the database service instance corresponding to the first candidate branch from the dual mapping candidate results as the target service instance; When the first candidate branch is missing from the dual mapping candidate results, the database service instance corresponding to the second candidate branch is selected as the target service instance. When the first and second candidate branches are missing from the dual mapping candidate results, the database service instance corresponding to the third candidate branch is selected as the target service instance. The request fragments in the request fragment queue are merged according to the target service instance to form a service execution unit. The service execution unit, the file fragment reading range and the target cache object identifier are then encapsulated to obtain the database service allocation plan.

7. The database service allocation method based on cloud storage as described in claim 6, characterized in that: The row group data filtering and cache object retention specifically include: According to the database service allocation plan, the target cache object is located and the compressed file fragment is parsed to obtain the target row group data. The target row group data is then processed using data filtering conditions and query operators to obtain the row group filtering results. When the target cached object is hit, the corresponding cached object is marked as an accessed object, and the cache access record is obtained. When the target cache object is not found, the continuous compressed file segments are obtained according to the file segment reading range, and a new cache block group is established to obtain the new cache object; When the separate cache directory lacks storage space for newly added cached objects, the cached objects in the cache queue are inspected, the access flags of accessed objects are cleared, and cached objects not referenced by the database service allocation plan are removed, thus obtaining the cached object retention result; The location of newly added cached objects is recorded by retaining the results of the cached objects, resulting in the updated separate cache directory.

8. The database service allocation method based on cloud storage as described in claim 7, characterized in that: The execution result of the database fragment generation service specifically includes: The row group filtering results are combined according to the query operators in the database service request to obtain the database service results. The common access path is corrected by using cached access records and newly added cached objects to obtain the service execution record. Based on the service execution records, the number of fragmented copies of the same database table stored on each storage node is counted, and non-primary copies that are not referenced by the service execution unit are selected to obtain the copies to be migrated; The replicas to be migrated are split according to the circular layout segment identifier to obtain the replica data segment. The available storage nodes that save the adjacent circular layout segments are called to process the replica data segment to obtain the encoded transmission segment. The target storage node restores the replica data segment through the encoded transmission segment, and combines the restored replica data segment to obtain a new fragmented replica; After verifying the new fragment replicas, delete the replicas to be migrated from the original storage nodes, and revise the storage node where the replicas are located, the replica version, the replica reordering number, and the circular layout segment identifier to obtain the updated fragment replica service catalog. The updated fragment replica service directory and the updated separate cache directory are used to record database service results, service execution records and new fragment replica locations, and database fragment service execution results are generated.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, it implements the steps of the cloud storage-based database service allocation method according to any one of claims 1 to 8.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by the processor, it implements the steps of the cloud storage-based database service allocation method according to any one of claims 1 to 8.

Images