Data fragmentation method and device, storage medium and electronic equipment
By disassembling key-value information on the historical execution transactions of the data to be sharded by the database, determining similar feature information and sharding data, the shard locking and conflict problems in transaction processing are solved, and the processing efficiency and performance of the database are improved.
Patent Information
- Application Number
- CN202410989353.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-23
- Publication Date
- 2024-11-12
AI Technical Summary
When multiple transactions need to be processed, the existing data sharding method causes other transactions to wait for the transaction being processed to complete, and even lead to transaction processing failure, reducing the processing power and utilization of the database.
By obtaining the data to be sharded in the database and its key-value information of the historical execution transaction, disassembly and process to determine similar feature information, data sharding is performed based on similar feature information, and data to be sharded are allocated to the same similar feature information to the same Sharding.
Reduces the number of shard locks during transaction processing, alleviates data conflicts between multiple transactions, and improves the processing efficiency and performance of the database.
Smart Images

Figure CN118939728A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of data processing technology, and in particular to a data slicing method, device, storage medium and electronic device. Background Art
[0002] Data sharding (Maximum Transmission Unit, MTU) refers to distributing the data in the database to multiple data nodes according to a certain strategy, and each node only stores a subset of the entire data.
[0003] At present, in the process of processing large-scale data in non-relational database clusters, the main sharding method used is hashing or range sharding. The data of a certain key value is distributed to the nodes of the database service cluster, and the database service cluster is organized by multiple nodes to avoid the problem of insufficient performance of a single node to support large-scale data.
[0004] However, with this data sharding method, the keys of the data items requested by a transaction are randomly distributed in the database nodes. When the database transaction is submitted, the data required to process a transaction will be randomly distributed in multiple shards. Then, during the transaction processing, all shards where the required data is distributed will be locked until the transaction processing is completed. In the case where multiple transactions need to be processed, if a large number of shards need to be locked for one transaction processing, other transactions will need to wait for the ongoing transaction processing to complete before they can start the transaction processing, or even cause other transactions to fail, thereby reducing the processing capacity and utilization of the database. Summary of the invention
[0005] In view of this, the present application provides a data sharding method, device, storage medium and electronic device, the main purpose of which is to improve the current technical problem that when multiple transactions need to be processed, other transactions need to wait for the ongoing transaction processing to be completed before starting transaction processing, and may even cause other transactions to fail, thereby reducing the processing capacity and utilization of the database.
[0006] In a first aspect, the present application provides a data sharding method, comprising:
[0007] Acquire the data to be sharded in the database and the historical executed transactions corresponding to the data to be sharded, wherein the historical executed transactions are transactions that have used the data to be sharded during processing;
[0008] Decomposing the key value information of the historically executed transaction to determine similarity feature information corresponding to the historically executed transaction, wherein the similarity feature information includes transaction similarity feature information and key similarity feature information;
[0009] The data to be sharded is processed according to the similar feature information, wherein the data to be sharded containing the same similar feature information is allocated to the same shard.
[0010] In a second aspect, the present application provides a data slicing device, comprising:
[0011] An acquisition module is configured to acquire the data to be sharded in the database and historical execution transactions corresponding to the data to be sharded, wherein the historical execution transactions are transactions that have used the data to be sharded during processing;
[0012] A determination module is configured to disassemble the key value information of the historically executed transaction to determine similar feature information corresponding to the historically executed transaction, wherein the similar feature information includes transaction similar feature information and key similar feature information;
[0013] The sharding module is configured to perform data sharding processing on the data to be sharded according to the similar feature information, wherein the data to be sharded containing the same similar feature information is allocated to the same shard.
[0014] In a third aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the data sharding method described in the first aspect is implemented.
[0015] In a fourth aspect, the present application provides an electronic device, comprising a storage medium, a processor, and a computer program stored on the storage medium and executable on the processor, wherein when the processor executes the computer program, a data segmentation method described in the first aspect is implemented.
[0016] In a fifth aspect, the present application provides a computer program product, which includes a computer program, and when the computer program is executed by a processor, the data sharding method of the first aspect is implemented.
[0017] By means of the above technical scheme, the present application provides a data sharding method, device, storage medium and electronic device, which specifically first obtain the data to be sharded in the database and the historical execution transactions corresponding to the data to be sharded, wherein the historical execution transactions are transactions that have used the data to be sharded during the processing; the key value information of the historical execution transactions is disassembled to determine the similar feature information corresponding to the historical execution transactions, wherein the similar feature information includes transaction similar feature information and key similar feature information; based on the similar feature information, the data to be sharded is subjected to data sharding processing, wherein the data to be sharded containing the same similar feature information is allocated to the same shard. Compared with the current existing technology, the present application disassembles and processes the key value information of the historical execution transaction corresponding to the sharded data, determines the transaction similarity feature information and key similarity feature information corresponding to the historical execution transaction, and distributes the sharded data containing the same similarity feature information to the same shard based on the transaction similarity feature information and the key similarity feature information. This can distribute the data of the same transaction in fewer shards. When performing transaction processing, the number of locked shards can be greatly reduced, thereby reducing the number of data required by other transactions being locked. When multiple transactions need to be processed, it can effectively alleviate the conflict between multiple transactions that need to use data in the same shard, thereby improving the processing efficiency of the database and effectively improving the database performance. BRIEF DESCRIPTION OF THE DRAWINGS
[0018] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.
[0019] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.
[0020] Figure 1 A schematic diagram of a data slicing method provided in an embodiment of the present application is shown;
[0021] Figure 2 A schematic diagram showing an example provided by an embodiment of the present application;
[0022] Figure 3 A schematic diagram of a data slicing method provided in an embodiment of the present application is shown;
[0023] Figure 4 A schematic diagram showing an example provided by an embodiment of the present application;
[0024] Figure 5A schematic diagram showing an example provided by an embodiment of the present application;
[0025] Figure 6 A schematic diagram showing an example provided by an embodiment of the present application;
[0026] Figure 7 A schematic diagram showing an example provided by an embodiment of the present application;
[0027] Figure 8 A schematic diagram showing an example provided by an embodiment of the present application;
[0028] Fig. 9 A schematic diagram showing an example provided by an embodiment of the present application;
[0029] Fig.10 A schematic diagram showing an example provided by an embodiment of the present application;
[0030] Fig.11 A schematic diagram of an example process provided by an embodiment of the present application is shown;
[0031] Fig.12 A structural schematic diagram of a data slicing device provided in an embodiment of the present application is shown. DETAILED DESCRIPTION
[0032] The embodiments of the present application will be described in more detail below with reference to the accompanying drawings. It should be noted that the embodiments and features in the embodiments of the present application can be combined with each other without conflict.
[0033] In order to improve the current technical problem that when multiple transactions need to be processed, other transactions need to wait for the ongoing transaction to be completed before starting the transaction, and even cause other transactions to fail, thereby reducing the processing capacity and utilization of the database. This embodiment provides a data sharding method, such as Figure 1 As shown, the method includes:
[0034] Step 101: Obtain the data to be sharded and the historical execution transactions corresponding to the data to be sharded in the database.
[0035] Among them, historical execution transactions are transactions that have used the data to be sharded during the processing process.
[0036] In the embodiments of the present application, data sharding is a common data partitioning technology in database management and distributed systems, which divides a large data set into multiple smaller, more manageable parts, called data fragments (shards). Each data fragment can exist independently on different physical servers or cluster nodes, thereby achieving horizontal expansion (scale out) and load balancing of data.
[0037] For this embodiment, the database is a warehouse that organizes, stores and manages data according to the data structure. It is an important part of the information system. The data in the database is organized, described and stored according to a specific data model. A distributed cluster refers to the use of multiple servers to form a unified whole to jointly provide services to the outside world. Each server in the cluster is peer-to-peer, and can achieve high availability, load balancing and computing power expansion. A distributed database refers to a database system that stores data in multiple physical nodes (servers) in a distributed manner, and is still a complete database logically. These nodes are interconnected through the network, share data distributed everywhere, and provide complete database services. Distributed databases are distributedly stored, and a single node failure will not affect the entire database. Localized data access can be achieved, reducing network transmission overhead. As demand increases, new nodes can be added dynamically. Common distributed databases include: Cassandra, CouchDB, MongoDB, Riak, etc. Designing and managing distributed database systems requires considering many challenges such as data partitioning, replication, consistency, and fault tolerance. A distributed database is a distributed cluster that provides database services. Build a highly available and high-performance database system. Data sharding and request routing are two common concepts in distributed systems. Figure 2 As shown, the client request 1 (Client request 1) is routed. The request 1 is routed to one of the data nodes among data node 1, data node 2, data node 3, and data node 4 through the Router for processing.
[0038] Exemplarily, the database can be a transactional database. Specifically, a transactional database is a database management system (DBMS) that supports transactions and has ACID (Atomicity, Consistency, Isolation, Durability) characteristics, which ensures that the consistency and integrity of data are properly maintained when reading and writing the database in a concurrent operation environment.
[0039] Step 102: Decompose the key value information of the historically executed transactions to determine similar feature information corresponding to the historically executed transactions.
[0040] The similarity feature information includes transaction similarity feature information and key similarity feature information.
[0041] In some examples, the key value information of historically executed transactions is the data key value corresponding to the historically executed transactions. Specifically, in a transaction database, data key values (Key-Value pairs) are the basic organization form of data, especially in a key-value storage database (such as a NoSQL database), where data is stored in the form of key-value pairs. In relational databases, although data is usually organized in a table form, transaction processing still revolves around operating on the key values of data records in the table.
[0042] It should be noted that the data key values within a transaction are related, because transactions are often used to process logical units of multiple database operations, which may involve updating, inserting or deleting multiple data items (key-value pairs). In the same transaction, different key value operations may have obvious correlations.
[0043] As an optional method, transaction similar feature information (TSP) is the similar feature information of the key extracted within the same transaction. In multiple transactions, some data operations (such as read and write operations on the same table or key value) are similar or repeated. In scenarios such as database optimization, transaction analysis, or concurrency control, identifying and processing this TSP can help us better design and optimize database systems, such as reducing repeated operations and improving system performance through batch processing, caching, or pre-computation. In practical applications, for example, in the process of transaction processing, if it is found that a large number of transactions are operating the same key-value pair, then this part of the operation can be regarded as TSP. In this case, you can consider using database indexes, caching technology, or pre-computation results to reduce unnecessary repeated calculations and I / O operations, improve transaction processing efficiency and the overall performance of the system. In addition, in distributed systems, identifying similar parts of transactions also helps to perform reasonable data partitioning and load balancing.
[0044] Optionally, the key similarity feature (Key Similar Part, KSP) is the key similarity feature information extracted after removing the TSP feature.
[0045] Step 103: Perform data sharding processing on the data to be sharded based on similar feature information.
[0046] Among them, the data to be sharded that contains the same or similar feature information is allocated to the same shard.
[0047] In an embodiment of the present application, data sharding processing for the data to be sharded can be performed based on similar feature information and key similar features, respectively. Specifically, data sharding based on transaction similar feature information is to route the transaction similar feature information extracted from all data keys of the transaction; data sharding based on key similar feature information is to match the data key with a previously imported set of key similar feature information, and use the matched key similar feature information for routing.
[0048] Compared with the current existing technology, this embodiment disassembles the key value information of the historical execution transaction corresponding to the sharded data, determines the transaction similarity feature information and key similarity feature information corresponding to the historical execution transaction, and distributes the sharded data containing the same similarity feature information to the same shard based on the transaction similarity feature information and the key similarity feature information. This can distribute the data of the same transaction in fewer shards. When performing transaction processing, the number of locked shards can be greatly reduced, thereby reducing the number of data required by other transactions being locked. When multiple transactions need to be processed, it can effectively alleviate the conflict between multiple transactions that need to use data in the same shard, thereby improving the processing efficiency of the database and effectively improving the database performance.
[0049] As a refinement and extension of the above embodiment, in order to fully illustrate the specific implementation process of the method of this embodiment, this embodiment provides the following Figure 3 The specific method shown includes:
[0050] Step 201: Obtain the data to be sharded and the historical execution transactions corresponding to the data to be sharded in the database.
[0051] Among them, historical execution transactions are transactions that have used the data to be sharded during the processing;
[0052] Exemplarily, if the data to be sharded is data 1, data 2, data 3, data 4, and data 5, the historical execution transactions corresponding to the sharded data may be transaction 1, transaction 2, and transaction 3.
[0053] Specifically, the specific transaction processing of transaction 1 can be to update data 1 and data 2 based on serviceB, the specific transaction processing of transaction 2 can be to update data 1 and data 3 based on serviceA, and accordingly, the specific transaction processing of transaction 3 can be to process data 3, data 4 and data 5 based on serviceA.
[0054] Step 202: Determine the corresponding key-value string of the key-value information and the key-value substring contained in the key-value string.
[0055] The key-value substring includes a portion of the characters in the key-value string.
[0056] In an embodiment of the present application, the corresponding key value string of the key value information can be a substring of the data key value (key). Specifically, the substring of the key means that a key (key) in the dictionary tree may be a part of another key, or all prefix strings of a key can be regarded as substrings of this key.
[0057] For example, the dictionary tree has the following keys ["apple", "app", "apply", "apt"], where "app" and "ap" are substrings of "apple", and so on. I will not give examples one by one here.
[0058] Step 203: configure the key-value string and the key-value substring in a preset dictionary tree to determine transaction similarity feature information.
[0059] Among them, each node in the preset dictionary tree corresponds to a key-value substring.
[0060] Optionally, step 203 may specifically include: configuring the key-value string and the key-value substring in a preset dictionary tree, and determining the node access information of the key-value information in the preset dictionary tree; according to the node access information, counting each node in the preset dictionary tree through a preset counter in the preset dictionary tree, and generating subtree weight information corresponding to each node in the preset dictionary tree; and determining the transaction similarity feature information based on the subtree weight information.
[0061] In some examples, a trie (also known as a prefix tree or a dictionary lookup tree) is a special tree data structure that is mainly used to efficiently store and find string data. In a trie, each node represents a prefix of a string, and the path from the root node to a leaf node represents a complete string. A node usually contains an array of pointers to its child nodes. The size of the array is generally equal to the size of the character set. Each element in the array corresponds to a character and points to the child node of the next character.
[0062] It should be noted that a counter is maintained on the node of the dictionary tree of the embodiment of the present application, and the counter is incremented by 1 each time the node is visited. The key and all its substrings are added to the dictionary tree, and the records of the visits from the root to the node form a string.
[0063] For this embodiment, step 203 specifically also includes: according to the node access information, each node in the preset dictionary tree is counted by the preset counter in the preset dictionary tree to generate the node weight information of each node in the preset dictionary tree; determining the target subtree corresponding to the target node in the preset dictionary tree; processing the node weight information of each node in the target subtree to obtain the subtree weight information of the target node.
[0064] The target subtree is a subtree corresponding to the preset dictionary tree when the target node is determined as the root node in the preset dictionary tree.
[0065] Optionally, the node weight information is a weight value of a certain node. The node weight information can be determined by formula 1, which is specifically as follows:
[0066] Node weight value = tree depth * node count (Formula 1)
[0067] Furthermore, the subtree weight information is the weight of the subtree of a node determined when a single node is used as the root. The subtree weight information can be determined by Formula 2, which is specifically as follows:
[0068] The weight of a node as the root subtree = the sum of the weights of all nodes on its subtree
[0069] (Formula 2)
[0070] It should be noted that the target node is any node in the preset dictionary tree, and correspondingly, the target subtree is a subtree with the target node in the preset dictionary tree as the root.
[0071] As an optional method, step 203 specifically also includes: selecting a target number of feature nodes from a preset dictionary tree according to subtree weight information from large to small; combining the key-value substrings corresponding to the target number of feature nodes in order to obtain a feature character string corresponding to the historical execution transaction; and determining the first key-value substring in the feature character string as the transaction similarity feature information.
[0072] For example, Figure 4 As shown, if the target number is 3, the transaction set is A, A: begin:
[0073] get("u1000001_pakage_item2356");
[0074] get("u1000001_assert_i5003");
[0075] set("u1000001_assert_i5003", 1);
[0076] set("u1000001_gift_d1",20)end.
[0077] Transaction similarity feature extraction for transaction set A can extract the key value substring "u1000001_"
[0078] If the transaction set is B, B: begin:
[0079] get("u1000002_pakage_item4346");
[0080] get("u1000002_assert_i3009");
[0081] set("u1000002_assert_i5204",1);
[0082] set("u1000002_achie_a1",30)end.
[0083] Then the transaction similarity feature extraction for transaction set B can extract the key value substring as "u1000002_"
[0084] If the transaction set is C, C: begin:
[0085] get("u1000003_pakage_item4346");
[0086] get("u1000003_global_r1567");
[0087] set("u1000003_assert_i5204",1);
[0088] set("u1000003_cach3",30)end.
[0089] Then, the transaction similarity feature extraction for the transaction set C may be to extract the key value substring "u1000003_".
[0090] Step 204: Determine key similarity feature information based on the transaction similarity feature information.
[0091] Optionally, step 204 may specifically include: removing the transaction similarity feature information in the feature character string, and determining the first key value substring in the feature character string after the removal as the key similarity feature information.
[0092] For example, based on step 203, after removing the transaction feature value from the transaction set A, the record string "pakage_item2356", "assert_i5003", "assert_i5003", "gift_d1" is obtained; then the key similarity feature information corresponding to the transaction set A is<pakage_item,assert_i,gift_d1> ; After removing the transaction feature values from transaction set B and obtaining the record strings "pakage_item4346", "assert_i3009", "assert_i5204", "gift_d1", the key similarity feature information corresponding to transaction set B is<pakage_item,assert_i,achie_a1> ; After removing the transaction feature values from transaction set C and obtaining the record strings "pakage_item4346", "global_r1567", "assert_i5204", "cach3", the key similarity feature information corresponding to transaction set C is<pakage_item,global_r,assert_i,cach3> .
[0093] Count the recorded strings (KSP records) collected above, remove the public key TSP features, merge them into the KSP record statistics, calculate and export the profile file. The file contains the collection items after KSP feature extraction. Directly merge KSP records to generate common substrings for the strings. KSP feature extraction uses feature algorithms to obtain a string collection and put it in the profile. In this example, the final profile contains the following data:<pakage_item,assert_i,gift_d1,achie_a1> . Import the generated profile file into the database cluster and update the index metadata information.
[0094] Step 205: Perform data sharding processing on the data to be sharded based on the transaction similarity feature information and / or the key similarity feature information.
[0095] Among them, the data to be sharded that contains the same or similar feature information is allocated to the same shard.
[0096] In the embodiment of the present application, the sharding of transaction similar feature information is prioritized. The same transaction similar feature information is placed in the same or adjacent shards. In this case, the performance loss is better in the scenario where the transaction data lock conflict is high, because this will make the transaction submission execution time window small, the lock context is short, and the possibility of conflict is reduced.
[0097] Prioritize key-similar feature information sharding: put the same key-similar feature information in the same or adjacent shards. In this case, the performance loss is better under the single-point resource bottleneck (such as CPU computing power, disk IO, etc.). Because this division allows the key value data in the transaction to be dispersed to different nodes as much as possible, the execution parallelism is good.
[0098] Correspondingly, such as Figure 5 As shown, you can use KSP classification or TSP classification (Range1) through router 1 (router1), use KSP classification or TSP classification (Range2) through router 2 (router2) to reach shard node1, shard node2, and shard node3, and use KSP classification or TSP classification (Range3) through router 3 (router3) to reach shard node4, shard node5, and shard node6. Use the above calculated and generated profile files, as well as TSP or KSP sharding. Match the data key to the route. Transaction similarity feature information is prioritized: By extracting the Top1 transaction similarity feature information from the transaction key set, the transaction similarity feature information is assigned to a node. Key similarity feature information is prioritized: The key similarity feature information is matched to the transaction key, and the matched key similarity feature information is assigned to a node.
[0099] After the above TSP or KSP node division, the key is divided into sub-divisions for secondary routing and assigned to the next layer of nodes. This allows different routing strategies to be used for different local data. For example, KSP routing is first performed on the data, and TSP routing is performed on the next layer.
[0100] For example, Figure 6 As shown in the figure, data 1 is on Database 1, and data 2 and 3 are on Database 2. After transaction B (Service B) initiates a transaction to update data 1, transaction A (Service A) initiates a transaction to update data 1, which causes a conflict and transaction A cannot proceed. Transaction B initiates a transaction to update data 2, and transaction A initiates a transaction to update data 3. Since update data 2 and update data 3 are in the same shard, transaction A cannot proceed again, causing a conflict and reducing efficiency. Figure 7 As shown in the figure, the routing strategy is optimized to make the data distribution transaction-friendly. Data 1 and 2 are on Database 1, and data 3 is on Database 2. Transaction B can only lock shard 1, and transaction A can update data 3. The probability of concurrent transaction conflicts is reduced (TSP shards are prioritized); for response time optimization involving concurrent execution of more data items, as shown in the figure, Figure 8As shown, data 1 is on Database 1, and data 2, 3, 4, and 5 are on Database 2. When updating data 2, 3, 4, and 5, the lock time of Database 2 is 40ms, and the transaction takes the longest response time of 40ms. Then the routing strategy is optimized (KSP sharding is preferred), as follows: Fig. 9 As shown, data 1 and 2 are on Database 1, data 3 and 4 are on Database 2, and data 5 is on Database 3. When updating data 5, the lock time of Database 3 is 20ms, and the transaction takes the longest response time of 20ms.
[0101] It should be noted that for complex scenarios, multi-parameter routing can be performed, such as Fig.10 As shown, you can first use routing rule 1 (router rule1) uniformly and then use routing rule 2 (router rule2) and routing rule 3 (router rule3) separately; another way is to use routing rule 3 and then use routing rule 4 (router rule3) and routing rule 5 (router rule5) and routing rule 6 (router rule6) separately to achieve the coexistence of two routing strategies (TSP, KSP).
[0102] In an embodiment of the present application, request routing is the process by which a client request reaches the server in a distributed system. It is responsible for intelligently forwarding client requests to the correct server node. Request routing needs to consider: the status information and load capacity of the server node, the type of request, the location of the client, and on which nodes the data involved in the request is located. In large-scale distributed systems, data sharding and request routing are often used in combination. The routing component determines the target node based on the sharding strategy and forwards the request to it. Achieving efficient data routing is critical to the performance and availability of distributed systems. Distributed storage systems such as Redis and Cassandra all use data sharding and request routing strategies.
[0103] The existing non-relational database cluster management method for processing large-scale data is often to hash and range the keys, distribute the data of a certain key value to the nodes of the database service cluster, and then organize the server cluster by multiple nodes to avoid the difficulty of a single node's insufficient performance to support large-scale data. The advantage of this is that the sharding process is simple and fast. However, the disadvantage is also obvious. It will cause the keys of a data item request generated by a business to be randomly distributed on the database nodes. To avoid this situation, the caller needs to cater to the internal sharding rules of the database when naming the data key, which causes inconvenience. When a database transaction is submitted, there is a group of related data distributed on multiple nodes for execution. Before the completion of the transaction, these data will be locked to prevent other transactions from modifying these data at the same time. However, in order to achieve higher concurrent performance, the database may have multiple similar transactions being processed at the same time, causing data lock conflicts, and the utilization rate of the database's processing capacity is reduced. There is also a situation where the operation of each data item in the transaction takes a long time to execute. It is hoped that these data items can be distributed to multiple nodes as much as possible to execute simultaneously to reduce the response time of transaction execution. It can be seen that during the execution of database transactions, the hot spot conflict and concurrency control of data are not ideal. This solution aims to achieve better hot spot performance and concurrent latency of the database by extracting information from the common part and characteristic part of the data key.
[0104] It should be noted that the embodiment of the present application records the transaction execution commands of the sampled transaction execution, analyzes the data key values of the database processing operations of the business program, performs key value analysis with the above database as the object, executes the above database processing information record, and collects and records the keys in similar transactions that match each processing method. The transaction of the key and the global feature extraction of the key are calculated in the above database processing information record. The above-mentioned exported feature information is used as the basis for routing and sharding, so that the keys with the same features are combined into a group of shards. Use program algorithms to build indexes in an automated manner without manually specifying the constraints of the data. Divide and manage the keys of the data to optimize the lock conflicts or parallel execution of transactions. When the business changes, the new profile file can be used to automatically re-migrate the data in the database service without the need for logical considerations by the database caller. Using a better feature algorithm, the data balance can be optimized by the discrimination of the key information after processing. Generally speaking, the algorithm with better discrimination will be more time-consuming. In actual engineering applications, it is necessary to make trade-offs according to the usage scenarios of the database to effectively improve the performance of the database.
[0105] Compared with the current existing technology, this embodiment disassembles the key value information of the historical execution transaction corresponding to the sharded data, determines the transaction similarity feature information and key similarity feature information corresponding to the historical execution transaction, and distributes the sharded data containing the same similarity feature information to the same shard based on the transaction similarity feature information and the key similarity feature information. This can distribute the data of the same transaction in fewer shards. When performing transaction processing, the number of locked shards can be greatly reduced, thereby reducing the number of data required by other transactions being locked. When multiple transactions need to be processed, it can effectively alleviate the conflict between multiple transactions that need to use data in the same shard, thereby improving the processing efficiency of the database and effectively improving the database performance.
[0106] In order to illustrate the specific implementation process of this embodiment, the following specific application examples are given: Fig.11 As shown, but not limited to:
[0107] By programmatically analyzing the similarity of transaction data keys, extracting features to generate profile files, and sharding accordingly. The purpose is to optimize the routing of keys based on the relevance of data key values within transactions. Specifically, the transactions executed by the database are sampled, the data keys of the sampled transactions are analyzed, and the transaction characteristic values TSP are obtained. The transaction data keys are stripped of the transaction characteristic values, and the processed keys are collected to obtain a new data set. The characteristic value analysis of this data set is performed to obtain a set of characteristic value KSP sets, and the KSP sets are imported into the database cluster as KSP sharding metadata. For transaction databases, the routing of data keys is divided into two modes: by TSP splitting of keys: routing is performed by TSP extracted from all data keys of the transaction, or by KSP splitting of keys: routing is performed by matching the data keys and the previously imported KSP set, and the matched KSP is used for routing.
[0108] Furthermore, by counting the executed transactions, the key sets read and written by the same transaction are collected and counted. The key characters are decomposed into Transaction Similar Part, Key Similar Part, and other parts. A dictionary tree is used, and a counter is maintained on the nodes of the dictionary tree. The counter is incremented by one each time the node is visited. The key and all its substrings are added to the dictionary tree, and the records of node access from the root to the node are formed into a string. The node string value is the sequential concatenation of the node characters from the root node of the dictionary tree to the node. All nodes are traversed, and the weight of the subtree with node i as the root is recorded as Wi. The topN nodes with the largest weight Wi are taken, and the string values of these nodes are taken out to form a string set, and the processed key feature information is collected. The KSP records collected above are counted, and after removing the public key TSP features, they are merged into the KSP record statistics, and the profile file is calculated and exported. The file contains the collection items after the KSP feature extraction. The KSPrecord is directly merged to generate a public substring for the string. The string set is obtained using the feature algorithm and placed in the profile. The generated profile file is imported into the database cluster to update the index metadata information.
[0109] Two ways of sharding logic: TSP sharding is preferred. The same TSP is placed in the same or adjacent shards. In this case, the performance loss is better in scenarios with high transaction data lock conflicts, because this will make the transaction submission execution time window smaller, the lock context is shorter, and the possibility of conflict is reduced. KSP sharding is preferred. The same KSP is placed in the same or adjacent shards. In this case, the performance loss is better under single-point resource bottlenecks (such as CPU computing power, disk IO, etc.). Because this division will disperse the key value data in the transaction to different nodes as much as possible, the execution parallelism is good.
[0110] Routing rules: Use the above calculated and generated profile files, as well as TSP or KSP sharding. Match the data key for routing. TSP priority: Extract the Top1 TSP from the transaction key set and assign the TSP to a node. KSP priority: Match the transaction key with KSP, and assign the matched KSP to a node. After the above TSP or KSP node division, perform secondary routing on the key and assign it to the next layer of nodes. This allows different routing strategies for different local data. For example: perform KSP routing on the data first, and then perform TSP routing on the next layer.
[0111] Compared with the current existing technology, this embodiment disassembles the key value information of the historical execution transaction corresponding to the sharded data, determines the transaction similarity feature information and key similarity feature information corresponding to the historical execution transaction, and distributes the sharded data containing the same similarity feature information to the same shard based on the transaction similarity feature information and the key similarity feature information. This can distribute the data of the same transaction in fewer shards. When performing transaction processing, the number of locked shards can be greatly reduced, thereby reducing the number of data required by other transactions being locked. When multiple transactions need to be processed, it can effectively alleviate the conflict between multiple transactions that need to use data in the same shard, thereby improving the processing efficiency of the database and effectively improving the database performance.
[0112] Further, as Figure 1 The specific implementation of the method shown in the embodiment provides a data slicing device, such as Fig.12 As shown, the device includes: an acquisition module 31, a determination module 32, and a slicing module 33.
[0113] An acquisition module 31 is configured to acquire the data to be sharded in the database and the historical execution transactions corresponding to the data to be sharded, wherein the historical execution transactions are transactions that have used the data to be sharded during processing;
[0114] The determination module 32 is configured to disassemble the key value information of the historically executed transaction and determine the similarity feature information corresponding to the historically executed transaction, wherein the similarity feature information includes transaction similarity feature information and key similarity feature information;
[0115] The sharding module 33 is configured to perform data sharding processing on the data to be sharded according to the similar feature information, wherein the data to be sharded containing the same similar feature information is allocated to the same shard.
[0116] In some examples of the present embodiment, the determination module 32 is specifically configured to determine the corresponding key-value string of the key-value information and the key-value substring contained in the key-value string, wherein the key-value substring contains a part of the characters of the key-value string; configure the key-value string and the key-value substring in a preset dictionary tree to determine the transaction similarity feature information, wherein each node in the preset dictionary tree corresponds to a key-value substring; based on the transaction similarity feature information, determine the key similarity feature information; accordingly, the sharding module 33 is specifically configured to perform data sharding processing on the data to be sharded based on the transaction similarity feature information and / or the key similarity feature information.
[0117] In some examples of this embodiment, the determination module 32 is further specifically configured to configure the key-value string and the key-value substring in a preset dictionary tree, determine the node access information of the key-value information in the preset dictionary tree; according to the node access information, count each node in the preset dictionary tree through a preset counter in the preset dictionary tree, and generate subtree weight information corresponding to each node in the preset dictionary tree; and determine the transaction similarity feature information based on the subtree weight information.
[0118] In some examples of this embodiment, the determination module 32 is further specifically configured to count each node in the preset dictionary tree according to the node access information through a preset counter in the preset dictionary tree to generate node weight information of each node in the preset dictionary tree; determine a target subtree corresponding to a target node in the preset dictionary tree, the target subtree being a subtree corresponding to the preset dictionary tree when the target node is determined as a root node in the preset dictionary tree; process the node weight information of each node in the target subtree to obtain the subtree weight information of the target node.
[0119] In some examples of the present embodiment, the determination module 32 is further specifically configured to select a target number of feature nodes from the preset dictionary tree according to the subtree weight information from large to small; combine the key-value substrings corresponding to the target number of feature nodes in sequence to obtain a feature character string corresponding to the historical execution transaction; and determine the first key-value substring in the feature character string as the transaction similarity feature information.
[0120] In some examples of this embodiment, the determination module 32 is further configured to remove the transaction similarity feature information in the feature string, and determine the first key value substring in the feature string after the removal as the key similarity feature information.
[0121] It should be noted that for other corresponding descriptions of the functional units involved in the data slicing device provided in this embodiment, please refer to Figure 1 The corresponding description in will not be repeated here.
[0122] Based on the above Figure 1 and Figure 3 The method shown in the embodiment also provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the above-mentioned Figure 1 and Figure 3 The method shown.
[0123] Based on this understanding, the technical solution of the present application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, USB flash drive, mobile hard disk, etc.), including a number of instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute the methods of various implementation scenarios of the present application.
[0124] Based on the above Figure 1 and Figure 3 The method shown, and Fig.12 In order to achieve the above-mentioned purpose, the embodiment of the present application also provides an electronic device, such as a smart phone, a tablet computer, a drone, a smart robot and other smart terminals, which includes a storage medium and a processor; the storage medium is used to store a computer program; the processor is used to execute the computer program to achieve the above-mentioned Figure 1 and Figure 3 The method shown.
[0125] Optionally, the above-mentioned physical device may also include a user interface, a network interface, a camera, a radio frequency (RF) circuit, a sensor, an audio circuit, a WI-FI module, etc. The user interface may include a display, an input unit such as a keyboard, etc., and the optional user interface may also include a USB interface, a card reader interface, etc. The network interface may optionally include a standard wired interface, a wireless interface (such as a WI-FI interface), etc.
[0126] Those skilled in the art will appreciate that the above-mentioned physical device structure provided in this embodiment does not constitute a limitation on the physical device, and may include more or fewer components, or a combination of certain components, or different arrangements of components.
[0127] The storage medium may also include an operating system and a network communication module. The operating system is a program that manages the hardware and software resources of the above-mentioned physical device, and supports the operation of the information processing program and other software and / or programs. The network communication module is used to realize the communication between the components inside the storage medium, and the communication with other hardware and software in the information processing physical device.
[0128] Based on the above Figures 1 to 3 The present application also provides a computer program product, which includes a computer program. When the computer program is executed by a processor, the above-mentioned Figures 1 to 3 The method shown, the method implemented when the computer program is executed by the processor can refer to the various embodiments of the present application, and will not be described in detail here.
[0129] Through the description of the above implementation methods, those skilled in the art can clearly understand that the present application can be implemented by means of software plus a necessary general hardware platform, or by hardware. By applying the solution of this embodiment, compared with the current prior art, this embodiment disassembles and processes the key value information of the historical execution transaction corresponding to the data to be sharded, determines the transaction similarity feature information and key similarity feature information corresponding to the historical execution transaction, and distributes the data to be sharded containing the same similarity feature information to the same shard based on the transaction similarity feature information and key similarity feature information. When multiple transactions need to be processed, it can alleviate the conflict between multiple transactions that need to use data in the same shard, improve the processing efficiency of the database, and effectively improve the performance of the database.
[0130] It should be noted that, in this article, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.
[0131] The above is only a specific implementation of the present application, so that those skilled in the art can understand or implement the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments described herein, but will conform to the widest scope consistent with the principles and novel features applied for herein.
Claims
1. A data sharding method, It is characterized in that include: Acquire the data to be sharded in the database and the historical executed transactions corresponding to the data to be sharded, wherein the historical executed transactions are transactions that have used the data to be sharded during the processing; Decomposing the key value information of the historically executed transaction to determine similarity feature information corresponding to the historically executed transaction, wherein the similarity feature information includes transaction similarity feature information and key similarity feature information; The data to be sharded is processed according to the similar feature information, wherein the data to be sharded containing the same similar feature information is allocated to the same shard.
2. The method according to claim 1, It is characterized in that The step of disassembling the key value information of the historically executed transaction to determine the similar feature information corresponding to the historically executed transaction includes: Determine a key-value string corresponding to the key-value information and a key-value substring contained in the key-value string, wherein the key-value substring contains a portion of characters of the key-value string; The key-value string and the key-value substring are configured in a preset dictionary tree to determine the transaction similarity feature information, wherein each node in the preset dictionary tree corresponds to a key-value substring; Based on the transaction similarity feature information, determining the key similarity feature information; The step of performing data sharding processing on the data to be sharded based on the similar feature information includes: The data to be sharded is processed based on the transaction similarity feature information and / or the key similarity feature information.
3. The method according to claim 2, It is characterized in that The configuring the key-value string and the key-value substring in a preset dictionary tree to determine the transaction similarity feature information includes: Configure the key-value string and the key-value substring in a preset dictionary tree, and determine node access information of the key-value information in the preset dictionary tree; According to the node access information, each node in the preset dictionary tree is counted by a preset counter in the preset dictionary tree to generate subtree weight information corresponding to each node in the preset dictionary tree; The transaction similarity feature information is determined according to the subtree weight information.
4. The method according to claim 3, It is characterized in that The method of counting each node in the preset dictionary tree by a preset counter in the preset dictionary tree according to the node access information to generate subtree weight information corresponding to each node in the preset dictionary tree includes: According to the node access information, each node in the preset dictionary tree is counted by a preset counter in the preset dictionary tree to generate node weight information of each node in the preset dictionary tree; Determine a target subtree corresponding to a target node in the preset dictionary tree, wherein the target subtree is a subtree corresponding to the preset dictionary tree when the target node is determined as a root node in the preset dictionary tree; The node weight information of each node in the target subtree is processed to obtain the subtree weight information of the target node.
5. The method according to claim 3, It is characterized in that The determining the transaction similarity feature information according to the subtree weight information includes: According to the subtree weight information from large to small, select a target number of feature nodes from the preset dictionary tree; The key value substrings corresponding to the target number of feature nodes are combined in order to obtain a feature character string corresponding to the historical execution transaction; The first key value substring in the feature character string is determined as the transaction similarity feature information.
6. The method according to claim 5, It is characterized in that The determining the key similarity feature information based on the transaction similarity feature information includes: The transaction similarity feature information in the feature character string is removed, and the first key value substring in the feature character string after the removal process is determined as the key similarity feature information.
7. A data slicing device, It is characterized in that include: An acquisition module is configured to acquire the data to be sharded in the database and historical execution transactions corresponding to the data to be sharded, wherein the historical execution transactions are transactions that have used the data to be sharded during processing; A determination module is configured to disassemble the key value information of the historically executed transaction to determine similar feature information corresponding to the historically executed transaction, wherein the similar feature information includes transaction similar feature information and key similar feature information; The sharding module is configured to perform data sharding processing on the data to be sharded according to the similar feature information, wherein the data to be sharded containing the same similar feature information is allocated to the same shard.
8. A computer-readable storage medium having a computer program stored thereon, It is characterized in that When the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.
9. An electronic device comprising a storage medium, a processor, and a computer program stored on the storage medium and executable on the processor, It is characterized in that When the processor executes the computer program, the method according to any one of claims 1 to 6 is implemented.
10. A computer program product, the computer program product comprising a computer program, It is characterized in that When the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.
