Data processing method and device for distributed database, equipment and storage medium

Abstract

The invention relates to a data processing method and device for a distributed database, equipment and a storage medium. The method comprises the steps of obtaining a database execution instruction sent by a client; in response to the database execution instruction, determining a target fragment function of the distributed database; determining a target data node by using the target fragmentation function; and issuing the database execution instruction to a target data node, so that the target data node executes a corresponding database operation and feeds back an execution result to the client. The calculation nodes do not need to be recompiled in the upgrading process, and database service does not need to be restarted; realizing self-definition of a fragmentation rule by self-defining a fragmentation rule function; according to the method, the distributed database is ensured to continuously serve for the outside without interruption, the simplicity and the flexibility of fragmentation rule replacement of the distributed database are improved, various self-defined requirements of a database user are met, and the distributed database is enabled to meet more business scenes.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a data processing method, device, equipment and storage medium for a distributed database. Background Art

[0002] Data sharding technology is a technology that reduces the data set on a single data node through data sharding, thereby reducing the pressure on database queries, speeding up queries, and improving performance. In a distributed database, data sharding rules are usually set during the data table construction phase to ensure that data can be evenly distributed to each sharding node. Sharding rules are one of the core functions of a distributed database. Currently, it is common for database developers to design new sharding rules and deploy these rules to the entire distributed database system through system upgrades.

[0003] However, upgrading a running distributed database system is a huge task, and the cluster needs to suspend service during the upgrade process, which not only affects the availability of the system but also may lead to the risk of data loss.

[0004] Therefore, there is an urgent need to develop a data processing method, device and storage medium for a distributed database to solve one or more of the above-mentioned problems. Summary of the invention

[0005] In view of this, in order to solve the above technical problems or part of the technical problems, the embodiments of the present invention provide a data processing method, device, equipment and storage medium for a distributed database.

[0006] In a first aspect, the present application provides a distributed database data processing method, the method comprising:

[0007] Get the database execution instructions sent by the client;

[0008] In response to the database execution instruction, determining a target sharding function of the distributed database;

[0009] Determine the target data node using the target sharding function;

[0010] The database execution instruction is sent to the target data node, so that the target data node executes the corresponding database operation and feeds back the execution result to the client.

[0011] In a possible implementation, determining a target sharding function of a distributed database in response to the database executing the instruction includes:

[0012] In response to the database execution instruction, receiving a database operation statement sent by a client;

[0013] Parsing the database operation statement through a syntax parser to obtain a first operation statement and a second operation statement corresponding to the database operation statement;

[0014] According to the second operation statement, a target sharding function of the distributed database is determined.

[0015] In a possible implementation, the determining the target data node by using the target sharding function includes:

[0016] According to the first operation statement, determine a target shard key of the distributed database;

[0017] The target data node corresponding to the target sharding key is determined using the target sharding function.

[0018] In a possible implementation, determining a target shard key of a distributed database according to the first operation statement includes:

[0019] Get the initial shard key of the distributed database;

[0020] Determine whether the first operation statement is an empty statement;

[0021] When the first operation statement is an empty statement, the initial sharding key is used as the target sharding key of the distributed database;

[0022] When the first operation statement is a non-empty statement, the initial sharding key is filled to obtain a target sharding key.

[0023] In a possible implementation, determining a target sharding function of a database of the client according to the second operation statement includes:

[0024] Get the initial sharding function of the distributed database;

[0025] Determine whether the second operation statement is an empty statement;

[0026] When the second operation statement is an empty statement, the initial sharding function is used as the target sharding function of the distributed database;

[0027] When the second operation statement is a non-empty statement, a custom sharding function corresponding to the dynamic compilation library is obtained, and the custom sharding function is used as a target sharding function of the distributed database.

[0028] In a possible implementation, the filling process on the initial shard key to obtain the target shard key includes:

[0029] Filling the initial shard key with a filling character to obtain an initial filled shard key;

[0030] Determine whether the character string of the initial filled shard key meets the preset shard key rule;

[0031] In the case of meeting the preset shard key rule, the initial filled shard key is used as the target shard key;

[0032] In the case of not meeting the preset shard key rules, the initially filled shard key is further filled with invalid characters so that the character string of the filled shard key meets the preset shard key rules.

[0033] In a possible implementation manner, sending the database execution instruction to the target data node includes:

[0034] Generate a syntax execution tree based on the database operation statements sent by the client;

[0035] According to the syntax execution tree, the database execution instruction is sent to the corresponding target data node.

[0036] In a first aspect, the present application provides a data processing device for a distributed database, the device comprising:

[0037] The acquisition module is used to obtain the database execution instruction sent by the client;

[0038] A first determination module, configured to determine a target sharding function of a distributed database in response to the database execution instruction;

[0039] A second determination module is used to determine a target data node using the target sharding function;

[0040] The sending module is used to send the database execution instruction to the target data node, so that the target data node executes the corresponding database operation and feeds back the execution result to the client.

[0041] In a third aspect, the present application provides an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the steps of the distributed database data processing method described in any one of the embodiments of the first aspect are implemented.

[0042] In a fourth aspect, the present application further provides a computer storage medium on which a computer program is stored, and when the computer program is executed by a processor, the steps of the distributed database data processing method described in any embodiment of the first aspect are implemented.

[0043] The above technical solution provided by the embodiment of the present application has the following advantages over the prior art: the data processing method of the distributed database provided by the embodiment of the present application responds to the database execution instruction to determine the target sharding function, and there is no need to recompile the computing node or restart the database service during the upgrade process; the sharding rule is customized through the customized sharding rule function; it ensures that the distributed database continues to provide external services without interruption, improves the simplicity and flexibility of replacing the sharding rules of the distributed database, meets the various customized needs of the database users, and enables the distributed database to meet more business scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0045] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0046] One or more embodiments are exemplarily described by pictures in the corresponding drawings, and these exemplified descriptions do not constitute limitations on the embodiments. Elements with the same reference numerals in the drawings represent similar elements, and unless otherwise stated, the figures in the drawings do not constitute proportional limitations.

[0047] Figure 1 A schematic diagram of a flow chart of a data processing method of a distributed database provided in an embodiment of the present application;

[0048] Figure 2 A schematic diagram of the process of another distributed database data processing method provided in an embodiment of the present application;

[0049] Figure 3 A schematic diagram of a flow chart of a data processing method of a distributed database provided in an embodiment of the present application;

[0050] Figure 4 A flow chart of a distributed database data processing method provided in an embodiment of the present application;

[0051] Figure 5 A schematic diagram of the structure of a data processing device for a distributed database provided in an embodiment of the present application;

[0052] Figure 6 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0053] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0054] The disclosure below provides many different embodiments or examples to implement different structures of the present invention. In order to simplify the disclosure of the present invention, the parts and settings of specific examples are described below. Of course, they are only examples, and the purpose is not to limit the present invention. In addition, the present invention can repeat reference numbers and / or letters in different examples. This repetition is for the purpose of simplification and clarity, and does not itself indicate the relationship between the various embodiments and / or settings discussed.

[0055] In order to solve the technical problem that in the prior art, the distribution characteristics of user data may not completely match several common sharding rules, it is necessary to design custom sharding rules according to actual conditions, and then apply them to the distributed database system, and then apply them to the entire distributed database system by upgrading the system. During the upgrade process, the distributed database system has a large workload, and the cluster to be upgraded needs to interrupt the service to affect the availability of the system, which may cause the risk of data loss. The present application provides a data processing method, device and storage medium for a distributed database, which respond to database execution instructions to determine the target sharding function. During the upgrade process, there is no need to recompile the computing nodes or restart the database service; through the custom sharding rule function, the sharding rules are customized, and the custom sharding rule plug-in is uploaded from the carrier to the operation and maintenance platform, and can be dynamically loaded after the operation and maintenance personnel execute the loading statement; it is ensured that the distributed database continues to serve the outside world without interruption, improves the simplicity and flexibility of replacing the sharding rules of the distributed database, meets the various custom needs of database users, and enables the distributed database to meet more business scenarios.

[0056] Figure 1 A flow chart of a distributed database data processing method provided in an embodiment of the present application is shown as follows: Figure 1 As shown, the method specifically includes:

[0057] S101, obtaining a database execution instruction sent by a client;

[0058] In this embodiment, the client sends DDL (Data Definition Language) statements to the computing node CN. Through these DDL statements, the database execution instructions issued by the client can be obtained.

[0059] Specifically, DDL statements are mainly used to define or modify the structure of a database, including creating, deleting or modifying database objects such as tables, indexes, and views. When a client sends DDL statements to a computing node CN, the computing node CN will parse these statements and generate corresponding database execution instructions to modify or define the structure of the database, ensuring that the structure of the database can be adjusted as needed to meet different data storage and query requirements.

[0060] S102, in response to the database execution instruction, determining a target sharding function of the distributed database;

[0061] The role of the sharding function is to evenly distribute data to different shards to improve the performance and scalability of the system; by selecting an appropriate target sharding function, efficient management and query of data in a distributed database can be ensured to meet user needs and optimize overall system performance.

[0062] In this embodiment, the computing node determines the target sharding function of the target node of the distributed database according to the received database execution instruction.

[0063] Prior to this, the operation and maintenance personnel uploaded the custom sharding rule plug-in to the operation and maintenance management system; the operation and maintenance management system distributed the plug-in to the program directory of each extreme node: so that the computing node can determine the target sharding function. After the computing node determines the target sharding function, it will further determine the target node according to the sharding rules in the plug-in; the operation and maintenance management system not only simplifies the management of sharding rules, but also ensures that each extreme node can work together efficiently to complete complex computing tasks.

[0064] S103, using the target sharding function to determine the target data node;

[0065] The role of the sharding function is to divide the entire data set into multiple small, manageable fragments, each fragment corresponds to a specific data node.

[0066] In this embodiment, the specific location of the target data node is quickly located through the calculation result of the sharding function, thereby achieving efficient data management and retrieval. The use of the sharding function not only improves the efficiency of data processing, but also ensures the accuracy and reliability of the data.

[0067] S104: Send the database execution instruction to the target data node, so that the target data node executes the corresponding database operation and feeds back the execution result to the client.

[0068] In this embodiment, the database execution instruction is sent to the target data node so that the target data node can perform the corresponding database operation. After the operation is completed, the target data node feeds back the operation result to the client to ensure that the client can receive the final result of the operation and ensure that the client will not perform other operations before receiving the result, thereby avoiding potential data inconsistency problems; it helps to improve the reliability and stability of the system.

[0069] For example, the developer will compile the customized sharding rules into a dynamic link library named libxxx.so, where xxx represents the plug-in name in the HINT information of the table creation SQL statement, i.e. plugin_name; then, this dynamic library is stored on media such as CDs and USB flash drives and used as a database plug-in; after receiving the plug-in, the operation and maintenance personnel upload it to the operation and maintenance management platform; the platform will then distribute the plug-in to the lib directory of each distributed database computing node, modify the file owner and grant corresponding permissions.

[0070] In the distributed database system, the original fixed sharding rule function is changed to an interface form so that the custom sharding rule function in the plug-in dynamic library can be dynamically called. When creating a table, HINT information is added to the SQL DDL statement to clearly specify the plug-in name, sharding rule name, and preprocessing settings for the sharding key. The complete DDL statement example is as follows:

[0071] CREATE TABLE t1(

[0072] a INT NOT NULL KEY,

[0073] name varchar(20) )

[0075] DISTRIBUTED BY policy(name)(g1, g2, g3),

[0076] ALGORITHM='plugin_name / algorithm_name',

[0077] PARAM = 'algorithm_param';

[0078] Where plugin_name is the plugin name and algorithm_name is the rule function name. The PARAM parameter is related to the specific custom sharding rule function, which can be used to fine-tune the specific rule algorithm or pre-process the original sharding value. If there is no PARAM parameter, the original sharding key value and default parameter value are used for calculation.

[0079] The data processing method of the distributed database provided in the present embodiment determines a customized target sharding function in response to the database execution instruction after obtaining the database execution instruction of the client, and then determines the target data node by using the target sharding function. After determining the target data node, the database execution instruction is directly sent to the target data node, so that the target data node executes the database operation corresponding to the database execution instruction, and feeds back the operation execution result to the client after execution; during the upgrade process, there is no need to recompile the computing node or restart the database service; through the customized sharding rule function, the sharding rules are flexibly set, so as to realize the customization of the sharding rules, ensuring that the distributed database can continue to provide services to the outside without any interruption when upgrading or replacing the sharding rules; not only the simplicity and flexibility of the replacement of the distributed database sharding rules are improved, but also various customized requirements of the database users can be met, so that the distributed database can adapt to more business scenarios and provide users with more efficient and flexible data processing capabilities.

[0080] In an optional solution of the embodiment of the present invention, determining the target sharding function of the distributed database in response to the database execution instruction includes:

[0081] In response to the database execution instruction, receiving a database operation statement sent by a client;

[0082] Parsing the database operation statement through a syntax parser to obtain a first operation statement and a second operation statement corresponding to the database operation statement;

[0083] In this embodiment, after receiving the database execution instruction sent by the client, the database operation statement sent by the client is received, and the corresponding database instruction is executed. Then, an efficient syntax parser is used to perform detailed parsing processing on these operation statements, and the complex database operation statement is decomposed into two independent parts through parsing: a first operation statement and a second operation statement; these two operation statements represent different aspects of the original operation statement, so that the subsequent processing and execution become more efficient and orderly.

[0084] Exemplarily, an SQL statement sent by a client is received and passed to a syntax parser, which parses the SQL statement and reads a PARAM statement and an ALGORITHM statement.

[0085] According to the second operation statement, a target sharding function of the distributed database is determined.

[0086] In this embodiment, the target sharding function of the distributed database is determined by reading the second operation statement; specifically, the content of the second operation statement is parsed to extract relevant instructions and parameters, and then the shard to which the data in the distributed database should be allocated is determined based on this information to ensure that the data can be correctly distributed to the location specified by the target sharding function; in this way, data storage and access in the distributed database can be effectively managed and optimized, thereby improving the overall performance and efficiency of the system.

[0087] In an optional solution of the embodiment of the present invention, the determining the target data node by using the target sharding function includes:

[0088] According to the first operation statement, determine a target shard key of the distributed database;

[0089] The target data node corresponding to the target sharding key is determined using the target sharding function.

[0090] The distributed database will locate the data on the corresponding sharding node according to the calculation results of the sharding function to ensure that the data can be correctly distributed according to the established sharding rules. The processing of sharding key values ​​includes preprocessing of key values, such as hashing and range partitioning, to adapt to different sharding strategies. In this way, the system can achieve uniform distribution of data, avoid data skew problems, and thus improve the performance of query and update operations.

[0091] In this embodiment, a detailed analysis is performed on the structure and data distribution of the database according to the first operation statement to determine the target sharding key of the distributed database, thereby ensuring that the data can be evenly distributed on each shard, thereby improving the query efficiency and the overall performance of the system. After the target sharding key is determined, the specific target data node corresponding to the target sharding key can be determined using the defined target sharding function.

[0092] Specifically, the role of the target sharding function is to calculate which data node the data should be stored on based on the value of the sharding key. The calculation process of the target sharding function usually involves some algorithms, such as hash algorithms or range algorithms, to ensure that the data can be evenly distributed to each node according to established rules. By determining the target data node corresponding to the target sharding key through the target sharding function, it can be ensured that each data node can efficiently process the data it is responsible for, thereby improving the performance and scalability of the entire distributed database system.

[0093] Figure 2 A flow chart of another distributed database data processing method provided in an embodiment of the present application is shown as follows: Figure 2 As shown, the method specifically includes:

[0094] S201, obtaining an initial sharding key of a distributed database;

[0095] In this embodiment, the initial sharding key is selected by analyzing the usage pattern and data access frequency of the database, and a field that can represent the data distribution characteristics is selected as the initial sharding key. The initial sharding key is crucial for subsequent data migration and load balancing. The initial sharding key comprehensively considers the data read-write ratio, the distribution of hot data, and the possible future data growth trend. After determining the initial sharding key, the data will be initially distributed to different data nodes according to the preset sharding strategy to achieve dynamic sharding and load balancing.

[0096] S202, determining whether the first operation statement is an empty statement;

[0097] In this embodiment, it is determined whether the first operation statement contains actual operation content or execution instructions. Specifically, it is analyzed whether the first operation statement involves a specific execution action, or whether it contains clear instructions to guide the execution of a certain operation.

[0098] S203: When the first operation statement is an empty statement, use the initial sharding key as the target sharding key of the distributed database;

[0099] In this embodiment, if it is determined through judgment that the first operation statement is actually an empty statement, that is, there is no specific operation instruction or content, then in this case, we use the shard key set initially as the target shard key of the distributed database system, and the originally set shard key will be directly applied to the distributed database to ensure the reasonable distribution and efficient management of data.

[0100] S204: When the first operation statement is a non-empty statement, fill the initial shard key to obtain a target shard key.

[0101] In this embodiment, when the first operation statement is not empty, a series of filling processes are performed on the initial shard key to obtain the final target shard key.

[0102] Specifically, the filling process includes but is not limited to adding some additional information or performing some specific conversion operations on the basis of the initial shard key, so as to ensure that the target shard key finally generated can meet specific requirements or standards, and ensure that in the subsequent data processing or storage process, the target shard key can play its due role and improve the efficiency and accuracy of the system.

[0103] Exemplarily, when performing data processing, the PARAM statement is read and parsed. If the PARAM statement is empty, that is, no specific parameters are provided, the original state of the shard key is maintained, and no additional processing is performed on the initial shard key; if specific parameters are provided in the PARAM statement, specific operations are performed according to the parameters of the PARAM statement.

[0104] Specifically, perform the lpad operation on the name string in the shard key, pad the string with specific characters on the left, and ensure that the name string of the shard key reaches the required length. If the length of the name string of the shard key does not reach 16 bits, continue to use the character 0 to pad the left until its length reaches 16 bits, thereby ensuring the consistency and standardization of the shard key, which is helpful for subsequent data processing and query optimization.

[0105] The data processing method of the distributed database provided by this embodiment determines the target sharding key by reading the second operation statement. After determining the target sharding key, the corresponding conversion operation is performed according to the parameters specified in the second operation statement. For example, if the second operation statement indicates that the sharding key needs to be converted into a numerical value, the character string in the sharding key is automatically converted into a numerical type to facilitate subsequent numerical calculations and comparisons. It can flexibly adapt to different data processing requirements and improve the efficiency and accuracy of data processing.

[0106] Figure 3 A flow chart of another distributed database data processing method provided in an embodiment of the present application is shown as follows: Figure 3 As shown, the method specifically includes:

[0107] S301, obtaining an initial sharding function of a distributed database;

[0108] In a distributed database, the initial sharding function is a key component in the distributed database system. It determines how data is distributed to each shard. The initial sharding function usually takes into account the characteristics and access patterns of the data to ensure that the distribution of data in each shard is as balanced as possible. By reasonably designing the initial sharding function, the uniform distribution of data can be ensured and the data skew problem can be avoided, thereby improving query efficiency and the overall performance of the system.

[0109] In this embodiment, an initial sharding function of a distributed database is obtained so that data can be effectively distributed on multiple nodes, thereby improving the scalability and performance of the system.

[0110] S302, determining whether the second operation statement is an empty statement;

[0111] In this embodiment, it is determined whether the second operation statement contains actual operation content or execution instructions. Specifically, it is analyzed whether the second operation statement involves a specific execution action, or whether it contains clear instructions to guide the execution of a certain operation.

[0112] S303: When the second operation statement is an empty statement, use the initial sharding function as the target sharding function of the distributed database;

[0113] In this embodiment, if it is determined that the second operation statement is an empty statement and has no actual operation content, in this case, the default initial sharding function built into the database system is directly used to perform the calculation work of the sharding node; there is no need to write or call any additional custom sharding functions, but to completely rely on the default mechanism provided by the database to complete the calculation of the sharding node. This simplifies the operation process, reduces the workload of development and maintenance, and also ensures that the calculation results of the sharding node meet the design specifications and performance requirements of the database system.

[0114] S304: When the second operation statement is a non-empty statement, obtain a custom sharding function corresponding to a dynamic compilation library, and use the custom sharding function as a target sharding function of the distributed database.

[0115] In this embodiment, when the second operation statement is not empty, a custom sharding function associated with the dynamic compilation library is obtained. The custom sharding function is specially designed to meet specific needs and can perform flexible sharding operations according to the specific needs of users.

[0116] Furthermore, after obtaining this custom sharding function, the custom sharding function is set as the target sharding function in the distributed database system, so that the distributed database uses the specified custom sharding function to perform operations when sharding data, thereby ensuring that the data can be efficiently and reasonably distributed in a preset manner, so as to achieve better control of data distribution and improve database performance and scalability.

[0117] Specifically, the custom sharding function obtained through the dynamic compilation library can be optimized for different application scenarios and data access patterns to achieve more refined data distribution control. For example, for data with high access hotspots, a specific sharding strategy can be designed to disperse the hotspots and reduce the pressure on a single node, thereby improving the load balancing capability of the overall system. In addition, the custom sharding function can also dynamically adjust the sharding strategy based on the access frequency and update frequency of the data to adapt to changes in data access patterns and further optimize performance.

[0118] Exemplarily, read the ALGORITHM statement. If it is empty, directly use the default sharding rule calculation built into the database. Otherwise, open the previously compiled dynamic library libplugin_name.so, read the algorithm_name custom sharding rule function therein, and use the custom sharding function to calculate the sharding node ID.

[0119] The data processing method of the distributed database provided in this embodiment reads the custom sharding rule function in the second operation statement, thereby directly adding a dynamic library and a custom sharding rule function without recompiling the computing node and restarting the database service; for example, after executing the DDL (Data Definition Language) statement, it is only necessary to add the corresponding HINT information therein to easily realize the customization of the sharding rule; not only does it simplify the operation process, but it also improves the flexibility and scalability of the system, so that the database administrator can more conveniently adjust the sharding strategy according to actual needs, thereby optimizing the performance and resource allocation of the database.

[0120] In an optional solution of the embodiment of the present invention, the initial shard key is padded with a padding character to obtain an initial padded shard key;

[0121] Determine whether the character string of the initial filled shard key meets the preset shard key rule;

[0122] In this embodiment, by judging whether the character string initially populated with the shard key meets preset specific conditions, the preset rules may include requirements on the length, format, character type, etc. of the character string; by analyzing and verifying to ensure that the character string fully complies with these rules, the validity and correctness of the shard key are ensured.

[0123] In the case of meeting the preset shard key rule, the initial filled shard key is used as the target shard key;

[0124] In this embodiment, under the premise of satisfying the pre-set shard key rules, the initial filled shard key is used as the target shard key.

[0125] In the case of not meeting the preset shard key rules, the initially filled shard key is further filled with invalid characters so that the character string of the filled shard key meets the preset shard key rules.

[0126] In this embodiment, when the preset shard key rules are not met, the initial filled shard key is further filled by adding invalid characters to ensure that the shard key string after the filling process can meet the preset shard key rules.

[0127] Specifically, you can insert the character 0 at the end of the shard key or at an appropriate position. The character 0 does not change the actual meaning of the shard key, but it can make the format of the shard key meet specific requirements.

[0128] Exemplarily, the length of the shard key string is checked to determine whether the number of bits of the shard key string has reached or exceeded 16 bits; if the length of the shard key string does not reach 16 bits, the character 0 is used to perform a left padding operation until the total length of the shard key string reaches 16 bits, ensuring that the shard key string can meet specific format requirements in subsequent operations, thereby ensuring the normal operation of the entire system and data consistency.

[0129] In addition, program logic can be used to check and verify whether the sharding key complies with the database's sharding strategy to ensure its effectiveness and stability in actual applications.

[0130] By analyzing and processing the string that initially fills the shard key, a specific algorithm can be set to adjust the length of the string, or convert it according to the character type to meet the requirements of the shard key rules, thereby achieving precise control of the shard key and ensuring that the generation of the shard key complies with the preset rules and can adapt to the ever-changing data processing needs, thereby improving the performance and reliability of the entire distributed database system.

[0131] In an optional solution of the embodiment of the present invention, sending the database execution instruction to the target data node includes:

[0132] Generate a syntax execution tree based on the database operation statements sent by the client;

[0133] According to the syntax execution tree, the database execution instruction is sent to the corresponding target data node.

[0134] In this embodiment, according to the database operation statement sent by the client,

[0135] The parsing and syntax analysis of the statement generates a syntax execution tree, and then, based on the generated syntax execution tree, the specific database execution instructions are sent to the corresponding target data node.

[0136] Specifically, the parsed operation instructions are converted into actual database operation instructions and sent to the target data node through the network or other communication methods. The target data node receives the database operation instructions and performs corresponding database operations, such as query, insert, update or delete, thereby completing the client's request.

[0137] Figure 4 A flow chart of a distributed database data processing method provided in an embodiment of the present application, such as Figure 4As shown in the figure, the implementation process of the data processing method of the plug-in custom sharding rule is as follows:

[0138] Step 1: The operation and maintenance personnel upload the customized sharding rule plug-in to the operation and maintenance management system; wherein the sharding rule plug-in meets the system requirements and is compatible with the existing architecture.

[0139] Step 2: After uploading, the system will automatically distribute the plug-in to the program directory of each computing node.

[0140] Step 3: When the client needs to perform data sharding operations, it sends a DDL statement to the computing node CN. The DDL statement contains the instructions for creating a table and the definition of sharding rules.

[0141] Step 4: After receiving the SQL statement, the computing node CN passes it to the syntax parser for parsing. The parser checks whether the syntax of the statement is correct and extracts relevant information about the sharding rules.

[0142] Step 5: During the parsing process, the system reads the PARAM statement; if the PARAM statement is empty, it means that no specific sharding parameters are specified, and step 7 will be executed; if the PARAM statement is not empty, step 6 will be executed.

[0143] Step 6: When performing the lpad operation, the system will use specific characters to left-fill the shard key name string; if the length of the shard key does not reach 16 bits, the system will use the character 0 to left-fill until the length reaches 16 bits.

[0144] Step 7: If the PARAM statement is empty, the shard key will remain in its original state without any processing, ensuring that the original data of the shard key will not be accidentally modified, thereby affecting subsequent data operations.

[0145] Step 8: After completing the processing of the sharding key, the system will read the ALGORITHM statement; if the ALGORITHM statement is empty, it means that no specific sharding algorithm is specified, and execute step 10; if the ALGORITHM statement is not empty, execute step 9.

[0146] Step 9: When a specific sharding algorithm is specified, the system will open the previously compiled dynamic library libplugin_name.so; then, the system will read the algorithm_name custom sharding rule function and use this custom sharding function to calculate the ID of the sharding node.

[0147] Step 10: If no specific sharding algorithm is specified, the default sharding rule built into the database is used to calculate the ID of the sharding node. The default rule is pre-set and can meet the sharding requirements in most cases.

[0148] Step 11: After obtaining the shard node ID, the computing node CN generates a syntax execution tree.

[0149] Step 12: Based on the generated syntax execution tree, the computing node CN will send the corresponding statements to each data node DN for execution.

[0150] Step 13: After the data operation is completed, the computing node CN returns the execution response to the client.

[0151] Figure 5 A structural diagram of a data processing device for a distributed database provided in an embodiment of the present application is shown in FIG. Figure 5 As shown, the method specifically includes:

[0152] The acquisition module 501 is used to acquire the database execution instruction sent by the client;

[0153] A first determination module 502, configured to determine a target sharding function of a distributed database in response to the database execution instruction;

[0154] A second determination module 503 is used to determine a target data node using the target sharding function;

[0155] The sending module 504 is used to send the database execution instruction to the target data node, so that the target data node executes the corresponding database operation and feeds back the execution result to the client.

[0156] In one possible implementation, the first determination module 502 is further used to receive a database operation statement sent by a client in response to the database execution instruction; parse the database operation statement through a syntax parser to obtain a first operation statement and a second operation statement corresponding to the database operation statement; and determine a target sharding function of the distributed database according to the second operation statement.

[0157] In a possible implementation, the first determination module 502 is further configured to determine a target sharding key of the distributed database according to the first operation statement; and determine a target data node corresponding to the target sharding key using the target sharding function.

[0158] In a possible implementation, the first determination module 502 is further used to obtain an initial sharding key of the distributed database; determine whether the first operation statement is an empty statement; if the first operation statement is an empty statement, use the initial sharding key as a target sharding key of the distributed database; if the first operation statement is a non-empty statement, fill the initial sharding key to obtain a target sharding key.

[0159] In one possible implementation, the first determination module 502 is further used to obtain an initial sharding function of the distributed database; determine whether the second operation statement is an empty statement; if the second operation statement is an empty statement, use the initial sharding function as the target sharding function of the distributed database; if the second operation statement is a non-empty statement, obtain a custom sharding function corresponding to a dynamic compilation library, and use the custom sharding function as the target sharding function of the distributed database.

[0160] In a possible implementation, the first determination module 502 is further configured to fill the initial shard key with fill characters to obtain an initial filled shard key; determine whether the character string of the initial filled shard key complies with a preset shard key rule; if it complies with the preset shard key rule, use the initial filled shard key as a target shard key; if it does not comply with the preset shard key rule, continue to fill the initial filled shard key with invalid characters so that the character string of the filled shard key complies with the preset shard key rule.

[0161] In a possible implementation, the sending module 504 is further configured to generate a syntax execution tree according to the database operation statement sent by the client; and send the database execution instruction to the corresponding target data node according to the syntax execution tree.

[0162] The intelligent recommendation system provided in this embodiment can be as follows Figure 5 The data processing device of the distributed database shown in can execute the following Figure 1-4 All steps of data processing in the distributed database are realized Figure 1-4 For details, please refer to the technical effect of data processing in the distributed database shown in Figure 1-4 For the sake of brevity, the relevant description is not repeated here.

[0163] The system embodiment described above is merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0164] Figure 6 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application is shown in FIG. Figure 6 As shown, an embodiment of the present application provides an electronic device, including a processor 601, a communication interface 602, a memory 603 and a communication bus 604, wherein the processor 601, the communication interface 602, and the memory 603 communicate with each other through the communication bus 604; the memory 603 is used to store computer programs; the processor 601 is used to execute the program stored in the memory 603, and implement the steps of the data processing method of the distributed database provided by any of the above method embodiments:

[0165] Obtain a database execution instruction sent by a client; determine a target sharding function of a distributed database in response to the database execution instruction; determine a target data node using the target sharding function; send the database execution instruction to the target data node so that the target data node executes a corresponding database operation and feeds back the execution result to the client.

[0166] In one possible implementation, in response to the database execution instruction, a database operation statement sent by the client is received; the database operation statement is parsed and processed by a syntax parser to obtain a first operation statement and a second operation statement corresponding to the database operation statement; and a target sharding function of the distributed database is determined according to the second operation statement.

[0167] In a possible implementation, a target sharding key of a distributed database is determined according to the first operation statement; and a target data node corresponding to the target sharding key is determined using the target sharding function.

[0168] In one possible implementation, an initial sharding key of a distributed database is obtained; it is determined whether the first operation statement is an empty statement; if the first operation statement is an empty statement, the initial sharding key is used as a target sharding key of the distributed database; if the first operation statement is a non-empty statement, the initial sharding key is filled to obtain a target sharding key.

[0169] In one possible implementation, an initial sharding function of a distributed database is obtained; it is determined whether the second operation statement is an empty statement; if the second operation statement is an empty statement, the initial sharding function is used as a target sharding function of the distributed database; if the second operation statement is a non-empty statement, a custom sharding function corresponding to a dynamic compilation library is obtained, and the custom sharding function is used as a target sharding function of the distributed database.

[0170] In one possible implementation, the initial shard key is padded with padding characters to obtain an initial padded shard key; whether the character string of the initial padded shard key complies with preset shard key rules is determined; if it complies with the preset shard key rules, the initial padded shard key is used as the target shard key; if it does not comply with the preset shard key rules, the initial padded shard key is further padded with invalid characters so that the character string of the padded shard key complies with the preset shard key rules.

[0171] In a possible implementation, a syntax execution tree is generated according to the database operation statement sent by the client; and according to the syntax execution tree, the database execution instruction is sent to the corresponding target data node.

[0172] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a general hardware platform, and of course, by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the relevant technology can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0173] It should be understood that the terms used herein are only for the purpose of describing specific example embodiments and are not intended to be limiting. Unless the context clearly indicates otherwise, the singular forms "one", "an" and "said" as used herein may also be meant to include plural forms. The terms "include", "comprise", "contain", and "have" are inclusive, and therefore specify the existence of stated features, steps, operations, elements and / or parts, but do not exclude the existence or addition of one or more other features, steps, operations, elements, parts, and / or combinations thereof. The method steps, processes, and operations described herein are not interpreted as necessarily requiring them to be performed in the specific order described or illustrated, unless the execution order is clearly indicated. It should also be understood that additional or alternative steps may be used.

[0174] The foregoing is merely a specific embodiment of the present invention, which enables those skilled in the art to understand or implement the present invention. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features claimed herein.

Claims

1. A data processing method for a distributed database, characterized in that: include: Get the database execution instructions sent by the client; In response to the database execution instruction, determining a target sharding function of the distributed database; Determine the target data node using the target sharding function; The database execution instruction is sent to the target data node, so that the target data node executes the corresponding database operation and feeds back the execution result to the client.

2. The method according to claim 1, characterized in that The step of determining a target sharding function of a distributed database in response to the database execution instruction includes: In response to the database execution instruction, receiving a database operation statement sent by a client; Parsing the database operation statement through a syntax parser to obtain a first operation statement and a second operation statement corresponding to the database operation statement; According to the second operation statement, a target sharding function of the distributed database is determined.

3. The method according to claim 2, characterized in that The step of using the target sharding function to determine the target data node includes: According to the first operation statement, determine a target shard key of the distributed database; The target data node corresponding to the target sharding key is determined using the target sharding function.

4. The method according to claim 2, characterized in that: Determining a target shard key of a distributed database according to the first operation statement includes: Get the initial shard key of the distributed database; Determine whether the first operation statement is an empty statement; When the first operation statement is an empty statement, the initial sharding key is used as the target sharding key of the distributed database; When the first operation statement is a non-empty statement, the initial sharding key is filled to obtain a target sharding key.

5. The method according to claim 2, characterized in that: The step of determining a target sharding function of a database of the client according to the second operation statement includes: Get the initial sharding function of the distributed database; Determine whether the second operation statement is an empty statement; When the second operation statement is an empty statement, the initial sharding function is used as the target sharding function of the distributed database; When the second operation statement is a non-empty statement, a custom sharding function corresponding to the dynamic compilation library is obtained, and the custom sharding function is used as a target sharding function of the distributed database.

6. The method according to claim 4, characterized in that The filling process of the initial shard key to obtain the target shard key includes: Filling the initial shard key with a filling character to obtain an initial filled shard key; Determine whether the character string of the initial filled shard key meets the preset shard key rule; In the case of meeting the preset shard key rule, the initial filled shard key is used as the target shard key; In the case of not meeting the preset shard key rules, the initially filled shard key is further filled with invalid characters so that the character string of the filled shard key meets the preset shard key rules.

7. The method according to claim 1, characterized in that The sending of the database execution instruction to the target data node includes: Generate a syntax execution tree based on the database operation statements sent by the client; According to the syntax execution tree, the database execution instruction is sent to the corresponding target data node.

8. A data processing device for a distributed database, characterized in that: include: The acquisition module is used to obtain the database execution instruction sent by the client; A first determination module, configured to determine a target sharding function of a distributed database in response to the database execution instruction; A second determination module is used to determine a target data node using the target sharding function; The sending module is used to send the database execution instruction to the target data node, so that the target data node executes the corresponding database operation and feeds back the execution result to the client.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the distributed database data processing method according to any one of claims 1 to 7 are implemented.

10. A storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the distributed database data processing method according to any one of claims 1 to 7 are implemented.

Images