Data processing method and device, electronic equipment, storage medium and program product

Abstract

The invention provides a data processing method which can be applied to the technical field of big data and data processing, and the data processing method comprises the following steps: in response to a received task instruction aiming at a distributed database, performing lexical analysis on the task instruction to obtain a task syntax tree; based on database configuration information of the distributed database, verifying the task syntax tree to obtain a verification result; under the condition that the verification result represents that the task instruction is normal, multiple sub-task instructions for multiple sub-databases are generated based on the task syntax tree and database configuration information, and the distributed database comprises the multiple sub-databases configured at different server nodes; and respectively sending the plurality of sub-task instructions to a plurality of server nodes, so that the server nodes update data in the sub-databases based on the sub-task instructions. The invention further provides a data processing device, electronic equipment, a storage medium and a program product.

Description

Technical Field

[0001] This disclosure relates to the fields of big data and data processing technology, and more specifically, to a data processing method, apparatus, electronic device, storage medium, and program product. Background Technology

[0002] A distributed database refers to a group of logically interconnected databases distributed across a computer network. With the continuous development of database technology, more and more distributed databases are using database sharding and table partitioning to store data.

[0003] In the process of realizing the concept of this disclosure, it was found that the related technology has at least the following problems: when updating data using Structured Query Language (SQL), there is a problem of slow data processing speed when parsing SQL. Summary of the Invention

[0004] In view of the above, this disclosure provides a data processing method, apparatus, electronic device, storage medium, and program product.

[0005] One aspect of this disclosure provides a data processing method, comprising: in response to receiving a task instruction for a distributed database, performing lexical analysis on the task instruction to obtain a task syntax tree; verifying the task syntax tree based on database configuration information of the distributed database to obtain a verification result; if the verification result indicates that the task instruction is normal, generating multiple sub-task instructions for multiple sub-databases based on the task syntax tree and the database configuration information, the distributed database including multiple sub-databases configured on different server nodes; and sending the multiple sub-task instructions to the multiple server nodes respectively, so that the server nodes update the data in the sub-databases based on the sub-task instructions.

[0006] According to embodiments of this disclosure, lexical analysis is performed on task instructions to obtain a task syntax tree, including: splitting the task instructions into fields to obtain a keyword set, wherein the keyword set includes an object word subset for representing the object to be processed and an operation word subset for representing the operation content; and dividing the object word subset and the operation word subset into nodes and determining the association relationship between multiple nodes to generate a task syntax tree.

[0007] According to embodiments of this disclosure, the task syntax tree is validated based on the database configuration information of the distributed database to obtain a validation result, including at least one of the following: a validation result determining whether the object attribute information used to represent the object to be processed in the task syntax tree is valid based on the object configuration information in the database configuration information; and a validation result determining whether the operation attribute information used to represent the operation content in the task syntax tree is valid based on the operation configuration information in the database configuration information.

[0008] According to embodiments of this disclosure, the objects to be processed in the task syntax tree include data tables; based on the task syntax tree and database configuration information, multiple sub-task instructions for multiple sub-databases are generated, including: dividing the data tables into multiple data sub-tables based on table attribute information used to characterize the data tables in the task syntax tree; determining multiple sub-databases that match the multiple data sub-tables from the database configuration information; and generating multiple sub-task instructions for multiple sub-databases based on the table operation information, table attribute information of each of the multiple data sub-tables involved in the task syntax tree and the database configuration information of each of the multiple sub-databases.

[0009] According to embodiments of this disclosure, based on the table operation information, table attribute information, and database configuration information of each of the multiple data sub-tables involved in the task syntax tree, multiple sub-task instructions for the multiple sub-databases are generated, including: performing syntax recognition on target information to obtain a syntax recognition result, wherein the target information includes at least one of table attribute information and table operation information; if the syntax recognition result indicates that the target information does not match the database configuration information, the target information is converted to obtain converted target information, wherein the information mismatch includes at least one of the following: rule mismatch, format mismatch, and language mismatch; and generating a task instruction set based on the converted target information.

[0010] According to embodiments of this disclosure, the data processing method further includes: receiving data processing results sent from multiple server nodes to obtain a processing result set; obtaining target data based on the processing result set if all processing results in the processing result set indicate successful data update; and sending a task rollback instruction to multiple server nodes if at least one processing result in the processing result set indicates data update failure.

[0011] According to embodiments of this disclosure, the data processing method further includes: if the verification result indicates that the task instruction is abnormal, deleting the node with the abnormality from the task syntax tree based on the verification result; or sending feedback information indicating that the task instruction is abnormal based on the verification result.

[0012] Another aspect of this disclosure provides a data processing apparatus, comprising: an analysis module, configured to perform lexical analysis on the task instruction in response to receiving a task instruction for a distributed database, to obtain a task syntax tree; a verification module, configured to verify the task syntax tree based on database configuration information of the distributed database, to obtain a verification result; a generation module, configured to generate multiple sub-task instructions for multiple sub-databases based on the task syntax tree and database configuration information, provided that the verification result indicates the task instruction is normal; the distributed database includes multiple sub-databases configured on different server nodes; and a sending module, configured to send the multiple sub-task instructions to the multiple server nodes respectively, so that the server nodes update the data in the sub-databases based on the sub-task instructions.

[0013] Another aspect of this disclosure provides an electronic device, including: one or more processors; and a memory for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors cause the one or more processors to implement the method described above.

[0014] Another aspect of this disclosure provides a computer-readable storage medium storing computer-executable instructions that, when executed, are used to implement the methods described above.

[0015] Another aspect of this disclosure provides a computer program product including computer-executable instructions that, when executed, are used to implement the methods described above.

[0016] According to embodiments of this disclosure, since a task syntax tree is obtained after lexical analysis of the task instructions, the task instructions can be split at a fine-grained level and then flexibly combined to obtain multiple sub-task instructions. This improves the flexibility and efficiency of sub-task instruction generation, enables dynamic adjustment of the execution strategy for database sharding and table partitioning, achieves accurate data updates in sub-data tables, improves data processing efficiency, reduces resource consumption, and provides convenience for users when defining and managing distributed data. Furthermore, by validating the task syntax tree, the reliability of task execution can be improved, preventing anomalies. Attached Figure Description

[0017] The above and other objects, features and advantages of this disclosure will become clearer from the following description of embodiments with reference to the accompanying drawings, in which:

[0018] Figure 1 The illustrations depict application scenarios of data processing methods, apparatuses, electronic devices, storage media, and program products according to embodiments of the present disclosure.

[0019] Figure 2 A flowchart illustrating a data processing method according to an embodiment of the present disclosure is shown schematically.

[0020] Figure 3 A schematic diagram illustrating the structure of a task syntax tree according to an embodiment of the present disclosure is shown.

[0021] Figure 4 The illustration schematically depicts a diagram of generating multiple subtask instructions for multiple sub-databases according to an embodiment of the present disclosure;

[0022] Figure 5 A schematic diagram illustrating a data processing method according to another embodiment of the present disclosure is shown.

[0023] Figure 6 A schematic block diagram of a data processing apparatus according to embodiments of the present disclosure is shown; and

[0024] Figure 7 A block diagram schematically illustrates an electronic device suitable for implementing a data processing method according to an embodiment of the present disclosure. Detailed Implementation

[0025] The embodiments of the present disclosure will now be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the disclosure. In the following detailed description, numerous specific details are set forth to provide a thorough understanding of the embodiments of the present disclosure for ease of explanation. However, it will be apparent that one or more embodiments may be practiced without these specific details. Furthermore, descriptions of well-known structures and techniques are omitted in the following description to avoid unnecessarily obscuring the concepts of the present disclosure.

[0026] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit this disclosure. The terms “comprising,” “including,” etc., as used herein indicate the presence of features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0027] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein are to be interpreted in a manner consistent with the context of this specification, and not in an idealized or overly rigid way.

[0028] When using expressions such as "at least one of A, B and C", they should generally be interpreted in accordance with the meaning that is commonly understood by those skilled in the art (e.g., "a system having at least one of A, B and C" should include, but is not limited to, a system having A alone, a system having B alone, a system having C alone, a system having A and B, a system having A and C, a system having B and C, and / or a system having A, B and C, etc.).

[0029] In the embodiments disclosed herein, the collection, updating, analysis, processing, use, transmission, provision, disclosure, and storage of data (e.g., including but not limited to user personal information) comply with relevant laws and regulations, are used for legitimate purposes, and do not violate public order and good morals. In particular, necessary measures have been taken to prevent unauthorized access to user personal information data and to safeguard user personal information security and network security.

[0030] In the process of implementing the embodiments of this disclosure, it was discovered that there are obvious limitations in the parsing of distributed database DDL.

[0031] For example, the parser can only support parsing the standard Structured Query Language (SQL). In a distributed environment, after database sharding and table partitioning, the table structure is split into multiple databases or tables. The SQL statements that were originally for single or multiple tables are no longer applicable. The SQL statements need to be rewritten and optimized in a complex way to adapt to the data distribution after database sharding and table partitioning. In addition, it needs to handle more data and more complex query logic, which consumes a lot of resources and has a slow processing speed.

[0032] Furthermore, the parser needs to understand the distribution of data across different database shards and tables in order to correctly route SQL statements to the relevant databases and tables. This requires the parser to be able to parse the conditions in the SQL statement and determine the shard where the data resides based on the rules of database sharding and table partitioning. However, in complex distributed environments, data distribution is complex and dynamically changing, making it difficult for the parser to obtain accurate data distribution information in real time, leading to routing errors or inefficiency.

[0033] Furthermore, the parser lacks support for the versatile DistSQL (DistSQL) syntax, limiting the ease with which users can define and manage distributed data.

[0034] In view of this, embodiments of the present disclosure provide a data processing method comprising: responding to receiving a task instruction for a distributed database, performing lexical analysis on the task instruction to obtain a task syntax tree; verifying the task syntax tree based on the database configuration information of the distributed database to obtain a verification result; if the verification result indicates that the task instruction is normal, generating multiple sub-task instructions for multiple sub-databases based on the task syntax tree and the database configuration information, wherein the distributed database includes multiple sub-databases configured on different server nodes; and sending the multiple sub-task instructions to the multiple server nodes respectively, so that the server nodes update the data in the sub-databases based on the sub-task instructions.

[0035] Figure 1The illustrations depict application scenarios of data processing methods, apparatuses, electronic devices, storage media, and program products according to embodiments of the present disclosure.

[0036] like Figure 1 As shown, application scenario 100 according to this embodiment may include a first terminal device 101, a second terminal device 102, a third terminal device 103, a network 104, a management server 105, and a server cluster 106. The network 104 serves as a medium for providing communication links between the first terminal device 101, the second terminal device 102, the third terminal device 103, and the management server 105, and between the management server 105 and the server cluster 106. The network 104 may include various connection types, such as wired and / or wireless communication links, etc.

[0037] Users can use the first terminal device 101, the second terminal device 102, and the third terminal device 103 to interact with the management server 105 via the network 104 to receive or send messages, etc. Various communication client applications can be installed on the first terminal device 101, the second terminal device 102, and the third terminal device 103, such as data processing applications, shopping applications, web browser applications, search applications, instant messaging tools, email clients, and / or social platform software, etc. (for example only).

[0038] The first terminal device 101, the second terminal device 102, and the third terminal device 103 can be various electronic devices with displays and support web browsing, including but not limited to smartphones, tablets, laptops, and desktop computers.

[0039] The management server 105 can be a server that provides various services, such as a backend management server that supports websites browsed by users using the first terminal device 101, the second terminal device 102, and the third terminal device 103 (this is just an example). The backend management server can analyze and process data such as received user requests, and feed back the processing results (such as web pages, information, or data obtained or generated according to user requests) to the terminal devices.

[0040] Server cluster 106 may include multiple servers, and multiple sub-data tables may be configured on these servers. Management server 105 may be used to distribute tasks to server cluster 106, and server cluster 106 may return processing results to management server 105.

[0041] It should be noted that the data processing method provided in this embodiment can generally be executed by the management server 105. Correspondingly, the data processing device provided in this embodiment can generally be located in the management server 105. The data processing method provided in this embodiment can also be executed by a server or server cluster that is different from the management server 105 and capable of communicating with the first terminal device 101, the second terminal device 102, the third terminal device 103, and / or the management server 105. Correspondingly, the data processing device provided in this embodiment can also be located in a server or server cluster that is different from the management server 105 and capable of communicating with the first terminal device 101, the second terminal device 102, the third terminal device 103, and / or the management server 105. Alternatively, the data processing method provided in this embodiment can also be executed by the first terminal device 101, the second terminal device 102, or the third terminal device 103, or by other terminal devices different from the first terminal device 101, the second terminal device 102, or the third terminal device 103. Accordingly, the data processing device provided in the embodiments of this disclosure may also be disposed in the first terminal device 101, the second terminal device 102 or the third terminal device 103, or disposed in other terminal devices different from the first terminal device 101, the second terminal device 102 or the third terminal device 103.

[0042] It should be understood that Figure 1 The number of terminal devices, networks, management servers, and server clusters shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, management servers, and server clusters can be included.

[0043] Figure 2 A flowchart illustrating a data processing method according to an embodiment of the present disclosure is shown schematically.

[0044] like Figure 2 As shown, the method includes operations S210~S240.

[0045] In operation S210, in response to receiving a task instruction for the distributed database, lexical analysis is performed on the task instruction to obtain the task syntax tree.

[0046] When operating S220, the task syntax tree is validated based on the database configuration information of the distributed database, and the validation result is obtained.

[0047] When operating S230, if the verification result indicates that the task instructions are normal, multiple sub-task instructions for multiple sub-databases are generated based on the task syntax tree and database configuration information.

[0048] In operation S240, multiple subtask instructions are sent to multiple server nodes so that the server nodes can update the data in the sub-database based on the subtask instructions.

[0049] In this embodiment of the disclosure, the distributed database may include multiple sub-databases configured on different server nodes.

[0050] Task instructions may include SQL statements, which may include, but are not limited to, statements that add columns to a data table, change column types and lengths, add constraints, or change indexes.

[0051] Lexical analysis can include, but is not limited to, sequential analysis of each word in an SQL statement and constructing a task syntax tree based on the relationships between words.

[0052] For example, sharding rules can be extracted from an SQL statement. If the sharding data is empty, it means the current SQL statement does not involve sharding rules, and default parameters can be returned directly. If the sharding data is not empty, it means the current SQL statement does not involve sharding rules, meaning the task instruction is for a distributed database.

[0053] The database configuration information for a distributed database may include, but is not limited to, configuration of table names, partitioning rules, syntax, format, and connection information. Partitioning rule configuration may include at least one of the following: partitioning key selection and partitioning algorithm settings. Syntax configuration may include DistSQL syntax configuration. Connection information may include the database address and port, database username and password, etc.

[0054] Validating the task syntax tree can include verifying the syntax and formatting of each node in the task syntax tree against the database syntax and formatting configurations. For example, if the syntax and formatting of each node in the task syntax tree conform to the database syntax and formatting configurations, the task instructions can be considered normal.

[0055] Each subtask instruction can be a task instruction for each sub-database. For example, it can include Data Definition Language (DDL) statements for the sub-database, such as statements for creating, modifying, and deleting databases, tables, views, indexes, etc.

[0056] For example, when a DDL statement adds a column to a sub-database, the server node can update the data by adding the column to the sub-database. Similarly, when a DDL statement changes the column type in a sub-database, the server node can change the column type in the sub-database, also updating the data.

[0057] Since lexical analysis of task instructions yields a task syntax tree, these instructions can be broken down into fine-grained segments and flexibly combined to generate multiple sub-task instructions. This improves the flexibility and efficiency of sub-task instruction generation, allows for dynamic adjustment of database sharding strategies within task instructions, enables precise data updates in sub-tables, improves data processing efficiency, reduces resource consumption, and provides convenience for users in defining and managing distributed data. Furthermore, validating the task syntax tree enhances the reliability of task execution and prevents anomalies.

[0058] According to another embodiment of this disclosure, the data processing method may include, in addition to the above-described methods, Figure 2 In addition to the operations S210 to S240 shown, the following operations may also be included: if the verification result indicates that the task instruction is abnormal, delete the node with the abnormality from the task syntax tree based on the verification result; or send feedback information indicating that the task instruction is abnormal based on the verification result.

[0059] For example, validating the task syntax tree can include verifying the syntax and format of each node in the task syntax tree against the database syntax configuration and format configuration. For instance, if a node in the task syntax tree fails to conform to the database syntax or format configuration, the task instruction can be determined to be abnormal. The abnormal node in the task syntax tree can be deleted, or feedback information about the abnormal task instruction can be sent to the client. This feedback information indicates that there are abnormal nodes in the task syntax tree whose syntax or format does not conform to the database syntax or format configuration.

[0060] Since the abnormal node in the task syntax tree is deleted or feedback information is sent when the verification result indicates that the task instruction is abnormal, the occurrence of abnormalities can be avoided.

[0061] According to embodiments of this disclosure, for example, Figure 2 Operation S210, as shown, performs lexical analysis on the task instructions to obtain a task syntax tree. This may include operations such as: splitting the task instructions into fields to obtain a keyword set; partitioning the object word subset and operation word subset into nodes and determining the relationships between multiple nodes to generate the task syntax tree.

[0062] For example, the keyword set may include a subset of object words used to characterize the object to be processed and a subset of operation words used to characterize the operation content. The object to be processed may include data tables, databases, indexes, views, etc. The operation content may include creating, modifying, and deleting databases, data tables, views, indexes, etc.

[0063] For example, if the task instruction is an SQL statement, each word in the SQL statement can include object terms and operation terms. Therefore, the subset of object terms could include, for example, table names, field names, etc. Field names are used to identify the name of each column in the table structure; for example, a distributed key can be specified. The subset of operation terms could include, for example, database sharding algorithm names, table sharding algorithm names, adding columns to a table, changing column types, lengths, adding constraints, etc.

[0064] For example, nodes can be hierarchically divided according to the order in which object terms and operation terms appear in the SQL statement, and then node relationships can be established based on the association between nodes to obtain the task syntax tree.

[0065] Figure 3 A schematic diagram illustrating the structure of a task syntax tree according to an embodiment of the present disclosure is shown.

[0066] For an SQL statement whose task instruction is to create a data table, field splitting yields a subset of object terms, such as "Table 1 Name," "Table 2 Name," "Table 3 Name," "Field 1 Name," "Field 2 Name," and "Field 3 Name." The subset of operation terms includes terms like "create," "Algorithm 1 Name," "Algorithm 2 Name," and "Algorithm 3 Name."

[0067] like Figure 3 As shown, creation 310 can be used as the root node of the task syntax tree. The first data table name 321, the second data table name 322, and the third data table name 323 can be used as the next-level nodes associated with the root node creation 310. The first field name 331, the second field name 332, and the third field name 333 can be associated with the first data table name 321, the second data table name 322, and the third data table name 323, respectively, as nodes at the next level. The first algorithm name 341, the second algorithm name 342, and the third algorithm name 343 can be associated with the first field name 331, the second field name 332, and the third field name 333, respectively, as nodes at yet another level, thus obtaining the task syntax tree.

[0068] By splitting task instructions into fields, the objects to be processed and the operations to be performed within each instruction can be clearly identified. Therefore, constructing a task syntax tree based on these objects and operations allows for the parsing of various task instructions, offering strong dynamic scalability and facilitating rule validation. Furthermore, the relationships between nodes clearly demonstrate complex connections, which is beneficial for the flexible splitting of task instructions.

[0069] According to another embodiment of this disclosure, while manually configuring and customizing the sharding and partitioning business process through code, and implementing the SQL parser in the business code, provides greater flexibility and control, the development and maintenance costs are higher. A series of sharding and partitioning templates (such as hash-based sharding, range-based sharding, broadcast tables, etc.) can be provided, and DDL statements can be automatically generated after the user selects a template based on the predefined templates.

[0070] According to embodiments of this disclosure, for example, Figure 2 Operation S220, as shown, verifies the task syntax tree based on the database configuration information of the distributed database, and obtains a verification result. This may include at least one of the following operations: determining the validity of object attribute information in the task syntax tree representing the object to be processed based on object configuration information in the database configuration information; and / or determining the validity of operation attribute information in the task syntax tree representing the operation content based on operation configuration information in the database configuration information.

[0071] For example, object configuration information may include, but is not limited to, data table configuration information, such as specifying distributed keys, whether the table should be broadcast on all nodes, whether the table should be stored on only one node and specifying that node, defining table sharding rules and algorithms, defining database sharding rules, and defining data encryption configuration information, etc.

[0072] Operation configuration information may include, but is not limited to, creation operation configuration information, deletion operation configuration information, database sharding configuration information, and table sharding configuration information.

[0073] Validation can include data mapping validation and conflict validation. For example, data mapping validation can include the following operations: determining the object address based on database configuration information, searching for data from the object address based on object attribute information, and determining the object attribute information as valid if the search is successful, otherwise invalid. Conflict validation can include the following operations: determining whether there is a conflict between operation configuration information and operation attribute information; if there is no conflict, the operation attribute information is valid, otherwise invalid.

[0074] For example, continuing with the above... Figure 3 The example shown demonstrates how to determine the table address based on the table configuration information. Based on the first table name 321, it checks if the first table name 321 exists in the table address. If it does, it checks if the first field name 331 exists. If both exist, then the first table name 321 and the first field name 331 are valid. This allows for the determination of the database sharding configuration information and table sharding configuration information. Figure 3 Check if there is a conflict between the operation information indicated by the first algorithm name 341. If there is, the operation attribute information is invalid; if not, the operation attribute information is valid.

[0075] By validating the object attribute information and operation attribute information used to represent the object to be processed in the constructed task syntax tree, the legality of the object attribute information and operation attribute information can be guaranteed. Therefore, it is possible to avoid the anomalies of subtask instructions generated based on the task syntax tree caused by task syntax tree anomalies, thereby improving the reliability and effectiveness of subtask instructions.

[0076] According to embodiments of this disclosure, the objects to be processed in the task syntax tree may include data tables.

[0077] For example, based on the task syntax tree and database configuration information, multiple subtask instructions for multiple sub-databases are generated, which may include the following operations: dividing the data table into multiple data sub-tables based on the table attribute information used to represent the data table in the task syntax tree; determining multiple sub-databases that match the multiple data sub-tables from the database configuration information; and generating multiple subtask instructions for the multiple sub-databases based on the table operation information, table attribute information, and database configuration information of each of the multiple data sub-tables involved in the task syntax tree and the respective database configuration information of the multiple sub-databases.

[0078] Table attribute information can include the table name, the names of the fields in the table, the data types of the fields, and the data length of the fields.

[0079] For example, you can divide the fields or columns in a data table to get multiple data sub-tables. You can also divide the fields or columns in a data table whose field names indicate the same region to get multiple data sub-tables. For example, if a data table includes data from region A and data from region B, you can divide the data from region A into a data sub-table for region A and the data from region B into a data sub-table for region B.

[0080] Table operation information can include data table deletion, addition, field modification, etc.

[0081] Multiple subtask instructions can convert user-defined task instructions into instructions that conform to standardized configurations, such as filling in the default hash shard number.

[0082] Since data tables can be partitioned based on table attribute information, task instructions can be split. Therefore, based on the split sub-task instructions, the flexibility and dynamic adjustment of task execution can be improved.

[0083] Figure 4 The illustration schematically depicts a diagram of generating multiple subtask instructions for multiple sub-databases according to an embodiment of the present disclosure.

[0084] For example, based on the table operation information, table attribute information, and database configuration information of each of the multiple data sub-tables involved in the task syntax tree, multiple sub-task instructions targeting multiple sub-databases can be generated, which may include, for example... Figure 4 Operations S401 to S406 are shown.

[0085] In operation S401, syntax recognition is performed on the target information.

[0086] When operating S402, check whether the target information matches the database configuration rules.

[0087] When operating S403, check if the target information matches the database configuration information format.

[0088] When operating S404, check if the target information matches the language of the database configuration information.

[0089] In operation S405, the target information is transformed to obtain the transformed target information.

[0090] When operating S406, a task instruction set is generated.

[0091] The target information may include at least one of table attribute information and table operation information. Information mismatch may include at least one of the following: rule mismatch, format mismatch, or language mismatch.

[0092] The execution of operation S401 may involve the recognition of target information and database configuration information rules, and / or format, and / or language.

[0093] For rule recognition, operation S402 can be executed to determine whether the target information matches the rules in the database configuration information. If they match, operation S406 can be executed. If they do not match, operation S405 can be executed. For example, based on the rules in the database configuration information, the target information can be transformed according to the rules to obtain the transformed target information that matches the rules, and then operation S406 can be executed.

[0094] For format recognition, operation S403 can be executed to determine whether the target information matches the format of the database configuration information. If they match, operation S406 can be executed. If they do not match, operation S405 can be executed. For example, the target information can be format-converted based on the database configuration information format to obtain the converted target information with a matching format, and then operation S406 can be executed.

[0095] For language recognition, operation S404 can be executed to determine whether the target information matches the language of the database configuration information. If they match, operation S406 can be executed. If they do not match, operation S405 can be executed. For example, the target information can be converted to a different language based on the language of the database configuration information to obtain the converted target information that matches the language, and then operation S406 can be executed.

[0096] When the table attribute information and table operation information in the task syntax tree do not match the database configuration information, the task instructions are split into multiple sub-task instructions for task processing, which cannot accurately process the task. Therefore, in the case of mismatch with the database configuration information, the task adaptability can be improved by flexibly adjusting the conversion of the table attribute information and / or table operation information.

[0097] In implementing the embodiments of this disclosure, it was also discovered that database sharding and table partitioning in a distributed database introduce distributed transaction problems. In a single-machine database, transaction management is relatively simple because all data is stored in one database, and the atomicity of transactions can be guaranteed using the database's built-in transaction mechanism. However, in a distributed environment, a transaction may involve operations across multiple databases, requiring the use of a distributed transaction protocol to coordinate transactions across different databases. This increases the complexity and overhead of transaction management and places higher demands on the parser's ability to handle transactions, making it difficult to guarantee database operation consistency and effective schema management. For example, when a user submits an asynchronous task instruction, they can set SQL rule policies. The distributed database can extract the policies through the SQL parser and then persist the task metadata, such as SQL, policies, and routing information, to the database in a transactional manner for task processing.

[0098] Based on this, according to the embodiments provided in this disclosure, when a user submits an asynchronous task instruction, they can set SQL rule strategies. The distributed database management system can obtain multiple sub-task instructions, such as DDL statements, by executing the operations shown in the above embodiments. The task instructions, database configuration information, and routing information are persisted to the database in a transactional manner, returning a unique task identifier. The scheduler of the distributed database management system can monitor the task queue and dynamically allocate resources. The distributed database management system distributes multiple sub-task instructions to multiple server nodes, where update operations are executed concurrently. During execution, the status is returned to the distributed database management system in real time, and temporary data can be cleaned up uniformly upon completion.

[0099] In addition, data processing methods may include, in addition to, the methods described above. Figure 2In addition to the operations S210 to S240 shown, the following operations may also be included: receiving data processing results sent from multiple server nodes to obtain a processing result set; if all processing results in the processing result set indicate successful data updates, obtaining the target data based on the processing result set; if at least one processing result in the processing result set indicates data update failure, sending a task rollback command to multiple server nodes.

[0100] Data processing results can be used to characterize the state of data updates. Target data can include updated data from the tables on each server node. Task rollback commands can be used to cancel uncommitted operations within a transaction, restoring the database state to its state before the transaction began.

[0101] Figure 5 A schematic diagram of a data processing method according to another embodiment of the present disclosure is shown.

[0102] For example, such as Figure 5 As shown, when there are N subtask instructions, where N is an integer greater than or equal to 2, the first subtask instruction 511, the second subtask instruction 512, ..., the Nth subtask instruction 51N can be sent to the first server node 521, the second server node 521, ..., the Nth server node 51N, respectively. Data updates are performed on the N server nodes 520, and the data processing results are sent to the processing result set 530 according to their respective update status. When all data processing results in the processing result set 530 indicate successful data updates, the target data 540 is obtained based on the data in the updated data tables of each server node. If any data processing result in the processing result set 530 indicates a data update failure, a task rollback instruction 550 is sent to all N server nodes 520.

[0103] Since the target data is determined only after the data has been successfully updated on all server nodes, if the data update fails on any server node, a task rollback command is sent to restore the database state to the state before the transaction started, thus ensuring the atomicity and consistency of the transaction.

[0104] This disclosure supports multiple distributed statements and can quickly support multiple database sharding and table partitioning algorithms to meet various complex database sharding and table partitioning requirements.

[0105] Figure 6 A schematic block diagram of a data processing apparatus according to an embodiment of the present disclosure is shown.

[0106] like Figure 6 As shown, the data processing apparatus 600 of the embodiments of this disclosure includes a content acquisition module 610, a search module 620, and a display module 630.

[0107] The analysis module 610 is used to perform lexical analysis on the task instruction received for the distributed database in response to obtain the task syntax tree. In one embodiment, the analysis module 610 can be used to perform the operation S210 described above, which will not be repeated here.

[0108] The verification module 620 is used to verify the task syntax tree based on the database configuration information of the distributed database and obtain the verification result. In one embodiment, the verification module 620 can be used to perform the operation S220 described above, which will not be repeated here.

[0109] The generation module 630 is used to generate multiple sub-task instructions for multiple sub-databases based on the task syntax tree and database configuration information, provided that the verification result indicates the task instructions are normal. The distributed database includes multiple sub-databases configured on different server nodes. In one embodiment, the generation module 630 can be used to perform the operation S230 described above, which will not be repeated here.

[0110] The sending module 640 is used to send multiple subtask instructions to multiple server nodes respectively, so that the server nodes can update the data in the sub-database based on the subtask instructions. In one embodiment, the sending module 640 can be used to perform the operation S240 described above, which will not be repeated here.

[0111] According to embodiments of this disclosure, the analysis module 610 includes a splitting unit and a first partitioning unit. The splitting unit is used to split the task instructions into fields to obtain a keyword set, wherein the keyword set includes a subset of object words representing the object to be processed and a subset of operation words representing the operation content. The first partitioning unit is used to partition the object word subset and the operation word subset into nodes and determine the relationships between multiple nodes to generate a task syntax tree.

[0112] According to embodiments of this disclosure, the verification module 620 includes at least one of the following: a first determining unit and a second determining unit. The first determining unit is used to determine, based on object configuration information in the database configuration information, whether the object attribute information in the task syntax tree representing the object to be processed is valid. The second determining unit is used to determine, based on operation configuration information in the database configuration information, whether the operation attribute information in the task syntax tree representing the operation content is valid.

[0113] According to embodiments of this disclosure, the objects to be processed in the task syntax tree include data tables. The generation module 630 includes a first partitioning unit, a third determining unit, and a sub-generation unit. The first partitioning unit is used to partition the data table based on table attribute information used to characterize the data table in the task syntax tree, obtaining multiple data sub-tables. The third determining unit is used to determine multiple sub-databases matching the multiple data sub-tables from database configuration information. The sub-generation unit is used to generate multiple sub-task instructions for the multiple sub-databases based on the table operation information and table attribute information of each of the multiple data sub-tables involved in the task syntax tree and the database configuration information of each of the multiple sub-databases.

[0114] According to embodiments of this disclosure, based on the table operation information, table attribute information, and database configuration information of each of the multiple data sub-tables involved in the task syntax tree, multiple sub-task instructions for the multiple sub-databases are generated, including: performing syntax recognition on target information to obtain a syntax recognition result, wherein the target information includes at least one of table attribute information and table operation information; if the syntax recognition result indicates that the target information does not match the database configuration information, the target information is converted to obtain converted target information, wherein the information mismatch includes at least one of the following: rule mismatch, format mismatch, and language mismatch; and generating a task instruction set based on the converted target information.

[0115] According to embodiments of this disclosure, the data processing apparatus 600 further includes: a receiving module, a data obtaining module, and a rollback command sending module. The receiving module receives data processing results sent from multiple server nodes to obtain a processing result set. The data obtaining module obtains target data based on the processing result set if all processing results in the processing result set indicate successful data updates. The rollback command sending module sends a task rollback command to the multiple server nodes if at least one processing result in the processing result set indicates a data update failure.

[0116] According to embodiments of this disclosure, the data processing apparatus 600 further includes a deletion module or a feedback information sending module. The deletion module is used to delete the node containing the error from the task syntax tree based on the verification result if the verification result indicates an error in the task instruction. The feedback information sending module is used to send feedback information indicating an error in the task instruction based on the verification result.

[0117] Any one or more of the modules, submodules, units, and subunits according to embodiments of the present disclosure, or at least part of the functions of any one or more of them, can be implemented in one module. Any one or more of the modules, submodules, units, and subunits according to embodiments of the present disclosure can be implemented by dividing them into multiple modules. Any one or more of the modules, submodules, units, and subunits according to embodiments of the present disclosure can be at least partially implemented as hardware circuitry, such as a Field-Programmable Gate Array (FPGA), a Programmable Logic Array (PLA), a System-on-Chip, a System-on-a-Substrate, a System-on-Package, an Application-Specific Integrated Circuit (ASIC), or implemented in hardware or firmware by any other reasonable means of integrating or packaging circuitry, or implemented in software, hardware, or firmware, or in any suitable combination of any of these three implementation methods. Alternatively, one or more of the modules, submodules, units, and subunits according to embodiments of the present disclosure can be at least partially implemented as computer program modules, which, when run, can perform corresponding functions.

[0118] For example, any plurality of the content acquisition module 610, search module 620, and display module 630 may be combined into one module / unit / subunit, or any one of these modules / units / subunits may be split into multiple modules / units / subunits. Alternatively, at least part of the functionality of one or more of these modules / units / subunits may be combined with at least part of the functionality of other modules / units / subunits and implemented in one module / unit / subunit. According to embodiments of this disclosure, at least one of the content acquisition module 610, search module 620, and display module 630 may be at least partially implemented as hardware circuitry, such as a field-programmable gate array (FPGA), a programmable logic array (PLA), a system-on-a-chip, a system-on-a-substrate, a system-on-package, an application-specific integrated circuit (ASIC), or any other reasonable means of integrating or packaging the circuitry, or implemented in software, hardware, or firmware, or in any suitable combination of any of these three implementation methods. Alternatively, at least one of the content acquisition module 610, the search module 620, and the display module 630 may be implemented at least partially as a computer program module, which can perform corresponding functions when the computer program module is run.

[0119] It should be noted that the data processing device part in the embodiments of this disclosure corresponds to the data processing method part in the embodiments of this disclosure. The specific description of the data processing device part is referred to in the data processing method part, and will not be repeated here.

[0120] Figure 7A block diagram schematically illustrates an electronic device suitable for implementing a data processing method according to an embodiment of the present disclosure.

[0121] Figure 7 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments disclosed herein.

[0122] like Figure 7 As shown, an electronic device 700 according to an embodiment of the present disclosure includes a processor 701, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 702 or a program loaded from a storage portion 708 into a random access memory (RAM) 703. The processor 701 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or an associated chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 701 may also include onboard memory for caching purposes. The processor 701 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present disclosure.

[0123] RAM 703 stores various programs and data required for the operation of electronic device 700. Processor 701, ROM 702, and RAM 703 are interconnected via bus 704. Processor 701 performs various operations of the method flow according to embodiments of the present disclosure by executing programs in ROM 702 and / or RAM 703. It should be noted that programs may also be stored in one or more memories other than ROM 702 and RAM 703. Processor 701 may also perform various operations of the method flow according to embodiments of the present disclosure by executing programs stored in one or more memories.

[0124] According to embodiments of this disclosure, the electronic device 700 may further include an input / output (I / O) interface 705, which is also connected to a bus 704. The system 700 may also include one or more of the following components connected to the input / output (I / O) interface 705: an input section 706 including a keyboard, mouse, etc.; an output section 707 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 708 including a hard disk, etc.; and a communication section 709 including a network interface card such as a LAN card, modem, etc. The communication section 709 performs communication processing via a network such as the Internet. A drive 710 is also connected to the input / output (I / O) interface 705 as needed. A removable medium 711, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on the drive 710 as needed so that computer programs read from it can be installed into the storage section 708 as needed.

[0125] According to embodiments of this disclosure, the method flow according to embodiments of this disclosure can be implemented as a computer software program. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a computer-readable storage medium, the computer program containing program code for performing the methods shown in the flowchart. In such embodiments, the computer program can be downloaded and installed from a network via communication section 709, and / or installed from removable medium 711. When the computer program is executed by processor 701, it performs the functions defined in the system of embodiments of this disclosure. According to embodiments of this disclosure, the systems, devices, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0126] This disclosure also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or it may exist independently and not assembled into the device / apparatus / system. The computer-readable storage medium carries one or more programs that, when executed, implement the method according to the embodiments of this disclosure.

[0127] According to embodiments of this disclosure, the computer-readable storage medium can be a non-volatile computer-readable storage medium. Examples include, but are not limited to: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this disclosure, the computer-readable storage medium can be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0128] For example, according to embodiments of this disclosure, a computer-readable storage medium may include the ROM 702 and / or RAM 703 described above and / or one or more memories other than ROM 702 and RAM 703.

[0129] Embodiments of this disclosure also include a computer program product comprising a computer program containing program code for performing the methods provided in the embodiments of this disclosure. When the computer program product is run on an electronic device, the program code is used to enable the electronic device to implement the methods provided in the embodiments of this disclosure.

[0130] When the computer program is executed by the processor 701, it performs the functions defined in the system / apparatus of this disclosure embodiments. According to embodiments of this disclosure, the systems, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0131] In one embodiment, the computer program may rely on a tangible storage medium such as an optical storage device or a magnetic storage device. In another embodiment, the computer program may also be transmitted and distributed in the form of signals over a network medium, and may be downloaded and installed via the communication section 709, and / or installed from a removable medium 711. The program code contained in the computer program can be transmitted using any suitable network medium, including but not limited to: wireless, wired, etc., or any suitable combination thereof.

[0132] According to embodiments of this disclosure, program code for executing the computer programs provided in embodiments of this disclosure can be written in any combination of one or more programming languages. Specifically, these computational programs can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, languages ​​such as Java, C++, Python, "C", or similar programming languages. The program code can execute entirely on a user's computing device, partially on a user's device, partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0133] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions. Those skilled in the art will understand that the features recited in the various embodiments and / or claims of this disclosure can be combined and / or combined in various ways, even if such combinations or combinations are not expressly described in this disclosure. In particular, the features described in the various embodiments and / or claims of this disclosure may be combined and / or combined in various ways without departing from the spirit and teachings of this disclosure. All such combinations and / or combinations fall within the scope of this disclosure.

[0134] The embodiments of this disclosure have been described above. However, these embodiments are for illustrative purposes only and are not intended to limit the scope of this disclosure. Although various embodiments have been described above, this does not mean that the measures in the various embodiments cannot be used advantageously in combination. The scope of this disclosure is defined by the appended claims and their equivalents. Various substitutions and modifications can be made by those skilled in the art without departing from the scope of this disclosure, and all such substitutions and modifications should fall within the scope of this disclosure.

Claims

1. A data processing method, comprising: In response to receiving a task instruction for a distributed database, lexical analysis is performed on the task instruction to obtain a task syntax tree; Based on the database configuration information of the distributed database, the task syntax tree is validated to obtain the validation result; If the verification result indicates that the task instruction is normal, based on the task syntax tree and the database configuration information, multiple sub-task instructions for multiple sub-databases are generated, and the distributed database includes multiple sub-databases configured on different server nodes; as well as The multiple subtask instructions are sent to the multiple server nodes respectively, so that the server nodes update the data in the sub-database based on the subtask instructions.

2. The method according to claim 1, wherein, The step of performing lexical analysis on the task instructions to obtain a task syntax tree includes: The task instruction is split into fields to obtain a keyword set, wherein the keyword set includes a subset of object words representing the object to be processed and a subset of operation words representing the operation content; and The task syntax tree is generated by dividing the object word subset and the operation word subset into nodes and determining the association relationship between multiple nodes.

3. The method according to claim 1, wherein, The database configuration information based on the distributed database is used to validate the task syntax tree, and the validation result includes at least one of the following: Based on the object configuration information in the database configuration information, determine the validation result of whether the object attribute information used to represent the object to be processed in the task syntax tree is valid; as well as Based on the operation configuration information in the database configuration information, determine the validity of the operation attribute information in the task syntax tree used to represent the operation content.

4. The method according to claim 1, wherein, The objects to be processed in the task syntax tree include data tables; The step of generating multiple sub-task instructions for multiple sub-databases based on the task syntax tree and the database configuration information includes: Based on the table attribute information in the task syntax tree used to represent the data table, the data table is divided to obtain multiple data sub-tables; Determine multiple sub-databases that match the multiple data sub-tables from the database configuration information; and Based on the table operation information, table attribute information, and database configuration information of each of the multiple data sub-tables involved in the task syntax tree, multiple sub-task instructions are generated for the multiple sub-databases.

5. The method according to claim 4, wherein, The process involves generating multiple sub-task instructions for multiple sub-databases based on the table operation information, table attribute information, and database configuration information of each of the multiple data sub-tables involved in the task syntax tree, including: The target information is subjected to syntax recognition to obtain a syntax recognition result, wherein the target information includes at least one of the table attribute information and the table operation information; If the target information does not match the database configuration information as indicated by the syntax recognition result, the target information is converted to obtain the converted target information. The information mismatch includes at least one of the following: rule mismatch, format mismatch, and language mismatch. Based on the transformed target information, the task instruction set is generated.

6. The method according to any one of claims 1 to 5, further comprising: Receive data processing results from multiple server nodes to obtain a processing result set; If all processing results in the processing result set indicate that the data update was successful, the target data is obtained based on the processing result set. as well as If at least one processing result in the processing result set fails to update the data, a task rollback instruction is sent to multiple server nodes.

7. The method according to any one of claims 1 to 5, further comprising: If the verification result indicates that the task instruction is abnormal, the node containing the abnormality is deleted from the task syntax tree based on the verification result. or Based on the verification result, feedback information indicating an anomaly in the task instruction is sent.

8. A data processing apparatus, comprising: The analysis module is used to respond to a task instruction received for a distributed database, perform lexical analysis on the task instruction, and obtain a task syntax tree; The verification module is used to verify the task syntax tree based on the database configuration information of the distributed database and obtain the verification result. A generation module is used to generate multiple sub-task instructions for multiple sub-databases based on the task syntax tree and the database configuration information, when the verification result indicates that the task instruction is normal. The distributed database includes multiple sub-databases configured on different server nodes. as well as The sending module is used to send multiple subtask instructions to multiple server nodes respectively, so that the server nodes can update the data in the sub-database based on the subtask instructions.

9. An electronic device, comprising: One or more processors; Memory, used to store one or more programs. Wherein, when the one or more programs are executed by the one or more processors, the one or more processors implement the method of any one of claims 1 to 7.

10. A computer-readable storage medium having executable instructions stored thereon, which, when executed by a processor, cause the processor to perform the method of any one of claims 1 to 7.

11. A computer program product comprising a computer program that, when executed by a processor, implements the method according to any one of claims 1 to 7.

Images