Tracking method, device, equipment and product for storage and computing separation distributed database

Abstract

The present invention provides a tracking method, device, equipment and product for a distributed database with storage and computing separation, which can be applied to the field of big data technology. The tracking method for a distributed database with storage and computing separation includes: determining a plan type according to a distributed execution plan obtained by parsing a structured query language by a computing node of a database system, wherein the distributed execution plan represents a data access strategy corresponding to the structured query language; determining a time to issue a first tracking task according to the plan type, wherein the first tracking task is used to record multiple preset tracking indicators related to the structured query language on at least one storage node of the database system, and at least one storage node is a node participating in the execution of the distributed execution plan; issuing the first tracking task to at least one storage node based on the time to issue, so that at least one storage node executes the first tracking task and obtains data corresponding to multiple preset tracking indicators.

Description

Technical Field

[0001] The present invention relates to the field of big data technology, specifically to the field of full-link tracking of a distributed transaction library with storage and computing separation, and more specifically to a tracking method, device, equipment and product for a distributed database with storage and computing separation. Background Art

[0002] The storage nodes (Data Node, DN) and computing nodes (DataNode, DN) of the distributed database system with storage and computing separation architecture are separate and have no direct coupling with each other. In the storage and computing separation architecture, storage and computing resources can be expanded independently, which means that storage nodes and computing nodes can be flexibly expanded according to the actual needs of the structured query language (Structured Query Language, SQL). In the storage and computing integration scenario in the related technology, the tracking of the structured query language can be achieved by always turning on the tracking tools or services on each node and recording the various data generated by each node when executing the structured query language.

[0003] In the process of realizing the concept of the present invention, there are at least the following problems in the related art: in a distributed database with separated storage and computing, since the execution of structured query language may involve one or more storage nodes, the method of tracking structured query language in the related art is prone to cause additional performance burden on the storage node due to the long-term activation of tracking tools or services, thereby causing the performance of the storage node to deteriorate. Summary of the invention

[0004] In view of the above problems, the present invention provides a tracking method, device, equipment and product for a distributed database with storage and computing separation.

[0005] According to a first aspect of the present invention, a tracking method for a distributed database with storage and computation separation is provided, comprising: determining a plan type based on a distributed execution plan obtained by parsing a structured query language by a computing node of a database system, wherein the distributed execution plan represents a data access strategy corresponding to the structured query language; determining a sending time of a first tracking task based on the plan type, wherein the first tracking task is used to record a plurality of preset tracking indicators related to the structured query language on at least one storage node of the database system, and at least one of the storage nodes is a node participating in executing the distributed execution plan; sending the first tracking task to at least one of the storage nodes based on the sending time, so that at least one of the storage nodes executes the first tracking task and obtains data corresponding to the plurality of preset tracking indicators.

[0006] According to an embodiment of the present invention, the distributed execution plan generated by parsing the structured query language by the computing nodes of the database system is used to determine the plan type of the distributed execution plan, including: when the distributed execution plan represents that the structured query language is executed by a single storage node, determining that the plan type is the first type; when the distributed execution plan represents that the structured query language is executed by multiple storage nodes and there is no data interaction between the multiple storage nodes, determining that the plan type is the second type; when the distributed execution plan represents that the structured query language is executed by multiple storage nodes and there is data interaction between the multiple storage nodes, determining that the plan type is the third type; when the distributed execution plan represents that the structured query language is not all sent to the storage nodes, determining that the plan type is the fourth type.

[0007] According to an embodiment of the present invention, determining the timing of issuing the first tracking task according to the plan type includes: when the plan type is the first type or the second type, determining the timing of issuing the first tracking task as the timing when the operation node issues the structured query language.

[0008] According to an embodiment of the present invention, the timing of issuing the first tracking task is determined according to the plan type, including: when the plan type is the third type, the timing of issuing the first tracking task is determined as the timing when the computing node issues the distributed execution plan.

[0009] According to an embodiment of the present invention, determining the timing of issuing the first tracking task according to the plan type includes: when the plan type is the fourth type, determining the timing of issuing the first tracking task as the timing when the operation node issues part of the structured query language.

[0010] According to an embodiment of the present invention, the method further includes: in response to receiving a data processing request sent by a client, sending an identification application request to the global transaction management node of the database system, wherein the data processing request includes the structured query language, and the identification application request is used to apply for an identification corresponding to the data processing request; upon receiving the identification returned by the global transaction management node, generating the second tracking task based on the identification and the plurality of preset tracking indicators, wherein the second tracking task and the first tracking task are tracking tasks under the same identification, and the second tracking task is used to record the plurality of preset tracking indicators on the computing node that are related to the data processing request.

[0011] According to an embodiment of the present invention, the preset tracking indicator includes at least one of the following: operation start time, operation end time, operation object, and operation type.

[0012] The second aspect of the present invention provides a tracking device for a distributed database with storage and computing separation, including: a type determination module, used to determine the plan type according to a distributed execution plan obtained by parsing a structured query language by a computing node of the database system, wherein the distributed execution plan represents a data access strategy corresponding to the structured query language; a timing determination module, used to determine the timing of issuing a first tracking task according to the plan type, wherein the first tracking task is used to record multiple preset tracking indicators related to the structured query language on at least one storage node of the database system, and at least one of the storage nodes is a node participating in the execution of the distributed execution plan; a task issuing module, used to issue the first tracking task to at least one of the storage nodes based on the issuing timing, so that at least one of the storage nodes executes the first tracking task and obtains data corresponding to the multiple tracking indicators.

[0013] A third aspect of the present invention provides an electronic device, comprising: one or more processors; a memory for storing one or more computer programs, wherein the one or more processors execute the one or more computer programs to implement the steps of the above method.

[0014] The fourth aspect of the present invention further provides a computer program product, comprising a computer program or instructions, which implement the steps of the above method when executed by a processor.

[0015] According to an embodiment of the present invention, a distributed execution plan generated by a computing node based on a structured query language is used to determine a plan type of the distributed execution plan, and a timing for issuing a first tracking task corresponding to the structured query language is determined according to the plan type, thereby executing the issuance of the first tracking task to at least one storage node of the structured query language so that the storage node executes the first tracking task and starts recording data corresponding to a plurality of preset tracking indicators during the execution of the structured query language on the storage node, thereby reducing the occupation of storage node resources and the impact on the storage node performance caused by the storage node opening tools or tasks for detecting the above data for a long time. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The above contents and other objects, features and advantages of the present invention will become more apparent through the following description of the embodiments of the present invention with reference to the accompanying drawings, in which:

[0017] Figure 1An application scenario diagram of a tracking method, apparatus, device and product for a storage-computation separation distributed database according to an embodiment of the present invention is shown.

[0018] Figure 2 A flow chart of a tracking method for a distributed database with storage and computation separation according to an embodiment of the present invention is shown.

[0019] Figure 3 A flowchart for determining a plan type according to an embodiment of the present invention is shown.

[0020] Figure 4 A schematic diagram of executing plan types according to an embodiment of the present invention is shown.

[0021] Figure 5 A structural block diagram of a tracking device for a distributed database with storage and computation separation according to an embodiment of the present invention is shown.

[0022] Figure 6 A block diagram of an electronic device suitable for implementing a tracking method for a distributed database with storage and computation separation according to an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0023] Below, embodiments of the present invention will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the present invention. In the following detailed description, for ease of explanation, many specific details are set forth to provide a comprehensive understanding of embodiments of the present invention. However, it is apparent that one or more embodiments may also be implemented without these specific details. In addition, in the following description, descriptions of known structures and technologies are omitted to avoid unnecessary confusion of concepts of the present invention.

[0024] The terms used herein are only for describing specific embodiments and are not intended to limit the present invention. The terms "comprise", "include", etc. used herein indicate the existence of the features, steps, operations and / or components, but do not exclude the existence or addition of one or more other features, steps, operations or components.

[0025] All terms (including technical and scientific terms) used herein have the meanings commonly understood by those skilled in the art unless otherwise defined. It should be noted that the terms used herein should be interpreted as having a meaning consistent with the context of this specification and should not be interpreted in an idealized or overly rigid manner.

[0026] When using expressions such as "at least one of A, B, and C, etc.", they should generally be interpreted according to the meaning of the expression commonly understood by those skilled in the art (for example, "a system having at least one of A, B, and C" should include but is not limited to a system having A alone, B alone, C alone, A and B, A and C, B and C, and / or A, B, C, etc.).

[0027] It should be noted that in the embodiments of the present invention, certain software, components, models and other existing solutions in the industry may be mentioned, which should be regarded as exemplary. Their purpose is only to illustrate the feasibility of implementing the technical solution of the present invention, but it does not mean that the present invention has or must use this solution.

[0028] In the technical solution of the present invention, the user information (including but not limited to user personal information, user image information, user device information, such as location information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved are all information and data authorized by the user or fully authorized by all parties, and the collection, storage, use, processing, transmission, provision, disclosure and application of the relevant data comply with relevant laws, regulations and standards, take necessary confidentiality measures, do not violate public order and good morals, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0029] In the scenario of using personal information for automated decision-making, the methods, devices, and systems provided by the embodiments of the present invention provide users with corresponding operation portals for users to choose to agree or reject the automated decision-making results; if the user chooses to reject, the expert decision-making process will be entered. The expression "automated decision-making" here refers to the activity of automatically analyzing and evaluating an individual's behavioral habits, interests and hobbies, or economic, health, credit status, etc. through computer programs, and making decisions. The expression "expert decision-making" here refers to the activity of making decisions by people who specialize in a certain field, have specialized experience, knowledge and skills, and have reached a certain level of professionalism.

[0030] According to an embodiment of the present invention, in a distributed database system with a storage-computation separation architecture, storage nodes and computing nodes are separated, which facilitates flexible expansion of storage nodes and computing nodes required for tasks according to actual conditions. The tracking method of the storage-computation separation distributed transaction library is responsible for tracking the execution process of structured query language in computing nodes, storage nodes and global transaction management nodes (Global Transaction Manager, GTM), and counting the time consumption of SQL processes between nodes and within nodes.

[0031] The SQL execution process includes: SQL parsing, snapshot acquisition, semantic analysis, query rewriting, execution plan generation, execution plan execution, operator execution, storage engine, etc. SQL tracking is to track the above-mentioned SQL execution process and its sub-processes. Due to the separation of storage and computing in a distributed database, the execution of an SQL statement may not involve all storage nodes. The tools or services used for tracking in the tracking method in the related art will remain open to track multiple indicator data belonging to the same structured query language.

[0032] The tracking methods in the related art are not suitable for tracking in complex scenarios where storage and computing are separated. That is, for idle storage nodes, there is no need to start tracking tasks. The tracking methods in the related art are prone to causing additional performance burdens on the storage nodes due to the long-term activation of tracking tools or services. Excessive tracking data may lead to a decline in the performance of the storage nodes.

[0033] An embodiment of the present invention provides a tracking method for a distributed database with storage and computing separation, comprising: determining a plan type according to a distributed execution plan obtained by parsing a structured query language by a computing node of a database system, wherein the distributed execution plan represents a data access strategy corresponding to the structured query language; determining a sending time of a first tracking task according to the plan type, wherein the first tracking task is used to record a plurality of preset tracking indicators related to the structured query language on at least one storage node of the database system, and at least one storage node is a node participating in executing the distributed execution plan; sending the first tracking task to at least one storage node based on the sending time, so that at least one storage node executes the first tracking task and obtains data corresponding to the plurality of preset tracking indicators.

[0034] The tracking method of the storage-computation separation distributed database of the present invention can be applied to complex storage-computation separation scenarios, track the interaction process of SQL between the computing node and multiple storage nodes and the global transaction management node, record the execution process and time consumption of SQL in each computing node, storage node, global transaction management node and their internals, and facilitate maintenance personnel to quickly locate and analyze problems such as slow SQL execution.

[0035] Figure 1 An application scenario diagram of a tracking method, apparatus, device and product for a storage-computation separation distributed database according to an embodiment of the present invention is shown.

[0036] like Figure 1As shown, the application scenario 100 according to this embodiment may include a first terminal device 101, a second terminal device 102, a third terminal device 103, a network 104, a computing node 105, and a storage node 106. The network 104 is used to provide a medium for communication links between the first terminal device 101, the second terminal device 102, the third terminal device 103 and the computing node 105, and between the computing node 105 and the storage node 106. The network 104 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0037] The user can use the first terminal device 101, the second terminal device 102, and the third terminal device 103 to interact with the computing node 105 through the network 104 to receive or send messages, etc. Various communication client applications can be installed on the first terminal device 101, the second terminal device 102, and the third terminal device 103, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social platform software, etc. (only for example).

[0038] The first terminal device 101, the second terminal device 102, and the third terminal device 103 may be various electronic devices having display screens and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, desktop computers, and the like.

[0039] The computing node 105 is a component responsible for processing data and performing computing tasks. In a distributed database system with storage and computing separation, this may include processors, memory and other computing resources on the database server, which perform tasks such as query processing, transaction management, and index maintenance. The computing node 105 can be a server running an instance of a database management system. The computing node 105 obtains data from the storage node, performs operations requested by the user, and returns the results to the user.

[0040] Storage node 106 is a component responsible for persistent storage of data. In a distributed database system with storage and computing separation, storage node 106 is a server used to store data and manage the physical storage of data. Storage node 106 may include a hard disk drive, a solid-state drive, or other forms of persistent storage, as well as related software to manage the storage, retrieval, and backup of data. Computing node 105 and storage node 106 belong to the database system.

[0041] It should be noted that the tracking method of the distributed database with storage and computing separation provided in the embodiment of the present invention can be executed in the computing node 105 and the storage node 106. Accordingly, the tracking device of the distributed database with storage and computing separation provided in the embodiment of the present invention can generally be set in the computing node 105 and the storage node 106.

[0042] The tracking method of the distributed database with storage and computing separation provided in the embodiment of the present invention may also be executed by a server or server cluster that is different from the computing node 105 and the storage node 106 and can communicate with the first terminal device 101, the second terminal device 102, the third terminal device 103 and / or the computing node 105 and the storage node 106. Correspondingly, the tracking device of the distributed database with storage and computing separation provided in the embodiment of the present invention may also be set in a server or server cluster that is different from the computing node 105 and the storage node 106 and can communicate with the first terminal device 101, the second terminal device 102, the third terminal device 103 and / or the computing node 105 and the storage node 106.

[0043] It should be understood that Figure 1 The number of terminal devices, networks, storage nodes, and computing nodes in the embodiment is only for illustration purposes. Any number of terminal devices, networks, storage nodes, and computing nodes may be provided as required.

[0044] The following will be based on Figure 1 The scene described by Figure 2~Figure 4 The tracking method of the storage-computation separation distributed database according to an embodiment of the present invention is described in detail.

[0045] Figure 2 A flow chart of a tracking method for a distributed database with storage and computation separation according to an embodiment of the present invention is shown.

[0046] like Figure 2 As shown, the tracking method of the storage-computation separation distributed database of this embodiment includes operations S210 to S230.

[0047] In operation S210 , a plan type is determined according to a distributed execution plan obtained by parsing a structured query language by a computing node of a database system, wherein the distributed execution plan represents a data access strategy corresponding to the structured query language.

[0048] In operation S220, a timing for issuing a first tracking task is determined based on a plan type, wherein the first tracking task is used to record a plurality of preset tracking indicators related to a structured query language on at least one storage node of a database system, and at least one storage node is a node participating in executing a distributed execution plan.

[0049] In operation S230, the first tracking task is sent to at least one storage node based on the sending opportunity, so that the at least one storage node executes the first tracking task and obtains data corresponding to a plurality of preset tracking indicators.

[0050] According to an embodiment of the present invention, a database system includes multiple storage nodes and multiple computing nodes. A user initiates a data processing request to the database system, and the computing node processes the structured query language included in the data processing request. The structured query language is a statement used to operate the database, for example, the structured query language used to insert data: "INSERT INTO Employees (EmployeeID, FirstName, LastName, Department) VALUES (1, 'John', 'Doe', 'Finance')".

[0051] According to an embodiment of the present invention, a computing node may include a parser and an optimizer. The parser performs lexical analysis, grammatical analysis, semantic analysis, etc. on the structured query language to obtain a logical execution plan corresponding to the structured query language. The logical execution plan describes the algorithm or operator used by the database engine to access tables, use indexes, connect tables, and perform other operations to obtain query results.

[0052] According to an embodiment of the present invention, the operation node optimizer performs query rewriting, query optimization and other operations on the obtained logical execution plan to obtain a distributed execution plan for execution. The optimizer is used to estimate the cost of the obtained logical plan and select the "optimal" execution plan as the physical execution plan, i.e., the distributed execution plan. The logical execution plan is an algebraic expression of the query, which describes the logical structure and operation sequence of the query, but does not involve specific physical implementation details. The distributed execution plan contains operators, i.e., the operations to be performed, result sets, i.e., the estimated number of rows, and specific information such as the addresses of accessed tables and nodes.

[0053] According to an embodiment of the present invention, a distributed execution plan converts a logical query into an execution plan of a series of physical operations. The distributed execution plan vertically splits the optimized logical execution plan according to the data distribution, and converts the corresponding logical plan into specific operators, which can be connected through data flows. The topological structure of the entire distributed execution plan is a directed acyclic graph.

[0054] According to an embodiment of the present invention, the distributed execution plan is for the physical structure of the database, which determines the database access method, data storage method, data transmission method, etc. It can be understood as the storage nodes involved in storing or accessing data, and the method of storing or accessing data in the storage nodes. The computing node of the database system parses the received structured query language to obtain a distributed execution plan corresponding to the structured query language.

[0055] According to an embodiment of the present invention, after obtaining a distributed execution plan, the type of the distributed execution plan, i.e., the plan type, is determined. The first tracking task is used to track at least one storage node participating in the execution of the structured query language to obtain data of preset tracking indicators corresponding to each of a plurality of operations on the storage node, wherein the plurality of operations are the plurality of operations generated in the distributed execution plan, wherein the plurality of operations may include reading, writing, updating, etc. of data.

[0056] According to an embodiment of the present invention, the timing of issuing the first tracking task to each storage node is determined based on the plan type to which the distributed execution plan belongs, so that before the storage node receives the execution plan issued by the computing node, the storage node starts the tracking service or component of the storage node based on the first tracking task issued by the computing node, thereby recording the data corresponding to the preset tracking indicators of each of the multiple operations on the current storage node.

[0057] According to an embodiment of the present invention, the issuance of the first tracking task can be achieved by embedding instructions or functions in a distributed execution plan. For example, a function or instruction for issuing the first tracking task is predefined, and the function or instruction is written into the distributed execution plan, so that in the process of processing the structured query language according to the distributed execution plan, the first tracking task is issued according to the issuance timing. By issuing the first tracking task, the tracking task identifier of at least one storage node is kept consistent with that of the computing node, and the first tracking task is completed by the storage node, wherein the tracking task identifier is used to distinguish tracking tasks of different structured query languages. Data corresponding to a plurality of preset tracking indicators can be stored in each storage node, that is, the tracking results are stored in each storage node.

[0058] According to an embodiment of the present invention, a distributed execution plan generated by a computing node based on a structured query language is used to determine a plan type of the distributed execution plan, and a timing for issuing a first tracking task corresponding to the structured query language is determined according to the plan type, thereby issuing the first tracking task to at least one storage node that executes the structured query language, so that the storage node executes the first tracking task and starts recording data corresponding to a plurality of preset tracking indicators during the execution of the structured query language on the storage node, thereby reducing the occupation of storage node resources and the impact on storage node performance caused by the storage node opening tools or tasks for detecting the above data for a long time.

[0059] According to an embodiment of the present invention, based on a distributed execution plan generated by parsing a structured query language by a computing node of a database system, a plan type of the distributed execution plan is determined, including: when the distributed execution plan represents that the structured query language is executed by a single storage node, the plan type is determined to be a first type; when the distributed execution plan represents that the structured query language is executed by multiple storage nodes and there is no data interaction between the multiple storage nodes, the plan type is determined to be a second type; when the distributed execution plan represents that the structured query language is executed by multiple storage nodes and there is data interaction between the multiple storage nodes, the plan type is determined to be a third type; when the distributed execution plan represents that the structured query language is not fully sent to the storage node, the plan type is determined to be a fourth type.

[0060] According to an embodiment of the present invention, the plan types of the distributed execution plan may include a LightProxy type distributed execution plan, a FastQueryShipping type distributed execution plan, a Stream type distributed execution plan, and a Remote-Query type distributed execution plan.

[0061] According to an embodiment of the present invention, identifiers of storage nodes included in a distributed execution plan (e.g., address information of the storage nodes) are collected, and the number of identifiers of different storage nodes in the distributed execution plan is determined, thereby determining the number of storage nodes in the distributed execution plan. In the case where the distributed execution plan represents that a structured query language is executed by a single storage node, the type of the distributed execution plan is determined to be the first type, i.e., the LightProxy type.

[0062] According to an embodiment of the present invention, when a distributed execution plan represents that a structured query language is executed by multiple storage nodes, it is possible to determine whether there is interaction between multiple storage nodes that execute the structured query language in the distributed execution plan by querying whether there is an operator (for example, a Stream operator, etc.) in the distributed execution plan for realizing interaction between storage nodes.

[0063] According to an embodiment of the present invention, when a distributed execution plan represents that a structured query language is executed by multiple storage nodes and there is no data interaction between the multiple storage nodes, the plan type is determined to be the second type, ie, the Stream type.

[0064] According to an embodiment of the present invention, when the distributed execution plan represents that the structured query language is executed by multiple storage nodes and there is data interaction between the multiple storage nodes, the plan type is determined to be the third type, ie, the FastQueryShipping type.

[0065] According to an embodiment of the present invention, by querying the distributed execution plan, when the distributed execution plan includes operators that cannot be pushed down, it is characterized that the query corresponding to the structured query language is not fully sent to the storage node for execution, and the plan type is determined to be the fourth type, i.e., the Remote-Query type. Operators that cannot be pushed down include operators such as LIMIT and DISTINCT.

[0066] According to an embodiment of the present invention, the characteristics of the distributed execution plan are determined by querying the content involved in the distributed execution plan. The characteristics of the distributed execution plan are matched with the respective plan execution characteristics of the first type, the second type, the third type and the fourth type to determine the plan type of the distributed execution plan, so as to facilitate the subsequent determination of the timing of issuing the first tracking task according to the respective execution processes of the first type, the second type, the third type and the fourth type. The accuracy of issuing the first tracking task is improved, thereby reducing the impact of the storage node on its performance due to the execution of the tracking task.

[0067] Figure 3 A flowchart for determining a plan type according to an embodiment of the present invention is shown.

[0068] like Figure 3 As shown, the process of determining the plan type in this embodiment includes operations S301 to S308.

[0069] In operation S301 , a distributed execution plan is received.

[0070] In operation S302, it is determined whether all query operations corresponding to the structured query language in the distributed execution plan are sent to the storage node for execution.

[0071] When all query operations corresponding to the structured query language in the distributed execution plan are sent to the storage node for execution, operation S303 is performed; otherwise, operation S306 is performed.

[0072] In operation S303, it is determined whether a plurality of storage nodes are used in the distributed execution plan.

[0073] When multiple storage nodes are used in the distributed execution plan, operation S304 is performed; otherwise, operation S307 is performed.

[0074] In operation S304, it is determined whether there is data interaction between multiple storage nodes used in the distributed execution plan.

[0075] In the case where data interaction exists between multiple storage nodes used in the distributed execution plan, operation S305 is performed; otherwise, operation S308 is performed.

[0076] In operation S305, the plan type is determined to be the third type.

[0077] In operation S306, the plan type is determined to be the fourth type.

[0078] In operation S307, the plan type is determined to be the first type.

[0079] In operation S308, the plan type is determined to be the second type.

[0080] According to an embodiment of the present invention, whether a distributed execution plan belongs to the fourth type, the first type, the second type and the third type is judged sequentially according to the characteristics of different plan types, so that when the plan type of the distributed execution plan is the Remote-Query type, the execution of subsequent programs is given priority, thereby reducing the time waste caused by querying or other operations on the distributed execution plan, thereby saving computing resources and improving the processing efficiency of the algorithm.

[0081] Figure 4 A schematic diagram of executing plan types according to an embodiment of the present invention is shown.

[0082] like Figure 4 As shown, the computing node parses the structured query language and determines a distributed execution plan corresponding to the structured query language. On this basis, the following takes the first type of distributed execution plan 410, the second type of distributed execution plan 420, the third type of distributed execution plan 430 and the fourth type of distributed execution plan 440 as examples to illustrate the tracking process for different plan types.

[0083] For the first type of distributed execution plan 410, the computing node sends the structured query language to a single storage node. In this case, the sending time of the first tracking task can also be determined as the time when the computing node sends the structured query language. Thus, for the first type of distributed execution plan 410, the structured query language executed on a single storage node can be tracked, and data corresponding to multiple preset tracking indicators during the execution of the structured query language can be recorded.

[0084] For the second type of distributed execution plan 420, the structured query language is issued to multiple storage nodes. In this case, the timing of issuing the first tracking task can also be determined as the timing of the computing node issuing the structured query language. Thus, for the second type of distributed execution plan 420, data corresponding to multiple preset tracking indicators during the execution of the structured query language can be recorded on each storage node. It should be noted here that the number of storage nodes in the figure is only schematic.

[0085] For the third type of distributed execution plan 430, the distributed execution plan generated at the computing node is sent to multiple storage nodes. In this case, the timing of sending the first tracking task can also be determined as the timing when the computing node sends the distributed execution plan. Thus, for the third type of distributed execution plan 430, data corresponding to multiple preset tracking indicators during the execution of the structured query language can be recorded on each storage node, including multiple operation processes of the storage node executing the structured query language, and records generated by data interaction between storage nodes. It should be noted here that the number of storage nodes in the figure is only schematic.

[0086] For the fourth type of distributed execution plan 440, the operation of data is mainly concentrated in the operation node, and the operation node sends part of the structured query language to multiple storage nodes, pulls data from the storage nodes, and then calculates it on the operation node. In this case, the timing of sending the first tracking task can also be determined as the timing when the operation node sends part of the structured query language. Therefore, for the fourth type of distributed execution plan 420, the data corresponding to multiple preset tracking indicators during the execution of part of the structured query language can be recorded on one or more storage nodes, and the subsequent operations after the storage node returns the data are recorded on the operation node. It should be noted here that the number of storage nodes in the figure is only schematic.

[0087] According to an embodiment of the present invention, determining the sending time of the first tracking task according to the plan type includes: when the plan type is the first type or the second type, determining the sending time of the first tracking task as the time when the computing node sends the structured query language.

[0088] According to an embodiment of the present invention, for a first type or second type of distributed execution plan, the timing of sending a structured query language to a storage node in the distributed execution plan is used as the timing of sending a first tracking task to track the structured query language executed on the storage node and record data corresponding to multiple preset tracking indicators during the execution of the structured query language.

[0089] According to an embodiment of the present invention, determining the sending time of the first tracking task according to the plan type includes: when the plan type is the third type, determining the sending time of the first tracking task as the time when the computing node sends the distributed execution plan.

[0090] According to an embodiment of the present invention, for the third type of distributed execution plan, the execution or code for sending the first tracking task can be written into the distributed execution plan, specifically, it can be written before the computing node executes and sends the distributed execution plan.

[0091] According to an embodiment of the present invention, each storage node completes the recording of data corresponding to multiple preset tracking indicators based on its own first tracking task, including the data interaction process between the current storage node and other storage nodes, and saves the tracking result of the current storage node on the current storage node.

[0092] According to an embodiment of the present invention, determining the sending time of the first tracking task according to the plan type includes: when the plan type is the fourth type, determining the sending time of the first tracking task as the time when the computing node sends part of the structured query language.

[0093] According to an embodiment of the present invention, when a computing node sends part of a structured query language to a storage node, a first tracking task is sent to one or more storage nodes that send the structured query language to record data corresponding to a plurality of preset tracking indicators during the execution of the part of the structured query language.

[0094] According to the embodiment of the present invention, the timing of issuing the first tracking task is determined according to the plan type, and the resource occupation of the storage node by the tracking task is reduced on the premise that the data to be tracked is recorded in time.

[0095] According to an embodiment of the present invention, the tracking method of a distributed database with storage and computing separation also includes: in response to receiving a data processing request sent by a client, sending an identification application request to a global transaction management node of the database system, wherein the data processing request includes a structured query language, and the identification application request is used to apply for an identification corresponding to the data processing request; upon receiving an identification returned by the global transaction management node, generating a second tracking task based on the identification and multiple preset tracking indicators, wherein the second tracking task and the first tracking task are tracking tasks under the same identification, and the second tracking task is used to record multiple preset tracking indicators on the computing node that are related to the data processing request.

[0096] According to an embodiment of the present invention, the data processing request includes a structured query language, which may refer to a request for initiating processing of the structured query language to a computing node. In response to the data processing request sent by the client, an identification application request is sent to a global transaction management node of the database system, wherein the identification may be a tracking task number corresponding to the structured query language, which is used to identify the tracking task of the entire execution process of the structured query language, and the obtained tracking result, that is, data corresponding to multiple preset tracking indicators.

[0097] According to an embodiment of the present invention, the second tracking task is used to record data corresponding to multiple preset tracking indicators during the operation of executing structured query language parsing, obtaining snapshots, query rewriting, etc. in the computing node. The second tracking task includes an identifier and multiple preset tracking indicators that need to be tracked. By executing the second tracking task, the tracking of multiple preset tracking indicators of the computing node is enabled.

[0098] According to an embodiment of the present invention, the second tracking task and the first tracking task are tracking tasks under the same tracking task number. When the computing node sends the first tracking task to the storage node, the storage node can start a local tracking process based on the identifier in the first tracking task to identify data corresponding to multiple preset tracking indicators.

[0099] According to an embodiment of the present invention, an identifier is applied to the global transaction management node of the database system, and the identifier of the storage node is kept consistent with the identifier of the computing node by issuing the first tracking task, which facilitates the subsequent retrieval of data corresponding to multiple preset tracking indicators. The staff can obtain and track the data obtained by the structured query language through the identifier.

[0100] According to an embodiment of the present invention, the storage node receiving the first tracking task can generate a lower-level identifier corresponding to the executed operation based on the identifier to characterize the link relationship between multiple groups of tracked data. For example, the first operation or operator executed by the storage node can be identified as spanID = 0.1.0. The executed operation can be an operation in a distributed execution plan generated by parsing the structured query language, and specifically, can be an operator or function in the distributed execution plan.

[0101] According to an embodiment of the present invention, the preset tracking indicator includes at least one of the following: operation start time, operation end time, operation object, and operation type.

[0102] According to an embodiment of the present invention, the operation start time represents the time node when the operation in the distributed execution plan starts to be executed, and the operation end time represents the time node when the operation in the distributed execution plan is completed. The operation object represents the address or identifier of the storage node where the operation is executed. The operation type represents the type of the current operation, which may include a connection type, a calculation type, etc., wherein the calculation type may include, for example, a source operator, etc.

[0103] According to an embodiment of the present invention, after each storage node and computing node completes the tracking, the log is retained in each node. If you need to view the tracking result of a certain SQL, the query command needs to summarize the tracking result from each storage node and computing node into a tree structure according to the tracking task number and present it to the operation and maintenance personnel. The parent-child and brother relationships of the nodes in the tree structure reflect the link relationship of each process or operation in the processing process.

[0104] According to an embodiment of the present invention, in addition to supporting the basic tracking task of storage and computing separation, the tracking task also supports tracking by unique_sql_id, that is, normalized SQL_ID, tracking by application name, tracking by login database account, tracking by session ID, and tracking by SQL execution duration.

[0105] Based on the above-mentioned tracking method of the distributed database with storage and computing separation, the present invention also provides a tracking device of the distributed database with storage and computing separation. Figure 5 The device is described in detail.

[0106] Figure 5 A structural block diagram of a tracking device for a distributed database with storage and computation separation according to an embodiment of the present invention is shown.

[0107] like Figure 5 As shown, the tracking device 500 of the storage-computation separation distributed database of this embodiment includes a type determination module 510 , a timing determination module 520 and a task issuing module 530 .

[0108] The type determination module 510 is used to determine the plan type according to the distributed execution plan obtained by parsing the structured query language by the computing node of the database system, wherein the distributed execution plan represents the data access strategy corresponding to the structured query language. In one embodiment, the type determination module 510 can be used to perform the operation S210 described above, which will not be described in detail here.

[0109] The timing determination module 520 is used to determine the timing of issuing the first tracking task according to the plan type, wherein the first tracking task is used to record multiple preset tracking indicators related to the structured query language on at least one storage node of the database system, and at least one storage node is a node participating in executing the distributed execution plan. In one embodiment, the timing determination module 520 can be used to perform the operation S220 described above, which will not be repeated here.

[0110] The task issuing module 530 is used to issue the first tracking task to at least one storage node based on the issuing timing, so that at least one storage node executes the first tracking task and obtains data corresponding to multiple tracking indicators. In one embodiment, the task issuing module 530 can be used to execute the operation S230 described above, which will not be repeated here.

[0111] According to an embodiment of the present invention, the type determination module 510 includes: a first determination unit, a second determination unit, a third determination unit, and a fourth determination unit.

[0112] The first determining unit is configured to determine that the plan type is a first type when the distributed execution plan represents that the structured query language is executed by a single storage node.

[0113] The second determining unit is configured to determine that the plan type is a second type when the distributed execution plan represents that the structured query language is executed by multiple storage nodes and there is no data interaction between the multiple storage nodes.

[0114] The third determining unit is configured to determine that the plan type is a third type when the distributed execution plan represents that the structured query language is executed by multiple storage nodes and there is data interaction between the multiple storage nodes.

[0115] The fourth determining unit is configured to determine that the plan type is a fourth type when the structured query language representing the distributed execution plan is not completely delivered to the storage node.

[0116] According to an embodiment of the present invention, the timing determination module 520 includes: a fifth determination unit.

[0117] The fifth determining unit is configured to determine, when the plan type is the first type or the second type, the timing of issuing the first tracking task as the timing of the computing node issuing the structured query language.

[0118] According to an embodiment of the present invention, the timing determination module 520 includes: a sixth determination unit.

[0119] The sixth determining unit is used to determine the sending timing of the first tracking task as the timing when the computing node sends the distributed execution plan when the plan type is the third type.

[0120] According to an embodiment of the present invention, the timing determination module 520 includes: a seventh determination unit.

[0121] The seventh determining unit is configured to determine, when the plan type is the fourth type, the timing of issuing the first tracking task as the timing of the computing node issuing a portion of the structured query language.

[0122] According to an embodiment of the present invention, the tracking device 500 of the storage-computation separation distributed database also includes: a request response module and a task generation module.

[0123] The request response module is used to send an identification application request to the global transaction management node of the database system in response to receiving a data processing request sent by a client, wherein the data processing request includes a structured query language, and the identification application request is used to apply for an identification corresponding to the data processing request.

[0124] A task generation module is used to generate a second tracking task based on the identifier and multiple preset tracking indicators when receiving the identifier returned by the global transaction management node, wherein the second tracking task and the first tracking task are tracking tasks under the same identifier, and the second tracking task is used to record multiple preset tracking indicators related to the data processing request on the computing node.

[0125] According to an embodiment of the present invention, the preset tracking indicator includes at least one of the following: operation start time, operation end time, operation object, and operation type.

[0126] According to an embodiment of the present invention, any multiple modules among the type determination module 510, the timing determination module 520 and the task delivery module 530 may be combined into one module for implementation, or any one of the modules may be split into multiple modules. Alternatively, at least part of the functions of one or more of these modules may be combined with at least part of the functions of other modules and implemented in one module.

[0127] According to an embodiment of the present invention, at least one of the type determination module 510, the timing determination module 520 and the task delivery module 530 may be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on a chip, a system on a substrate, a system on a package, an application-specific integrated circuit (ASIC), or may be implemented by hardware or firmware such as any other reasonable way of integrating or packaging the circuit, or by any one of the three implementation methods of software, hardware and firmware or by a proper combination of any of them. Alternatively, at least one of the type determination module 510, the timing determination module 520 and the task delivery module 530 may be at least partially implemented as a computer program module, and when the computer program module is run, the corresponding function may be executed.

[0128] Figure 6 A block diagram of an electronic device suitable for implementing a tracking method for a distributed database with storage and computation separation according to an embodiment of the present invention is shown.

[0129] like Figure 6As shown, the electronic device 600 according to an embodiment of the present invention includes a processor 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage part 608 to a random access memory (RAM) 603. The processor 601 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or a related chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 601 may also include an onboard memory for caching purposes. The processor 601 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present invention.

[0130] In RAM 603, various programs and data required for the operation of electronic device 600 are stored. Processor 601, ROM 602 and RAM 603 are connected to each other via bus 604. Processor 601 performs various operations of the method flow according to the embodiment of the present invention by executing the program in ROM 602 and / or RAM 603. It should be noted that the program can also be stored in one or more memories other than ROM 602 and RAM 603. Processor 601 can also perform various operations of the method flow according to the embodiment of the present invention by executing the program stored in the one or more memories.

[0131] According to an embodiment of the present invention, the electronic device 600 may further include an input / output (I / O) interface 605, which is also connected to the bus 604. The electronic device 600 may further include one or more of the following components connected to the input / output (I / O) interface 605: an input portion 606 including a keyboard, a mouse, etc.; an output portion 607 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage portion 608 including a hard disk, etc.; and a communication portion 609 including a network interface card such as a LAN card, a modem, etc. The communication portion 609 performs communication processing via a network such as the Internet. A drive 610 is also connected to the input / output (I / O) interface 605 as needed. A removable medium 611, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 610 as needed, so that a computer program read therefrom is installed into the storage portion 608 as needed.

[0132] The present invention also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiment; or may exist independently without being assembled into the device / apparatus / system. The above computer-readable storage medium carries one or more programs, and when the above one or more programs are executed, the method according to the embodiment of the present invention is implemented.

[0133] According to an embodiment of the present invention, the computer-readable storage medium may be a non-volatile computer-readable storage medium, for example, it may include but is not limited to: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present invention, the computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, an apparatus or a device. For example, according to an embodiment of the present invention, the computer-readable storage medium may include the ROM 602 and / or RAM 603 described above and / or one or more memories other than ROM 602 and RAM 603.

[0134] The embodiment of the present invention also includes a computer program product, which includes a computer program, and the computer program contains program code for executing the method shown in the flowchart. When the computer program product is run in a computer system, the program code is used to enable the computer system to implement the tracking method of the storage-computation separation distributed database provided by the embodiment of the present invention.

[0135] When the computer program is executed by the processor 601 , the above functions defined in the system / apparatus of the embodiment of the present invention are performed.

[0136] In one embodiment, the computer program may rely on tangible storage media such as optical storage devices, magnetic storage devices, etc. In another embodiment, the computer program may also be transmitted and distributed in the form of signals on a network medium, and downloaded and installed through the communication part 609, and / or installed from a removable medium 611. The program code contained in the computer program may be transmitted using any appropriate network medium, including but not limited to: wireless, wired, etc., or any suitable combination of the above.

[0137] In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 609, and / or installed from the removable medium 611. When the computer program is executed by the processor 601, the above functions defined in the system of the embodiment of the present invention are performed. According to the embodiment of the present invention, the system, device, means, module, unit, etc. described above can be implemented by a computer program module.

[0138] According to an embodiment of the present invention, the program code for executing the computer program provided by the embodiment of the present invention can be written in any combination of one or more programming languages. Specifically, these computing programs can be implemented using high-level process and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, Java, C++, python, "C" language or similar programming languages. The program code can be executed entirely on the user computing device, partially on the user device, partially on the remote computing device, or entirely on the remote computing device or server. In the case of a remote computing device, the remote computing device can be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computing device (e.g., using an Internet service provider to connect through the Internet).

[0139] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present invention. In this regard, each box in the flow chart or block diagram can represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0140] It will be appreciated by those skilled in the art that the features described in the various embodiments of the present invention may be combined and / or combined in various ways, even if such combinations or combinations are not explicitly described in the present invention. In particular, without departing from the spirit and teachings of the present invention, the features described in the various embodiments of the present invention may be combined and / or combined in various ways. All of these combinations and / or combinations fall within the scope of the present invention.

[0141] The embodiments of the present invention are described above. However, these embodiments are only for the purpose of illustration, and are not intended to limit the scope of the present invention. Although each embodiment is described above, it does not mean that the measures in each embodiment cannot be used in combination advantageously. Without departing from the scope of the present invention, those skilled in the art may make various substitutions and modifications, which should all fall within the scope of the present invention.

Claims

1. A tracking method for a distributed database with storage and computing separation, characterized in that: The method comprises: Determining a plan type according to a distributed execution plan obtained by parsing a structured query language by a computing node of a database system, wherein the distributed execution plan represents a data access strategy corresponding to the structured query language; Determining, according to the plan type, a time to issue a first tracking task, wherein the first tracking task is used to record a plurality of preset tracking indicators related to the structured query language on at least one storage node of the database system, and at least one of the storage nodes is a node participating in executing the distributed execution plan; Sending the first tracking task to at least one of the storage nodes based on the sending timing, so that at least one of the storage nodes executes the first tracking task and obtains data corresponding to the plurality of preset tracking indicators; The step of determining the plan type of the distributed execution plan according to the distributed execution plan generated by the computing node of the database system parsing the structured query language includes: In a case where the distributed execution plan represents that the structured query language is executed by a single storage node, determining that the plan type is a first type; In a case where the distributed execution plan represents that the structured query language is executed by a plurality of the storage nodes and there is no data interaction between the plurality of the storage nodes, determining that the plan type is the second type; In a case where the distributed execution plan represents that the structured query language is executed by a plurality of the storage nodes and there is data interaction between the plurality of storage nodes, determining that the plan type is the third type; When the distributed execution plan indicates that the structured query language has not been fully delivered to the storage node, the plan type is determined to be the fourth type.

2. The method according to claim 1, characterized in that The step of determining the timing of issuing the first tracking task according to the plan type includes: When the plan type is the first type or the second type, the timing of issuing the first tracking task is determined as the timing when the computing node issues the structured query language.

3. The method according to claim 1, characterized in that: The step of determining the timing of issuing the first tracking task according to the plan type includes: When the plan type is the third type, the timing of issuing the first tracking task is determined as the timing when the computing node issues the distributed execution plan.

4. The method according to claim 1, characterized in that: The step of determining the timing of issuing the first tracking task according to the plan type includes: When the plan type is the fourth type, the timing of issuing the first tracking task is determined to be the timing when the computing node issues part of the structured query language.

5. The method according to claim 1, characterized in that The method further comprises: In response to receiving a data processing request sent by a client, sending an identification application request to a global transaction management node of the database system, wherein the data processing request includes the structured query language, and the identification application request is used to apply for an identification corresponding to the data processing request; Upon receiving the identifier returned by the global transaction management node, a second tracking task is generated based on the identifier and the multiple preset tracking indicators, wherein the second tracking task and the first tracking task are tracking tasks under the same identifier, and the second tracking task is used to record the multiple preset tracking indicators related to the data processing request on the computing node.

6. The method according to any one of claims 1 to 5, characterized in that: The preset tracking indicator includes at least one of the following: operation start time, operation end time, operation object, and operation type.

7. A tracking device for a distributed database with storage and computation separation, characterized in that: The device comprises: A type determination module, configured to determine a plan type according to a distributed execution plan obtained by parsing a structured query language by a computing node of a database system, wherein the distributed execution plan represents a data access strategy corresponding to the structured query language; a timing determination module, configured to determine a timing for issuing a first tracking task according to the plan type, wherein the first tracking task is configured to record a plurality of preset tracking indicators related to the structured query language on at least one storage node of the database system, and at least one of the storage nodes is a node participating in executing the distributed execution plan; a task issuing module, configured to issue the first tracking task to at least one of the storage nodes based on the issuing timing, so that at least one of the storage nodes executes the first tracking task and obtains data corresponding to the plurality of tracking indicators; The type determination module comprises: A first determining unit, configured to determine that the plan type is a first type when the distributed execution plan represents that the structured query language is executed by a single storage node; A second determining unit, configured to determine that the plan type is a second type when the distributed execution plan represents that the structured query language is executed by a plurality of the storage nodes and there is no data interaction between the plurality of the storage nodes; A third determining unit, configured to determine that the plan type is a third type when the distributed execution plan represents that the structured query language is executed by a plurality of the storage nodes and there is data interaction between the plurality of the storage nodes; The fourth determining unit is configured to determine that the plan type is a fourth type when the distributed execution plan indicates that the structured query language has not been fully delivered to the storage node.

8. An electronic device comprising: one or more processors; a memory for storing one or more computer programs, It is characterized in that the one or more processors execute the one or more computer programs to implement the steps of the method according to any one of claims 1 to 6.

9. A computer program product comprising a computer program or instructions, characterized in that When the computer program or instruction is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images