A method and device for federal processing of data

Abstract

The present application provides a method and device for federated processing of data. Through the present application, at least two execution engines that support different operators are combined to jointly process data in a database, so that one of the at least two execution engines can be supplemented by the other execution engines, that is, when one of the execution engines supports limited operators, the operators supported by the other execution engines can provide supplements, so that data can be processed in the database in more processing methods, and federated processing of data in the database (such as federated query, etc.) can be realized, and more database functions can be enjoyed when one of the execution engines supports limited operators, so as to improve the user experience of using the database.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a method and device for federal processing of data. Background Art

[0002] With the rapid development of technology, databases are used more and more widely. Users can store data in databases and then process the data in the database (such as merging, filtering, mapping, and aggregation, etc.) to meet their needs.

[0003] In order to support users to process data in the database, developers can develop an execution engine, so that users can process data in the database based on the execution engine (such as a calculation engine, etc.).

[0004] However, in the process of developing an execution engine, it is often impossible to perfect the functions of the execution engine at one time. For example, there are theoretically many ways to process data in the database based on the execution engine, and each processing method requires the use of an operator, that is, one processing method can be implemented using an operator. The operators supported by the initially developed execution engine may be incomplete. In this way, users can only process data in the database in limited processing methods based on the execution engine, which may not meet the actual needs of users. The developer will continue to improve the functions of the execution engine in the future, for example, developing more operators corresponding to the processing methods of the execution engine so that users can gradually process data in the database in more processing methods.

[0005] However, as developers continue to improve the execution engine's functions, users can only process data in the database in limited ways based on the execution engine and cannot enjoy more database functions, which will affect users' experience of using the database. Summary of the invention

[0006] The present application shows a method and device for federated processing of data.

[0007] In a first aspect, the present application shows a method for federated processing of data, the method comprising:

[0008] Acquire a data processing statement, wherein the data processing statement includes a plurality of original instructions, and each original instruction includes an operator and an operation object;

[0009] Parsing the data processing statement to obtain each original instruction in the data processing statement, and an operation object and an operator in each original instruction;

[0010] Generate a data processing plan for running on at least two execution engines according to the operation objects and operators in each original instruction, wherein the operators supported by the at least two execution engines are not all the same, and the data processing plan includes a plurality of target instructions, each target instruction includes an operator and an operation object, and the operator in each target instruction is supported by at least one of the at least two execution engines;

[0011] Each target instruction is dispatched to an execution engine that supports the operator included therein, so that each execution engine responds to the respective dispatched target instruction.

[0012] In an optional implementation, generating a data processing plan for running on at least two execution engines according to the operation objects and operators in each original instruction includes:

[0013] For any original instruction, determine a target execution engine that supports the operator in the original instruction in the at least two execution engines respectively, and generate a target instruction supported by the target execution engine according to the operation object and the operator in the original instruction;

[0014] The data processing plan is generated at least according to each generated target instruction.

[0015] In an optional implementation, the determining, in the at least two execution engines, respectively a target execution engine that supports the operator in the original instruction includes:

[0016] In the correspondence between the execution engine and the set of operators supported by the execution engine, searching for a set of operators included in the original instruction;

[0017] In the correspondence between the execution engine and the set of operators supported by the execution engine, the execution engine corresponding to the found set is searched and used as the target execution engine supporting the operator in the original instruction.

[0018] In an optional implementation, generating a target instruction supported by the target execution engine according to the operation object and the operator in the original instruction includes:

[0019] Generate instructions whose syntax can be supported by the target execution engine according to the operation objects and operators in the original instructions;

[0020] Attribute information is set in the generated instruction to obtain the target instruction, the attribute information at least including: sequence information of the operation objects, distribution information of the operation objects and an execution engine identifier of the target execution engine supporting the operator in the generated instruction.

[0021] In an optional implementation, the generated target instruction includes at least a first target instruction and a second target instruction, and the operation object in the first target instruction includes: output data obtained after executing the second target instruction;

[0022] Generating the data processing plan at least according to each generated target instruction includes:

[0023] determining whether the sequence information of the output data is supported by an operator in the first target instruction;

[0024] In a case where the sequence information of the output data is not supported by the operator in the first target instruction, generating an auxiliary instruction, wherein the auxiliary instruction is used to adjust the sequence information of the output data to sequence information that can be supported by the operator in the first target instruction;

[0025] The data processing plan is generated at least according to each generated target instruction and auxiliary instruction.

[0026] In an optional implementation, the generated target instruction includes at least a first target instruction and a second target instruction, and the operation object in the first target instruction includes: output data obtained after executing the second target instruction;

[0027] Generating the data processing plan at least according to each generated target instruction includes:

[0028] determining whether the distribution information of the output data is supported by an operator in the first target instruction;

[0029] In a case where the distribution information of the output data is not supported by the operator in the first target instruction, generating an auxiliary instruction, the auxiliary instruction being used to adjust the distribution information of the output data to distribution information supported by the operator in the first target instruction;

[0030] The data processing plan is generated at least according to each generated target instruction and auxiliary instruction.

[0031] In an optional implementation, the generated target instruction includes at least a first target instruction and a second target instruction, and the operation object in the first target instruction includes: output data obtained after executing the second target instruction;

[0032] Generating the data processing plan at least according to each generated target instruction includes:

[0033] Determining whether an operator in the first target instruction and an operator in the second target instruction are supported by the same execution engine;

[0034] In a case where the operator in the first target instruction and the operator in the second target instruction are not supported by the same execution engine, generating an auxiliary instruction, the auxiliary instruction being used to pass the output data to an execution engine that supports the operator in the first target instruction;

[0035] The data processing plan is generated at least according to each generated target instruction and auxiliary instruction.

[0036] In an optional implementation, generating the data processing plan at least according to each generated target instruction and auxiliary instruction includes:

[0037] Acquire the execution order between each generated target instruction; set the execution order of the auxiliary instruction between each generated target instruction, the execution order of the auxiliary instruction between each generated target instruction includes: the auxiliary instruction is located after the second target instruction and before the first target instruction;

[0038] According to the execution order between the generated target instructions and the execution order of the auxiliary instructions between the generated target instructions, the generated target instructions and the auxiliary instructions are combined to obtain the data processing plan.

[0039] In a second aspect, the present application shows a device for federated processing of data, the device comprising:

[0040] An acquisition module, used for acquiring a data processing statement, wherein the data processing statement includes a plurality of original instructions, and each original instruction includes an operator and an operation object;

[0041] A parsing module, used for parsing the data processing statement to obtain each original instruction in the data processing statement, and an operation object and an operator in each original instruction;

[0042] A generating module, configured to generate a data processing plan for running on at least two execution engines according to the operation objects and operators in each original instruction, wherein the operators supported by the at least two execution engines are not all the same, and the data processing plan includes a plurality of target instructions, each target instruction includes an operator and an operation object, and the operator in each target instruction is supported by at least one of the at least two execution engines;

[0043] The scheduling module is used to schedule each target instruction to the execution engine supporting the operator included therein, so that each execution engine responds to the respectively scheduled target instruction.

[0044] In an optional implementation, the generating module includes:

[0045] a determination unit, configured to determine, for any original instruction, a target execution engine that supports an operator in the original instruction in the at least two execution engines, and a first generation unit, configured to generate a target instruction supported by the target execution engine according to an operation object and an operator in the original instruction;

[0046] The second generating unit is used to generate the data processing plan at least according to each generated target instruction.

[0047] In an optional implementation, the determining unit includes:

[0048] A first search subunit, configured to search for a set of operators included in the original instruction in a correspondence between an execution engine and a set of operators supported by the execution engine;

[0049] The second search subunit is used to search for an execution engine corresponding to the found set in the correspondence between the execution engine and the set of operators supported by the execution engine, and use it as a target execution engine supporting the operator in the original instruction.

[0050] In an optional implementation, the first generating unit includes:

[0051] A first generating subunit, configured to generate an instruction whose syntax can be supported by the target execution engine according to an operation object and an operator in the original instruction;

[0052] The setting subunit is used to set attribute information in the generated instruction to obtain the target instruction, and the attribute information at least includes: sequence information of the operation objects, distribution information of the operation objects and the execution engine identifier of the target execution engine supporting the operator in the generated instruction.

[0053] In an optional implementation, the generated target instruction includes at least a first target instruction and a second target instruction, and the operation object in the first target instruction includes: output data obtained after executing the second target instruction;

[0054] The second generating unit comprises:

[0055] A first determining subunit, configured to determine whether the sequence information of the output data is supported by an operator in the first target instruction;

[0056] a second generating subunit, configured to generate an auxiliary instruction, when the sequence information of the output data is not supported by the operator in the first target instruction, wherein the auxiliary instruction is used to adjust the sequence information of the output data to sequence information that can be supported by the operator in the first target instruction;

[0057] The third generating subunit is used to generate the data processing plan at least according to each generated target instruction and auxiliary instruction.

[0058] In an optional implementation, the generated target instruction includes at least a first target instruction and a second target instruction, and the operation object in the first target instruction includes: output data obtained after executing the second target instruction;

[0059] The second generating unit comprises:

[0060] a second determining subunit, configured to determine whether the distribution information of the output data is supported by an operator in the first target instruction;

[0061] a fourth generating subunit, configured to generate an auxiliary instruction, if the distribution information of the output data is not supported by the operator in the first target instruction, wherein the auxiliary instruction is used to adjust the distribution information of the output data to distribution information supported by the operator in the first target instruction;

[0062] The fifth generating subunit is used to generate the data processing plan at least according to the generated target instructions and auxiliary instructions.

[0063] In an optional implementation, the generated target instruction includes at least a first target instruction and a second target instruction, and the operation object in the first target instruction includes: output data obtained after executing the second target instruction;

[0064] The second generating unit comprises:

[0065] a third determination subunit, configured to determine whether the operator in the first target instruction and the operator in the second target instruction are supported by the same execution engine;

[0066] a sixth generating subunit, configured to generate an auxiliary instruction when the operator in the first target instruction and the operator in the second target instruction are not supported by the same execution engine, wherein the auxiliary instruction is used to transfer the output data to an execution engine that supports the operator in the first target instruction;

[0067] The seventh generating subunit is used to generate the data processing plan at least according to each generated target instruction and auxiliary instruction.

[0068] In an optional implementation, the third generation subunit / fifth generation subunit / seventh generation subunit is specifically used to: obtain the execution order between the generated target instructions; set the execution order of the auxiliary instructions between the generated target instructions, and the execution order of the auxiliary instructions between the generated target instructions includes: the auxiliary instructions are located after the second target instruction and before the first target instruction; according to the execution order between the generated target instructions and the execution order of the auxiliary instructions between the generated target instructions, the generated target instructions and the auxiliary instructions are combined to obtain the data processing plan.

[0069] In a third aspect, the present application shows an electronic device, the electronic device comprising:

[0070] processor;

[0071] a memory for storing processor-executable instructions;

[0072] The processor is configured to execute the method for federated processing of data as described in the first aspect.

[0073] In a fourth aspect, the present application illustrates a non-temporary computer-readable storage medium, which, when instructions in the storage medium are executed by a processor of an electronic device, enables the electronic device to perform the method for federated processing of data as described in the first aspect.

[0074] In a fifth aspect, the present application illustrates a computer program product. When instructions in the computer program product are executed by a processor of an electronic device, the electronic device is enabled to perform the method for federated processing of data as described in the first aspect.

[0075] Compared with the prior art, the embodiments of the present application have the following advantages:

[0076] In the present application, a data processing statement is obtained. The data processing statement includes a plurality of original instructions, and each original instruction includes an operator and an operation object. The data processing statement is parsed to obtain each original instruction in the data processing statement, the operation object and the operator in each original instruction. A data processing plan for running on at least two execution engines is generated according to the operation object and the operator in each original instruction. The operators supported by at least two execution engines are not all the same. The data processing plan includes a plurality of target instructions, and each target instruction includes an operator and an operation object. The operator in each target instruction is supported by at least one execution engine of at least two execution engines. Each target instruction is respectively dispatched to an execution engine that supports the operator included therein, so that each execution engine responds to the target instruction that is dispatched.

[0077] Through the present application, at least two execution engines that support different operators are combined to jointly process data in the database, so that one of the at least two execution engines can be supplemented by the other execution engines. That is, when one of the execution engines supports limited operators, the operators supported by the other execution engines can provide supplements, thereby making it possible to process data in the database in more processing methods, to implement federated processing of data in the database (such as federated query, etc.), and to achieve more database functions when one of the execution engines supports limited operators, thereby improving the user experience of the database. BRIEF DESCRIPTION OF THE DRAWINGS

[0078] Figure 1 It is a flowchart of a method for federated processing of data shown in an exemplary embodiment of the present application.

[0079] Figure 2 It is a flowchart of a method for federated processing of data shown in an exemplary embodiment of the present application.

[0080] Figure 3 It is a structural block diagram of a data federation processing device shown in an exemplary embodiment of the present application.

[0081] Figure 4 It is a schematic diagram of the structure of a device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0082] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, the present application is further described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0083] Reference Figure 1 , shows a flow chart of a method for federated data processing of the present application, the method is applied in an electronic device, and the method may include:

[0084] In step S101, a data processing statement is obtained. The data processing statement includes a plurality of original instructions, and each original instruction includes an operator and an operation object.

[0085] In the present application, data is stored in a database, the database includes a distributed database, etc., at least two execution engines are loaded in the electronic device, the operators supported by at least two execution engines are not all the same, the operators include processing methods for processing data in the database, etc., the processing methods include mapping, filtering, aggregation and connection, etc. Each execution engine can process data in the database according to the operators supported by each. The purpose of processing data in the database includes adding, deleting, modifying and checking data in the database.

[0086] This application can be applied to various existing databases, such as MySQL database, SqlServer database and SQLite database, etc. Each database has an interface, which is used to allow the outside to process the data in the database, so that this application can perform federated processing on the data in the database through the database interface based on at least two execution engines.

[0087] The number of execution engines based on the "federated processing of data in the database" of this application is more than two. The specific number of execution engines to be used can be determined based on actual needs or actual conditions, and this application does not limit this.

[0088] Since the operators supported by at least two execution engines are not all the same, the present application can combine at least two execution engines to process data in the database, so that the present application can support processing data in the database with more operators, that is, it can support processing data in the database with more processing methods to meet the actual needs of users as much as possible.

[0089] In one example, among the at least two execution engines, one of the execution engines may include an execution engine independently developed by a developer, and the other execution engines may include execution engines already existing in the market, etc. For example, one of the at least two execution engines includes an independently developed execution engine Hologres, etc., and the execution engine Hologres supports PostgreSQL syntax, and the other of the at least two execution engines includes an execution engine PostgreSQL, etc., and the execution engine PostgreSQL supports more comprehensive operators. Therefore, based on the independently developed execution engine, combined with the execution engine PostgreSQL, data can be processed in the database with more operators, that is, it can support processing data in the database in more processing methods to meet the actual needs of users as much as possible.

[0090] In the present application, a data processing statement may include a computer program code developed by a user for processing data in a database, etc. A data processing statement includes a plurality of original instructions, and each original instruction includes an operator and an operation object.

[0091] For any original instruction in a data processing statement, the operation object in the original instruction can be used as input data of the operator in the original instruction, and the operator in the original instruction can process the operation object in the original instruction to obtain output data. The operation object in the original instruction may include data in a database, or output data of other original instructions, etc. For example, each original instruction has an execution order including output data of the previous original instruction whose execution order is before and adjacent to the original instruction, etc. The same is true for each other original instruction in the data processing statement.

[0092] When the user needs to process data in the database, the user may input a data processing statement to the electronic device. Afterwards, the electronic device may receive the data processing statement input by the user and then execute step S102.

[0093] In an example, it is assumed that the data processing statement includes: "SELECT A.id FROM Table1 Join Table2 on A.id=B.id where B.id>10".

[0094] The database includes Table 1 and Table 2, both of which store data. The data processing statement is used to connect the data with id A obtained from Table 1 with the data with id greater than 10 and the same as A obtained from Table 2.

[0095] The original instructions included in the data processing statement include:

[0096] Original instruction 1: Scan A id from table 1.

[0097] Original instruction 2: Map the first data with id A in Table 1.

[0098] Original instruction 3: Scan from Table 2 the ids that are the same as A and greater than 10.

[0099] Original instruction 4: Filter out the second data in Table 2 whose ID is the same as A and greater than 10.

[0100] Original instruction 5: concatenate the first data with the second data.

[0101] The execution order of the five original instructions includes: the execution order of original instruction 1 is before the execution order of original instruction 2, the execution order of original instruction 3 is before the execution order of original instruction 4, and the execution order of original instructions 2 and 4 is before the execution order of original instruction 5. Original instructions 1 and 2 can be executed in parallel, etc.

[0102] The operator in the original instruction 1 includes filter, and the operation objects include table 1 and A id. The original instruction 1 is used to perform a filter operation on table 1 according to A id to obtain all A ids in table 1.

[0103] The operator in the original instruction 2 includes a mapping (Project), and the operation objects include Table 1 and Aid. The original instruction 2 is used to perform a mapping (Project) operation on Table 1 according to A id to obtain all data in Table 1 with id A.

[0104] The operator in the original instruction 3 includes filter, and the operation objects include table 2, A id, and id greater than 10. The original instruction 3 is used to perform a filter operation on table 2 according to A id and id greater than 10, and obtain all ids in table 2 that are the same as A and greater than 10.

[0105] The operator in the original instruction 4 includes a mapping (Project), and the operation objects include Table 2 and the id that is the same as A and greater than 10. The original instruction 4 is used to perform a mapping (Project) operation on Table 2 according to the id that is the same as A and greater than 10, and obtain all the data in Table 2 whose id is the same as A and greater than 10.

[0106] The operator Join in the original instruction 5 operates on all data with id A in Table 1 and all data with id equal to A and greater than 10 in Table 2. The original instruction 5 is used to perform a Join operation on all data with id A in Table 1 and all data with id equal to A and greater than 10 in Table 2 to obtain a data, which is the data that needs to be obtained in the end.

[0107] In step S102, the data processing statement is parsed to obtain each original instruction in the data processing statement, and the operation object and operator in each original instruction.

[0108] In this application, the data processing statements can be parsed using existing parsing methods on the market, for example, the data processing statements are subjected to grammatical analysis, semantic analysis, and query rewriting, etc., to obtain each original instruction in the data processing statement, the operation object and the operator in each original instruction. This application does not limit the specific parsing method.

[0109] In addition, the execution order between the original instructions and the relationship between the output data obtained by the operators in each original instruction and the input data of the operators in other original instructions can also be obtained.

[0110] For example, in the above example, the data processing statement "SELECT A.id FROM Table1JoinTable2 on A.id=B.id where B.id>10" can be parsed to obtain the original instructions 1 to 5 in the data processing statement "SELECT A.id FROMTable1 Join Table2 on A.id=B.id where B.id>10", the operation objects and operators in each original instruction, and the execution order between the original instructions 1 to 5 can also be obtained.

[0111] In step S103, a data processing plan for running on at least two execution engines is generated according to the operation objects and operators in each original instruction.

[0112] The operators supported by at least two execution engines are not all the same. The data processing plan includes multiple target instructions, each target instruction includes an operator and an operation object, and the operator in each target instruction is supported by at least one execution engine of the at least two execution engines.

[0113] In one embodiment of the present application, a data processing plan for running on at least two execution engines may be generated based on the optimizer ORCA according to the operation objects and operators in each original instruction.

[0114] In another embodiment of the present application, at least two candidate data processing plans can be generated based on the optimizer ORCA, and at least two candidate data processing plans are not completely the same, for example, the number of target instructions included is not completely the same, or the operators in the target instructions are not completely the same, or the execution order between the target instructions is not completely the same, etc.

[0115] Then, based on the CBO (Cost-Based Optimization) optimizer, cost estimation can be performed on the at least two candidate data processing plans respectively to determine the execution costs of the at least two candidate data processing plans respectively, for example, the time consumed by the execution process and / or the computing resources consumed by the execution process, etc., and then the data processing plan with the lowest cost is selected from the at least two candidate data processing plans as the final data processing plan for running on at least two execution engines, so as to reduce the execution cost, for example, reduce the time consumed by the execution process and / or the computing resources consumed by the execution process, etc., so as to improve the efficiency of data processing and save system resources.

[0116] The specific method of generating the data processing plan in this step can be found in the following embodiments and will not be described in detail here.

[0117] For example, in the above example, it is assumed that at least two execution engines include execution engine A and execution engine B, execution engine A has the functions of scanning, mapping, and filtering, but does not have the function of joining, while execution engine B has the function of joining.

[0118] Therefore, in the original instructions 1-5, the mapping (Project) and filtering (Filter) operations can be performed on the execution engine A, and the connection (Join) operation can be performed on the execution engine B.

[0119] For example, in one embodiment, if execution engine A can support the syntax of data processing statements, original instruction 1 can be directly used as target instruction 1, original instruction 2 can be used as target instruction 2, original instruction 3 can be used as target instruction 3, and original instruction 4 can be used as target instruction 4. If execution engine A does not support the syntax of data processing statements, target instruction 1 whose syntax can be supported by execution engine A can be generated based on the operator and operation object in original instruction 1, target instruction 2 whose syntax can be supported by execution engine A can be generated based on the operator and operation object in original instruction 2, target instruction 3 whose syntax can be supported by execution engine A can be generated based on the operator and operation object in original instruction 3, and target instruction 4 whose syntax can be supported by execution engine A can be generated based on the operator and operation object in original instruction 4. And target instruction 5 whose syntax can be supported by execution engine B can be generated based on the operator and operation object in original instruction 5. Target instructions 1-4 are used to be executed by execution engine A, and target instruction 5 is used to be executed by execution engine B.

[0120] Afterwards, a data processing plan may be generated based on at least the target instructions 1-5. The data processing plan may include at least the target instructions 1-5. The execution order between the target instructions 1-5 may refer to the execution order between the original instructions 1-5, etc.

[0121] In step S104, each target instruction is dispatched to an execution engine that supports the operator included therein, so that each execution engine responds to the respective dispatched target instruction.

[0122] For any target instruction, one of the at least two execution engines that supports the operator in the target instruction can be determined, and then the target instruction can be scheduled to the determined one execution engine, and the same is true for each other target instruction.

[0123] Among them, each target instruction in the data processing plan has an execution order. After each target instruction is scheduled to the execution engine that supports the operator it includes, each execution engine can respond to its own scheduled target instruction according to the execution order between each target instruction, thereby completing the data processing plan and achieving the purpose of processing data in the database.

[0124] Among them, after the target instruction with the last execution order in each target instruction is executed by an execution engine, an output data will be output, and the output data is the final processing result of the data.

[0125] For example, in the above example, target instructions 1-4 can be scheduled to execution engine A, so that execution engine A executes target instructions 1-2 to obtain all data with id A in Table 1, and executes target instructions 3-4 to obtain all data with id the same as A and greater than 10 in Table 2.

[0126] The target instruction 5 is scheduled to the execution engine B, and all the data with id A in Table 1 and all the data with id the same as A and greater than 10 in Table 2 are scheduled to the execution engine B, so that the execution engine B executes the target instruction 5 to perform a join operation on all the data with id A in Table 1 and all the data with id the same as A and greater than 10 in Table 2, and the data obtained is the final data required.

[0127] In the present application, a data processing statement is obtained. The data processing statement includes a plurality of original instructions, and each original instruction includes an operator and an operation object. The data processing statement is parsed to obtain each original instruction in the data processing statement, the operation object and the operator in each original instruction. A data processing plan for running on at least two execution engines is generated according to the operation object and the operator in each original instruction. The operators supported by at least two execution engines are not all the same. The data processing plan includes a plurality of target instructions, and each target instruction includes an operator and an operation object. The operator in each target instruction is supported by at least one execution engine of at least two execution engines. Each target instruction is respectively dispatched to an execution engine that supports the operator included therein, so that each execution engine responds to the target instruction that is dispatched.

[0128] Through the present application, at least two execution engines that support different operators are combined to jointly process data in the database, so that one of the at least two execution engines can be supplemented by the other execution engines. That is, when one of the execution engines supports limited operators, the operators supported by the other execution engines can provide supplements, thereby making it possible to process data in the database in more processing methods, to implement federated processing of data in the database (such as federated query, etc.), and to achieve more database functions when one of the execution engines supports limited operators, thereby improving the user experience of the database.

[0129] In one embodiment of the present application, see Figure 2 , step S103 includes:

[0130] In step S201, for any original instruction, a target execution engine supporting an operator in the original instruction is determined in at least two execution engines respectively, and a target instruction supported by the target execution engine is generated according to an operation object and an operator in the original instruction.

[0131] In one embodiment of the present application, when determining the execution engine that supports the operator in the original instruction in at least two execution engines respectively, it can be implemented through the following process, including:

[0132] 11) In the correspondence between the execution engine and the set of operators supported by the execution engine, search for the set of operators included in the original instruction.

[0133] In the present application, for any one of the at least two execution engines loaded on the electronic device, the operators supported by the execution engine can be formed into a set of operators in advance, and then the execution engine and the set of operators are formed into corresponding table items, and stored in the corresponding relationship between the execution engine and the set of operators supported by the execution engine. The same is true for each of the other at least two execution engines integrated in the database.

[0134] Therefore, when determining the execution engines that support the operator in the original instruction in at least two execution engines respectively, the set including the operator in the original instruction can be searched in the correspondence between the execution engine and the set of operators supported by the execution engine, and then step 12) is executed.

[0135] 12) In the correspondence between the execution engine and the set of operators supported by the execution engine, find the execution engine corresponding to the found set and use it as the target execution engine that supports the operator in the original instruction.

[0136] In another embodiment of the present application, when generating a target instruction supported by the target execution engine according to the operation object and the operator in the original instruction, it can be implemented through the following process, including:

[0137] 21) Generate instructions whose syntax can be supported by the target execution engine based on the operation objects and operators in the original instruction.

[0138] In the present application, the target instruction includes an operator and an operation object. In one embodiment, the operator and the operation object included in the target instruction are obtained according to the operator and the operation object in the original instruction. In one example, the operator and the operation object included in the target instruction can be the same as the operator and the operation object in the original instruction.

[0139] “Target instructions supported by the target execution engine” can be understood to include at least two aspects of support. On the one hand, the operators in the target instructions are supported by the target execution engine and the syntax of the target instructions is supported by the target execution engine.

[0140] 22) Setting attribute information in the generated instruction to obtain the target instruction, wherein the attribute information at least includes: sequence information of the operation objects, distribution information of the operation objects, and the execution engine identifier of the target execution engine supporting the operator in the generated instruction.

[0141] In one embodiment of the present application, in the attribute information of the target instruction, the sequence information of the operation object is used to indicate the requirements of the operator in the target instruction for the sequence information of the operation object (input data), that is, the operator in the target instruction can only process the operation object data that meets the "requirements for the sequence information of the operation object" to obtain output data.

[0142] The sequence information includes: when there are at least two operation objects, the sequence information between the at least two operation objects, etc.

[0143] For example, assuming that the operator in the target instruction includes "Filter", the operator "Filter" has no special requirements for the sequence information of the operation object. Therefore, the sequence information of the operation object of the attribute information of the target instruction can be "none", which is used to indicate that the operator "Filter" can process the operation object data that meets any sequence information requirements to obtain output data.

[0144] In one embodiment of the present application, in the attribute information of the target instruction, the distribution information of the operation object is used to indicate the requirements of the operator in the target instruction for the distribution information of the operation object (input data), that is, the operator in the target instruction can only process the operation object data that meets the "requirements of the distribution information of the operation object" to obtain output data.

[0145] The distribution information includes: when there are at least two operation objects and the operation objects are distributed in the distributed database, the distribution information of the at least two operation objects in the distributed database, for example, including arbitrary distribution or hash distribution.

[0146] For example, assuming that the operator in the target instruction includes "Join", the operator "Join" has no special requirements for the distribution information of the operation object, and it can be arbitrarily distributed. Therefore, the distribution information of the operation object of the attribute information of the target instruction can be "any", which is used to indicate that the operator "Join" can process any distributed operation object to obtain output data.

[0147] For another example, assuming that the operator in the target instruction includes "Hash Join", the operator "Hash Join" has special requirements for the distribution information of the operation object. For example, the requirement for the distribution information of the operation object is that the operation object needs to be distributed in the form of hash distribution. For example, the Join Key (Key can be understood as a data identifier, etc.) needs to be hashed and distributed in a hash manner. Therefore, the distribution information of the operation object of the attribute information of the target instruction can be "Hash Distribution", which is used to indicate that the operator "Hash Join" can process the operation object of HashDistribution to obtain output data.

[0148] In step S202, a data processing plan is generated at least according to each generated target instruction.

[0149] In one embodiment of the present application, in this step, the execution order between each generated target instruction can be obtained. For example, the execution order between each original instruction in the data processing statement can be referred to to determine the execution order between each generated target instruction, etc. Of course, the execution order between each generated target instruction can also be obtained by other means. The present application does not limit the specific method for obtaining the execution order between each target instruction. Then, according to the execution order between each generated target instruction, each generated target instruction can be combined to obtain a data processing plan.

[0150] In another embodiment of the present application, the generated target instruction includes at least a first target instruction and a second target instruction, and the operation object in the first target instruction includes: output data obtained after executing the second target instruction. That is, the execution order of the second target instruction is before the execution order of the first target instruction, and the output data obtained after executing the second target instruction can be used as at least one input data of the first target instruction. Therefore, the first target instruction needs to obtain the output data of the second target instruction, and then use the operator in the first target instruction to process the obtained output data of the second target instruction. But sometimes.

[0151] However, sometimes the sequence information of the output data of the second target instruction is not supported by the operator in the first target instruction, which may cause the operator in the first target instruction to be unable to process the output data of the second target instruction, resulting in interruption of the data processing plan and failure to achieve the purpose of data processing.

[0152] Therefore, in order to avoid this situation, in another embodiment of the present application, when generating a data processing plan at least according to each generated target instruction, it can be determined whether the sequence information of the output data is supported by the operator in the first target instruction. In the case where the sequence information of the output data is not supported by the operator in the first target instruction, an auxiliary instruction is generated, and the auxiliary instruction is used to adjust the sequence information of the output data of the second target instruction to sequence information that can be supported by the operator in the first target instruction.

[0153] Among them, the auxiliary instruction includes an operator and an operation object, the operation object includes the output data of the second target instruction, the operator is used to adjust the sequence information of the output data of the second target instruction to obtain the output data of the second target instruction after the sequence information is adjusted, and the sequence information of the output data of the second target instruction after the sequence information is adjusted is supported by the operator in the first target instruction.

[0154] The output data of the second target instruction after the sequence information is adjusted can be used as the operation object (input data) in the first target instruction, that is, the operator in the first target instruction can process the output data of the second target instruction after the sequence information is adjusted.

[0155] Afterwards, a data processing plan may be generated at least based on the generated target instructions and auxiliary instructions. For example, the execution order between the generated target instructions may be obtained, for example, the execution order between the generated target instructions may be determined by referring to the execution order between the original instructions in the data processing statement, and the execution order between the generated target instructions may be set, and the execution order of the auxiliary instructions includes: the auxiliary instructions are located after the second target instruction and before the first target instruction. Then, according to the execution order between the generated target instructions and the execution order of the auxiliary instructions, the generated target instructions and the auxiliary instructions may be combined to obtain a data processing plan.

[0156] In another embodiment of the present application, the generated target instruction includes at least a first target instruction and a second target instruction, and the operation object in the first target instruction includes: output data obtained after executing the second target instruction. That is, the execution order of the second target instruction is before the execution order of the first target instruction, and the output data obtained after executing the second target instruction can be used as at least one input data of the first target instruction. Therefore, the first target instruction needs to obtain the output data of the second target instruction, and then use the operator in the first target instruction to process the obtained output data of the second target instruction.

[0157] However, sometimes the distribution information of the output data of the second target instruction is not supported by the operator in the first target instruction, which may cause the operator in the first target instruction to be unable to process the output data of the second target instruction, resulting in interruption of the data processing plan and failure to achieve the purpose of data processing.

[0158] Therefore, in order to avoid this situation, in another embodiment of the present application, when generating a data processing plan at least according to each generated target instruction, it can be determined whether the distribution information of the output data is supported by the operator in the first target instruction. In the case where the distribution information of the output data is not supported by the operator in the first target instruction, an auxiliary instruction is generated, and the auxiliary instruction is used to adjust the distribution information of the output data to the distribution information supported by the operator in the first target instruction.

[0159] Among them, the auxiliary instruction includes an operator and an operation object, the operation object includes the output data of the second target instruction, the operator includes an enforcer node, such as a motion node, such as a Hash motion node, and the operator is used to adjust the distribution information of the output data of the second target instruction to obtain the output data of the second target instruction after the distribution information is adjusted, and the distribution information of the output data of the second target instruction after the distribution information is adjusted is supported by the operator in the first target instruction.

[0160] The output data of the second target instruction after the distribution information is adjusted can be used as the operation object (input data) in the first target instruction, that is, the operator in the first target instruction can process the output data of the second target instruction after the distribution information is adjusted.

[0161] Afterwards, a data processing plan may be generated at least based on the generated target instructions and auxiliary instructions. For example, the execution order between the generated target instructions may be obtained, for example, the execution order between the generated target instructions may be determined by referring to the execution order between the original instructions in the data processing statement, and the execution order between the generated target instructions may be set, and the execution order of the auxiliary instructions includes: the auxiliary instructions are located after the second target instruction and before the first target instruction. Then, according to the execution order between the generated target instructions and the execution order of the auxiliary instructions, the generated target instructions and the auxiliary instructions may be combined to obtain a data processing plan.

[0162] In another embodiment of the present application, the generated target instruction includes at least a first target instruction and a second target instruction, the operator in the first target instruction and the operator in the second target instruction are supported by different execution engines respectively, and the operation object in the first target instruction includes: the output data obtained after executing the second target instruction. That is, the execution order of the second target instruction is before the execution order of the first target instruction, and the output data obtained after executing the second target instruction can be used as at least one input data of the first target instruction. Therefore, the first target instruction needs to obtain the output data of the second target instruction, and then use the operator in the first target instruction to process the obtained output data of the second target instruction. But sometimes.

[0163] However, since the operator in the first target instruction and the operator in the second target instruction are supported by different execution engines respectively, the execution engine that executes the operator in the first target instruction is different from the execution engine that executes the operator in the second target instruction. After the execution engine of the operator in the second target instruction processes the operation object in the second target instruction through the operator in the second target instruction and obtains the output data of the second target instruction, it needs to pass the output data of the second target instruction to the execution engine that supports the operator in the first target instruction, so that the execution engine that supports the operator in the first target instruction can obtain the output data of the second target instruction and the operator in the first target instruction processes the obtained output data of the second target instruction.

[0164] Therefore, in this embodiment, the output data of the second target instruction needs to be passed to the execution engine that supports the operator in the first target instruction. Otherwise, the execution engine that supports the operator in the first target instruction cannot obtain the output data of the second target instruction, resulting in the interruption of the data processing plan and the inability to achieve the purpose of data processing.

[0165] Therefore, in order to avoid this situation, in another embodiment of the present application, when generating a data processing plan at least according to each generated target instruction, it can be determined whether the operator in the first target instruction and the operator in the second target instruction are supported by the same execution engine. In the case that the operator in the first target instruction and the operator in the second target instruction are not supported by the same execution engine, an auxiliary instruction is generated, and the auxiliary instruction is used to pass the output data to an execution engine that supports the operator in the first target instruction.

[0166] Among them, the auxiliary instruction includes an operator and an operation object, the operation object includes the output data of the second target instruction, the operator includes an enforcer node, such as a forward node, etc., and the operator is used to pass the output data of the second target instruction to the execution engine that supports the operator in the first target instruction.

[0167] Among them, if the execution engine supporting the operator in the first target instruction and the execution engine supporting the operator in the second target instruction are respectively carried in different hardware devices, the output data of the second target instruction can be transmitted through a network connection.

[0168] Afterwards, a data processing plan can be generated based at least on the generated target instructions and auxiliary instructions. For example, the execution order between the generated target instructions can be obtained, for example, the execution order between the generated target instructions can be determined by referring to the execution order between the original instructions in the data processing statement, etc., and the execution order of the auxiliary instructions between the generated target instructions can be set, and the execution order of the auxiliary instructions between the generated target instructions includes: the auxiliary instructions are located after the second target instruction and before the first target instruction. Then, according to the execution order between the generated target instructions and the execution order of the auxiliary instructions between the generated target instructions, the generated target instructions and the auxiliary instructions can be combined to obtain a data processing plan.

[0169] It should be noted that, for the method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the order of the actions described, because according to the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all optional embodiments, and the actions involved are not necessarily required by the present application.

[0170] Reference Figure 3 , shows a structural block diagram of an embodiment of a data federation processing device of the present application, which may specifically include the following modules:

[0171] An acquisition module 11 is used to acquire a data processing statement, wherein the data processing statement includes a plurality of original instructions, and each original instruction includes an operator and an operation object;

[0172] A parsing module 12, used for parsing the data processing statement to obtain each original instruction in the data processing statement, and an operation object and an operator in each original instruction;

[0173] A generating module 13, configured to generate a data processing plan for running on at least two execution engines according to the operation objects and operators in each original instruction, wherein the operators supported by the at least two execution engines are not all the same, and the data processing plan includes a plurality of target instructions, each target instruction includes an operator and an operation object, and the operator in each target instruction is supported by at least one of the at least two execution engines;

[0174] The scheduling module 14 is used to schedule each target instruction to the execution engine supporting the operator included therein, so that each execution engine responds to the respectively scheduled target instruction.

[0175] In an optional implementation, the generating module includes:

[0176] a determination unit, configured to determine, for any original instruction, a target execution engine that supports an operator in the original instruction in the at least two execution engines, and a first generation unit, configured to generate a target instruction supported by the target execution engine according to an operation object and an operator in the original instruction;

[0177] The second generating unit is used to generate the data processing plan at least according to each generated target instruction.

[0178] In an optional implementation, the determining unit includes:

[0179] A first search subunit, configured to search for a set of operators included in the original instruction in a correspondence between an execution engine and a set of operators supported by the execution engine;

[0180] The second search subunit is used to search for an execution engine corresponding to the found set in the correspondence between the execution engine and the set of operators supported by the execution engine, and use it as a target execution engine supporting the operator in the original instruction.

[0181] In an optional implementation, the first generating unit includes:

[0182] A first generating subunit, configured to generate an instruction whose syntax can be supported by the target execution engine according to an operation object and an operator in the original instruction;

[0183] The setting subunit is used to set attribute information in the generated instruction to obtain the target instruction, and the attribute information at least includes: sequence information of the operation objects, distribution information of the operation objects and the execution engine identifier of the target execution engine supporting the operator in the generated instruction.

[0184] In an optional implementation, the generated target instruction includes at least a first target instruction and a second target instruction, and the operation object in the first target instruction includes: output data obtained after executing the second target instruction;

[0185] The second generating unit comprises:

[0186] A first determining subunit, configured to determine whether the sequence information of the output data is supported by an operator in the first target instruction;

[0187] a second generating subunit, configured to generate an auxiliary instruction, when the sequence information of the output data is not supported by the operator in the first target instruction, wherein the auxiliary instruction is used to adjust the sequence information of the output data to sequence information that can be supported by the operator in the first target instruction;

[0188] The third generating subunit is used to generate the data processing plan at least according to each generated target instruction and auxiliary instruction.

[0189] In an optional implementation, the generated target instruction includes at least a first target instruction and a second target instruction, and the operation object in the first target instruction includes: output data obtained after executing the second target instruction;

[0190] The second generating unit comprises:

[0191] a second determining subunit, configured to determine whether the distribution information of the output data is supported by an operator in the first target instruction;

[0192] a fourth generating subunit, configured to generate an auxiliary instruction, if the distribution information of the output data is not supported by the operator in the first target instruction, wherein the auxiliary instruction is used to adjust the distribution information of the output data to distribution information supported by the operator in the first target instruction;

[0193] The fifth generating subunit is used to generate the data processing plan at least according to the generated target instructions and auxiliary instructions.

[0194] In an optional implementation, the generated target instruction includes at least a first target instruction and a second target instruction, and the operation object in the first target instruction includes: output data obtained after executing the second target instruction;

[0195] The second generating unit comprises:

[0196] a third determination subunit, configured to determine whether the operator in the first target instruction and the operator in the second target instruction are supported by the same execution engine;

[0197] a sixth generating subunit, configured to generate an auxiliary instruction when the operator in the first target instruction and the operator in the second target instruction are not supported by the same execution engine, wherein the auxiliary instruction is used to transfer the output data to an execution engine that supports the operator in the first target instruction;

[0198] The seventh generating subunit is used to generate the data processing plan at least according to each generated target instruction and auxiliary instruction.

[0199] In an optional implementation, the third generation subunit / fifth generation subunit / seventh generation subunit is specifically used to: obtain the execution order between the generated target instructions; set the execution order of the auxiliary instructions between the generated target instructions, and the execution order of the auxiliary instructions between the generated target instructions includes: the auxiliary instructions are located after the second target instruction and before the first target instruction; according to the execution order between the generated target instructions and the execution order of the auxiliary instructions between the generated target instructions, the generated target instructions and the auxiliary instructions are combined to obtain the data processing plan.

[0200] In the present application, a data processing statement is obtained. The data processing statement includes a plurality of original instructions, and each original instruction includes an operator and an operation object. The data processing statement is parsed to obtain each original instruction in the data processing statement, the operation object and the operator in each original instruction. A data processing plan for running on at least two execution engines is generated according to the operation object and the operator in each original instruction. The operators supported by at least two execution engines are not all the same. The data processing plan includes a plurality of target instructions, and each target instruction includes an operator and an operation object. The operator in each target instruction is supported by at least one execution engine of at least two execution engines. Each target instruction is respectively dispatched to an execution engine that supports the operator included therein, so that each execution engine responds to the target instruction that is dispatched.

[0201] Through the present application, at least two execution engines that support different operators are combined to jointly process data in the database, so that one of the at least two execution engines can be supplemented by the other execution engines. That is, when one of the execution engines supports limited operators, the operators supported by the other execution engines can provide supplements, thereby making it possible to process data in the database in more processing methods, to implement federated processing of data in the database (such as federated query, etc.), and to achieve more database functions when one of the execution engines supports limited operators, thereby improving the user experience of the database.

[0202] The embodiment of the present application also provides a non-volatile readable storage medium, which stores one or more modules (programs). When the one or more modules are applied to a device, the device can execute instructions (instructions) of each method step in the embodiment of the present application.

[0203] The present application embodiment provides one or more machine-readable media on which instructions are stored, and when executed by one or more processors, the electronic device executes one or more of the methods described in the above embodiments. In the present application embodiment, the electronic device includes a server, a gateway, a sub-device, etc., and the sub-device is an Internet of Things device or other device.

[0204] The embodiments of the present disclosure may be implemented as an apparatus configured as desired using any appropriate hardware, firmware, software, or any combination thereof, and the apparatus may include electronic devices such as servers (clusters), terminal devices such as IoT devices, and the like.

[0205] Figure 4 An exemplary apparatus 1300 that can be used to implement various embodiments described in this application is schematically shown.

[0206] For one embodiment, Figure 4 An exemplary apparatus 1300 is shown having one or more processors 1302, a control module (chip set) 1304 coupled to at least one of the (one or more) processors 1302, a memory 1306 coupled to the control module 1304, a non-volatile memory (NVM) / storage device 1308 coupled to the control module 1304, one or more input / output devices 1310 coupled to the control module 1304, and a network interface 1312 coupled to the control module 1306.

[0207] The processor 1302 may include one or more single-core or multi-core processors, and the processor 1302 may include any combination of general-purpose processors or special-purpose processors (such as graphics processors, application processors, baseband processors, etc.). In some embodiments, the device 1300 can be used as a server device such as a gateway described in the embodiments of the present application.

[0208] In some embodiments, the device 1300 may include one or more computer-readable media (e.g., memory 1306 or NVM / storage device 1308) having instructions 1314 and one or more processors 1302 configured to execute the instructions 1314 in combination with the one or more computer-readable media to implement a module to perform the actions described in the present disclosure.

[0209] For one embodiment, the control module 1304 may include any suitable interface controller to provide any suitable interface to at least one of the processor(s) 1302 and / or any suitable device or component in communication with the control module 1304 .

[0210] The control module 1304 may include a memory controller module to provide an interface to the memory 1306. The memory controller module may be a hardware module, a software module, and / or a firmware module.

[0211] The memory 1306 may be used, for example, to load and store data and / or instructions 1314 for the device 1300. For one embodiment, the memory 1306 may include any suitable volatile memory, such as a suitable DRAM. In some embodiments, the memory 1306 may include double data rate type four synchronous dynamic random access memory (DDR4 SDRAM).

[0212] For one embodiment, control module 1304 may include one or more input / output controllers to provide an interface to NVM / storage device 1308 and input / output device(s) 1310 .

[0213] For example, NVM / storage 1308 may be used to store data and / or instructions 1314. NVM / storage 1308 may include any suitable non-volatile memory (e.g., flash memory) and / or may include any suitable non-volatile storage device(s) (e.g., one or more hard disk drives (HDDs), one or more compact disk (CD) drives, and / or one or more digital versatile disk (DVD) drives).

[0214] NVM / storage device 1308 may include storage resources that are physically part of the device on which apparatus 1300 is installed, or it may be accessible to the device without being part of the device. For example, NVM / storage device 1308 may be accessed via input / output device(s) 1310 over a network.

[0215] (One or more) input / output devices 1310 may provide an interface for the apparatus 1300 to communicate with any other appropriate device, and the input / output device 1310 may include a communication component, a phonetic component, a sensor component, etc. The network interface 1312 may provide an interface for the apparatus 1300 to communicate through one or more networks, and the apparatus 1300 may wirelessly communicate with one or more components of a wireless network according to any of one or more wireless network standards and / or protocols, for example, accessing a wireless network based on a communication standard, such as WiFi, 2G, 3G, 4G, 5G, etc., or a combination thereof for wireless communication.

[0216] For one embodiment, at least one of the processor(s) 1302 may be packaged together with the logic of one or more controllers (e.g., a memory controller module) of the control module 1304. For one embodiment, at least one of the processor(s) 1302 may be packaged together with the logic of one or more controllers of the control module 1304 to form a system-in-package (SiP). For one embodiment, at least one of the processor(s) 1302 may be integrated on the same die with the logic of one or more controllers of the control module 1304. For one embodiment, at least one of the processor(s) 1302 may be integrated on the same die with the logic of one or more controllers of the control module 1304 to form a system-on-chip (SoC).

[0217] In various embodiments, the device 1300 may be, but is not limited to, a terminal device such as a server, a desktop computing device, or a mobile computing device (e.g., a laptop computing device, a handheld computing device, a tablet computer, a netbook, etc.). In various embodiments, the device 1300 may have more or fewer components and / or a different architecture. For example, in some embodiments, the device 1300 includes one or more cameras, a keyboard, a liquid crystal display (LCD) screen (including a touch screen display), a non-volatile memory port, multiple antennas, a graphics chip, an application-specific integrated circuit (ASIC), and a speaker.

[0218] An embodiment of the present application provides an electronic device, comprising: one or more processors; and one or more machine-readable media having instructions stored thereon, which, when executed by the one or more processors, enable the electronic device to perform one or more of the methods for federal processing of data as described in the present application.

[0219] As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0220] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0221] The embodiments of the present application are described with reference to the flowcharts and / or block diagrams of the methods, terminal devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing terminal device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0222] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing terminal device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0223] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device so that a series of operating steps are executed on the computer or other programmable terminal device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable terminal device to implement the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0224] Although the preferred embodiments of the present application have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present application.

[0225] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or terminal device including the elements.

[0226] The above is a detailed introduction to a method and device for federal processing of data provided by the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method and core idea of ​​the present application. At the same time, for those skilled in the art, according to the idea of ​​the present application, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A method for federal processing of data, characterized in that: The method comprises: Acquire a data processing statement, wherein the data processing statement includes a plurality of original instructions, and each original instruction includes an operator and an operation object; Parsing the data processing statement to obtain each original instruction in the data processing statement, and an operation object and an operator in each original instruction; Generate a data processing plan for running on at least two execution engines according to the operation objects and operators in each original instruction, wherein the operators supported by the at least two execution engines are not all the same, and the data processing plan includes a plurality of target instructions, each target instruction includes an operator and an operation object, and the operator in each target instruction is supported by at least one of the at least two execution engines; Dispatching each target instruction to an execution engine supporting the operator included therein, so that each execution engine responds to the respective dispatched target instruction; The step of generating a data processing plan for running on at least two execution engines according to the operation objects and operators in each original instruction includes: For any original instruction, determine a target execution engine that supports the operator in the original instruction in the at least two execution engines respectively, and generate a target instruction supported by the target execution engine according to the operation object and the operator in the original instruction; The data processing plan is generated at least according to each generated target instruction.

2. The method according to claim 1, characterized in that The step of respectively determining, in the at least two execution engines, a target execution engine that supports the operator in the original instruction comprises: In the correspondence between the execution engine and the set of operators supported by the execution engine, searching for a set of operators included in the original instruction; In the correspondence between the execution engine and the set of operators supported by the execution engine, the execution engine corresponding to the found set is searched and used as the target execution engine supporting the operator in the original instruction.

3. The method according to claim 1, characterized in that: The step of generating a target instruction supported by the target execution engine according to the operation object and the operator in the original instruction includes: Generate instructions whose syntax can be supported by the target execution engine according to the operation objects and operators in the original instructions; Attribute information is set in the generated instruction to obtain the target instruction, the attribute information at least including: sequence information of the operation objects, distribution information of the operation objects and an execution engine identifier of the target execution engine supporting the operator in the generated instruction.

4. The method according to claim 1, characterized in that The generated target instructions include at least a first target instruction and a second target instruction, and the operation object in the first target instruction includes: output data obtained after executing the second target instruction; Generating the data processing plan at least according to each generated target instruction includes: determining whether the sequence information of the output data is supported by an operator in the first target instruction; In a case where the sequence information of the output data is not supported by the operator in the first target instruction, generating an auxiliary instruction, wherein the auxiliary instruction is used to adjust the sequence information of the output data to sequence information that can be supported by the operator in the first target instruction; The data processing plan is generated at least according to each generated target instruction and auxiliary instruction.

5. The method according to claim 1, characterized in that The generated target instructions include at least a first target instruction and a second target instruction, and the operation object in the first target instruction includes: output data obtained after executing the second target instruction; Generating the data processing plan at least according to each generated target instruction includes: determining whether the distribution information of the output data is supported by an operator in the first target instruction; In a case where the distribution information of the output data is not supported by the operator in the first target instruction, generating an auxiliary instruction, the auxiliary instruction being used to adjust the distribution information of the output data to distribution information supported by the operator in the first target instruction; The data processing plan is generated at least according to each generated target instruction and auxiliary instruction.

6. The method according to claim 1, characterized in that The generated target instructions include at least a first target instruction and a second target instruction, and the operation object in the first target instruction includes: output data obtained after executing the second target instruction; Generating the data processing plan at least according to each generated target instruction includes: Determining whether an operator in the first target instruction and an operator in the second target instruction are supported by the same execution engine; In a case where the operator in the first target instruction and the operator in the second target instruction are not supported by the same execution engine, generating an auxiliary instruction, the auxiliary instruction being used to pass the output data to an execution engine that supports the operator in the first target instruction; The data processing plan is generated at least according to each generated target instruction and auxiliary instruction.

7. The method according to any one of claims 4 to 6, characterized in that: Generating the data processing plan at least according to each generated target instruction and auxiliary instruction includes: Acquire the execution order between each generated target instruction; set the execution order of the auxiliary instruction between each generated target instruction, the execution order of the auxiliary instruction between each generated target instruction includes: the auxiliary instruction is located after the second target instruction and before the first target instruction; According to the execution order between the generated target instructions and the execution order of the auxiliary instructions between the generated target instructions, the generated target instructions and the auxiliary instructions are combined to obtain the data processing plan.

8. A federated data processing device, characterized in that: The device comprises: An acquisition module, used for acquiring a data processing statement, wherein the data processing statement includes a plurality of original instructions, and each original instruction includes an operator and an operation object; A parsing module, used for parsing the data processing statement to obtain each original instruction in the data processing statement, and an operation object and an operator in each original instruction; A generating module, configured to generate a data processing plan for running on at least two execution engines according to the operation objects and operators in each original instruction, wherein the operators supported by the at least two execution engines are not all the same, and the data processing plan includes a plurality of target instructions, each target instruction includes an operator and an operation object, and the operator in each target instruction is supported by at least one of the at least two execution engines; A scheduling module, used to schedule each target instruction to an execution engine supporting the operator included therein, so that each execution engine responds to the respective scheduled target instruction; The generation module comprises: a determination unit, configured to determine, for any original instruction, a target execution engine that supports an operator in the original instruction in the at least two execution engines, and a first generation unit, configured to generate a target instruction supported by the target execution engine according to an operation object and an operator in the original instruction; The second generating unit is used to generate the data processing plan at least according to each generated target instruction.

9. The device according to claim 8, characterized in that The determining unit comprises: A first search subunit, configured to search for a set of operators included in the original instruction in a correspondence between an execution engine and a set of operators supported by the execution engine; The second search subunit is used to search for an execution engine corresponding to the found set in the correspondence between the execution engine and the set of operators supported by the execution engine, and use it as a target execution engine supporting the operator in the original instruction.

10. The device according to claim 8, characterized in that The first generating unit comprises: A first generating subunit, configured to generate an instruction whose syntax can be supported by the target execution engine according to an operation object and an operator in the original instruction; The setting subunit is used to set attribute information in the generated instruction to obtain the target instruction, and the attribute information at least includes: sequence information of the operation objects, distribution information of the operation objects and the execution engine identifier of the target execution engine supporting the operator in the generated instruction.

11. The device according to claim 8, characterized in that The generated target instructions include at least a first target instruction and a second target instruction, and the operation object in the first target instruction includes: output data obtained after executing the second target instruction; The second generating unit comprises: A first determining subunit, configured to determine whether the sequence information of the output data is supported by an operator in the first target instruction; a second generating subunit, configured to generate an auxiliary instruction, when the sequence information of the output data is not supported by the operator in the first target instruction, wherein the auxiliary instruction is used to adjust the sequence information of the output data to sequence information that can be supported by the operator in the first target instruction; The third generating subunit is used to generate the data processing plan at least according to each generated target instruction and auxiliary instruction.

12. The device according to claim 8, characterized in that The generated target instructions include at least a first target instruction and a second target instruction, and the operation object in the first target instruction includes: output data obtained after executing the second target instruction; The second generating unit comprises: a second determining subunit, configured to determine whether the distribution information of the output data is supported by an operator in the first target instruction; a fourth generating subunit, configured to generate an auxiliary instruction, if the distribution information of the output data is not supported by the operator in the first target instruction, wherein the auxiliary instruction is used to adjust the distribution information of the output data to distribution information supported by the operator in the first target instruction; The fifth generating subunit is used to generate the data processing plan at least according to the generated target instructions and auxiliary instructions.

13. The device according to claim 8, characterized in that The generated target instructions include at least a first target instruction and a second target instruction, and the operation object in the first target instruction includes: output data obtained after executing the second target instruction; The second generating unit comprises: a third determination subunit, configured to determine whether the operator in the first target instruction and the operator in the second target instruction are supported by the same execution engine; a sixth generating subunit, configured to generate an auxiliary instruction when the operator in the first target instruction and the operator in the second target instruction are not supported by the same execution engine, wherein the auxiliary instruction is used to transfer the output data to an execution engine that supports the operator in the first target instruction; The seventh generating subunit is used to generate the data processing plan at least according to each generated target instruction and auxiliary instruction.

14. The device according to any one of claims 11 to 13, characterized in that: The third generation subunit / fifth generation subunit / seventh generation subunit is specifically used to: obtain the execution order between the generated target instructions; set the execution order of the auxiliary instructions between the generated target instructions, and the execution order of the auxiliary instructions between the generated target instructions includes: the auxiliary instructions are located after the second target instruction and before the first target instruction; according to the execution order between the generated target instructions and the execution order of the auxiliary instructions between the generated target instructions, the generated target instructions and the auxiliary instructions are combined to obtain the data processing plan.

15. An electronic device, characterized in that: The electronic device comprises: processor; a memory for storing processor-executable instructions; The processor is configured to execute the method for federated processing of data as described in any one of claims 1-7.

16. A non-transitory computer-readable storage medium, when instructions in the storage medium are executed by a processor of an electronic device, enables the electronic device to execute the method for federated processing of data as described in any one of claims 1-7.

Images