Data processing method, apparatus, system, device, and medium
By adjusting the execution order of the target aggregation function and the target scalar function, the problem of redundant computation in distributed systems is solved, and more efficient data processing is achieved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- ALIBABA (CHINA) CO LTD
- Filing Date
- 2023-03-23
- Publication Date
- 2026-07-31
AI Technical Summary
Existing distributed systems, with limited resources and poor scalability, cannot effectively improve data processing capabilities, especially when executing aggregation functions, which involves redundant computational tasks, resulting in low overall work efficiency.
By determining whether the relationship between the target aggregate function and the target scalar function conforms to the execution optimization rules, and adjusting their execution order if they do, the target aggregate function and the target scalar function are executed synchronously, reducing the number of loop calculations of the scalar function.
It improves the data processing efficiency of distributed systems, reduces hardware resource consumption, and enhances data processing capabilities.
Smart Images
Figure CN116401277B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of computer technology, and in particular to data processing methods, apparatus, systems, devices and media. Background Technology
[0002] With the rapid development of computer technology, distributed systems have been widely used. For example, distributed systems are used to meet the application requirements of large-scale data storage and real-time performance.
[0003] However, as the amount of data in distributed systems grows, and as data access demands increase, distributed system resources struggle to meet the actual application needs of users. This is especially true when existing distributed systems have limited resources and poor scalability, making it impossible to improve data processing capabilities through resource expansion. Therefore, a solution is needed that can enhance data processing capabilities based on existing distributed system resources. Summary of the Invention
[0004] To address or improve the problems existing in the prior art, various embodiments of this application provide data processing methods, apparatus, systems, devices, and media.
[0005] Firstly, in one embodiment of this application, a data processing method is provided. The method includes:
[0006] In response to a data processing request, determine the target aggregate function and target scalar function contained in the target query statement;
[0007] Determine whether the relationship between the parameter types contained in the target aggregation function and the target scalar function conforms to the execution optimization rules;
[0008] If the execution optimization rules are met, the execution order of the target aggregate function and the target scalar function in the target query statement is adjusted so that the data processing task is executed based on the adjusted execution order.
[0009] Secondly, in one embodiment of this application, a data processing apparatus is provided, the apparatus comprising:
[0010] The determination module is used to determine the target aggregate function and target scalar function contained in the target query statement in response to a data processing request;
[0011] The judgment module is used to determine whether the relationship between the parameter types contained in the target aggregation function and the target scalar function conforms to the execution optimization rules;
[0012] The adjustment module is used to adjust the execution order of the target aggregate function and the target scalar function in the target query statement if the execution optimization rules are met, so as to execute the data processing task based on the adjusted execution order.
[0013] Thirdly, in one embodiment of this application, a data processing system is provided, comprising:
[0014] Data nodes are used to store at least one data packet containing the target data and to perform computational tasks.
[0015] A server is used to execute the method described in the first aspect.
[0016] Fourthly, in one embodiment of this application, an electronic device is provided, including a memory and a processor; wherein,
[0017] The memory is used to store programs;
[0018] The processor, coupled to the memory, is configured to execute the program stored in the memory for implementing the data processing method described in the first aspect.
[0019] Fifthly, in one embodiment of this application, a non-transitory machine-readable storage medium is provided, wherein executable code is stored on the non-transitory machine-readable storage medium, and when the executable code is executed by a processor of an electronic device, the processor performs the data processing method as described in the first aspect.
[0020] The technical solution provided in this application includes multiple data nodes in a distributed system, and also stores multiple rows of target data in the distributed system. After determining the target query statement and the target data to be processed, the target query statement is processed to generate an abstract syntax tree containing multiple nodes. The target aggregate function and target scalar function contained in the target query statement are further determined. When the matching relationship between the parameter type in the target aggregate function and the scalar function conforms to the execution optimization rules, the parameters in the target aggregate function are replaced with the target scalar function according to the previously obtained execution optimization rules. Based on the replaced target aggregate function and target scalar function, the task of retrieving the target data is executed synchronously, which can effectively reduce the number of scalar function loop calculations during the sequential execution of the scalar function and the target aggregate function. This can improve the data processing efficiency in the distributed system. Attached Figure Description
[0021] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0022] Figure 1 A flowchart illustrating the data processing method provided in an embodiment of this application;
[0023] Figure 2 A flowchart illustrating the process of determining association relationships provided in an embodiment of this application;
[0024] Figure 3 This is a schematic flowchart of a method for optimizing the execution order provided in an embodiment of this application;
[0025] Figure 4 A schematic diagram illustrating the data processing based on the adjusted execution order, as exemplified in an embodiment of this application;
[0026] Figure 5 A schematic diagram illustrating the generation of a target logic execution plan, as exemplified in an embodiment of this application;
[0027] Figure 6 A schematic diagram illustrating the random value selection process of an aggregation function as an example of an embodiment of this application;
[0028] Figure 7 This is a schematic diagram of the structure of a data processing device provided in an embodiment of this application;
[0029] Figure 8 This is a schematic diagram of the structure of a data processing system provided in an embodiment of this application;
[0030] Figure 9 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation
[0031] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings.
[0032] In some processes described in the specification, claims, and accompanying drawings of this application, multiple operations appearing in a specific order are included. These operations may be executed out of order or in parallel. Operation numbers such as 101, 102, etc., are merely used to distinguish different operations and do not represent any execution order. Furthermore, these processes may include more or fewer operations, and these operations may be executed sequentially or in parallel. It should be noted that the terms "first," "second," etc., used herein are used to distinguish different messages, devices, modules, etc., and do not represent a sequential order, nor do they limit "first" and "second" to different types. Moreover, the embodiments described below are only a part of the embodiments of this application, not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.
[0033] Terminology Explanation:
[0034] Distributed system: A system consisting of multiple distributed computers connected by an interconnected network, where the system's processing and control functions are distributed across the individual computers.
[0035] An Abstract Syntax Tree (AST), or simply a syntax tree, is an abstract representation of the syntactic structure of source code. It represents the syntactic structure of a programming language in a tree-like structure, where each node in the tree represents a structure within the source code.
[0036] Structured Query Language (SQL) is a special-purpose programming language used for database querying and programming. It is used to access, query, update, and manage relational database systems.
[0037] Antlr is an open-source parser that can automatically generate and visualize a syntax tree based on input.
[0038] The arbitrary function returns any non-empty value in x.
[0039] In distributed system applications, as data volume and data access demands increase, distributed system resources struggle to meet actual user needs. This is especially true when existing distributed systems have limited resources and poor scalability, making it impossible to improve data processing capabilities through resource expansion. In data processing, such as log storage services, various aggregate functions are frequently used. The `arbitrary` function (hereinafter referred to as the target aggregate function) is a commonly used one, randomly selecting a non-null value from data groups belonging to the same group. In structured query languages, aggregate function parameters can be of various types, such as nested references to scalar functions or subqueries. When nested references to subqueries, the aggregate function parameter is an identifier, and the value corresponding to this identifier is the result of the scalar function's processing.
[0040] During the execution of computational tasks by an aggregate function, if the target data group contains 1000 rows, the nested scalar function is executed first, meaning the scalar function performs 1000 computations. Then, the aggregate function performs a random value selection task. It's important to note that when the aggregate function performs its computation, it randomly selects a non-null value. In other words, theoretically, the aggregate function only needs to obtain one non-null value from the 1000 computations performed by the scalar function; the scalar function's computations on other target data are redundant. In distributed systems with limited resources, the negative impact of redundant computational tasks on overall efficiency is even more pronounced. Therefore, a technical solution is needed to effectively improve data processing efficiency in database system application scenarios.
[0041] The technical solution implemented in this application will be explained and described below with reference to specific embodiments.
[0042] like Figure 1 This is a flowchart illustrating the data processing method provided in an embodiment of this application. The execution entity of this method can be a server. From Figure 1 The specific steps can be seen as follows:
[0043] 101: In response to a data processing request, determine the target aggregate function and target scalar function contained in the target query statement.
[0044] 102: Determine whether the relationship between the parameter types contained in the target aggregation function and the target scalar function conforms to the execution optimization rules.
[0045] 103: If the execution optimization rules are met, the execution order of the target aggregate function and the target scalar function in the target query statement is adjusted so that the data processing task is executed based on the adjusted execution order.
[0046] It should be noted that the technical solution of this application can be applied to both single-machine database systems and distributed database systems. For ease of understanding, the following embodiments will be illustrated using an application to a distributed database system.
[0047] The target query statement mentioned here is in Structured Query Language (SQL). For example, "select project,arbitrary(json_extract(content,'$.country'))FROM new_table group by project". Here, `arbitrary` is the target aggregate function, and `json_extract(content,'$.country')` is the target scalar function.
[0048] The data processing requests mentioned here can be log storage requests, log query requests, etc., initiated by the client to the server. The server can be a server in a single-machine database system, or a server in a distributed system (e.g., a coordinating node).
[0049] It should be noted that the target scalar function nested within the target aggregate function mentioned here can take several forms. It can be a direct nesting of the scalar function itself, or it can be done by nesting a subquery statement, but the value of the subquery statement is calculated using the scalar function.
[0050] When analyzing and processing the target query statement, the Antlr4 tool can be used to perform lexical and syntactic analysis on the target query statement to obtain the corresponding Abstract Syntax Tree (AST).
[0051] After determining the target aggregate function and the target scalar function, we will analyze the specific parameter types defined in the target aggregate function, such as identifier types and numeric types. If the parameter type is numeric, it means that the current target aggregate function does not support nested scalar functions. If the parameter type is identifier, it means that the current target aggregate function supports nested scalar functions, subqueries, and other identifier-type parameters.
[0052] The process of determining whether the relationship between the parameter types contained in the target aggregation function and the target scalar function conforms to the execution optimization rules will be illustrated in the following examples, and will not be repeated here.
[0053] If the above judgment process determines that the execution optimization rules are met, the execution order of the target aggregate function and the target scalar function in the target query statement will be adjusted. Specifically, previously, the execution order of the target aggregate function and the nested target scalar function was as follows: first, the target scalar function was used to calculate and process the target data for all rows in the data group, obtaining multiple corresponding processing results; then, the target aggregate function was used to randomly select one of these results as the desired random value. After the adjustment, the target aggregate function and the target scalar function are executed synchronously. This means that the target scalar function can process a row of target data, which can then be used as the result of the target aggregate function's random value selection. When the target aggregate function performs random value selection, it no longer needs to wait for the target scalar function to complete the processing of all rows in the grouped data; it only needs to provide one result to complete the random value selection task. This effectively reduces the computational workload, decreases the consumption of hardware resources in the database system, and thus improves the data processing efficiency and capability of the database system (including distributed and standalone systems).
[0054] In one or more embodiments of this application, such as Figure 2 This is a flowchart illustrating the process of determining association relationships provided in an embodiment of this application. Figure 2 As can be seen, determining whether the relationship between the parameter types contained in the target aggregation function and the target scalar function conforms to the execution optimization rules specifically includes the following steps:
[0055] 201: Determine the parameter type of the parameters in the target aggregation function.
[0056] 202: When the parameter type is an identifier, determine whether the identifier is obtained by processing the target data by the target scalar function.
[0057] 203: If the judgment result is yes, then determine whether the target scalar function is referenced by other functions besides the target aggregate function.
[0058] 204: If the judgment result is negative, then the association relationship is determined to conform to the execution optimization rule.
[0059] The compliance with the execution optimization rules mentioned here means that there is a direct or indirect reference relationship between the target aggregate function and the target scalar function, the parameter type of the target aggregate function is an identifier type, the identifier is obtained by the target scalar function processing the target data, and the target scalar function is only referenced by the target aggregate function and is not referenced by any other function.
[0060] The following section will elaborate on the above judgment process.
[0061] In the parameters set for the target aggregation function, the parameter type can be of various types, such as numeric types, identifier types, etc. Since scalar functions are identifier types when used as nested references, referencing scalar functions is only possible when the parameter type set for the target aggregation function is an identifier type. In other words, only then can subsequent data processing tasks be executed according to the execution optimization rules. When the parameter type is not an identifier type, subsequent data processing tasks are not executed according to the execution optimization rules.
[0062] After determining that the parameter type is an identifier, the next step is to determine whether the identifier can be obtained by processing the target data using the target scalar function. In other words, whether the output of the target scalar function can be referenced as a parameter of the target aggregate function. This reference can be direct or indirect. If the identifier cannot be calculated using the target scalar function, it means that there is no reference relationship between the target aggregate function and the target scalar function.
[0063] After determining that there is a reference relationship between the target scalar function and the target aggregate function, the next step is to determine whether the target scalar function is also referenced by other functions besides the target aggregate function. When the target scalar function is referenced by other functions simultaneously (that is, when a node in the obtained abstract syntax tree or a logical node in the logical execution plan corresponds to the target scalar function and is associated with other referencing functions), optimizing the execution order of the target aggregate function and the target scalar function may affect the correctness of the execution results of other referencing functions. Of course, if a risk analysis of the other functions referencing the target scalar function reveals that optimizing the execution order of the target aggregate function and the target scalar function will not adversely affect the other functions, then the association relationship can be considered to conform to the execution optimization rules. For example, after risk analysis of the other functions mentioned here, if the other functions are output functions, meaning they output and store any execution result of the target scalar function indiscriminately, then optimizing the execution order will not adversely affect the correctness of the output results.
[0064] It should be noted that the relationships mentioned here can be understood as the reference relationships between the target aggregate function and the target scalar function, as well as the reference relationships between the target aggregate function and other functions, and the reference relationships between the target scalar function and other functions.
[0065] When analyzing relationships, one can do so based on an abstract syntax tree or a logical execution plan.
[0066] Furthermore, it should be noted that the execution order of the above judgment process can be adjusted as needed. Although Figure 2The corresponding embodiments use labels to define the execution content, but do not limit the execution order. In practical applications, users can adjust the execution order of the judgment steps according to their needs or actual circumstances. Figure 2 This is merely an example and does not constitute a limitation on the technical solution of this application.
[0067] In one or more embodiments of this application, such as Figure 3 This is a schematic flowchart illustrating a method for optimizing execution order provided in an embodiment of this application. Figure 3 As can be seen, if the execution optimization rules are met, the execution order of the target aggregate function and the target scalar function in the target query statement is adjusted so that the data processing task is executed based on the adjusted execution order. Specifically, this includes the following steps:
[0068] 301: When the relationship between the parameter types contained in the target aggregation function and the target scalar function conforms to the execution optimization rule, determine the grouping aggregation statement type in the target query statement.
[0069] 302: If the grouping aggregation statement type is a single grouping aggregation statement, then if the target aggregation function does not have a randomly determined value, the execution result of the scalar function will be used as the task result of the target aggregation function's value assignment task.
[0070] 303: If the group aggregation statement type is a multi-group aggregation statement, then any single-group aggregation statement currently being processed is determined from the multiple groups; when the target aggregation function of any single-group aggregation statement has not been determined to take a random value, the execution result of the scalar function is taken as the task result of the value taking task of the target aggregation function.
[0071] As mentioned above, after the above judgment, it can be seen that the relationship between the parameter types contained in the target aggregation function and the target scalar function conforms to the execution optimization rules. Further, the type of grouping aggregation statement in the target query statement will be judged. Generally, there are two types of grouping aggregation statements: single-group aggregation and multi-group aggregation. The physical execution plan generated for different types of grouping aggregation statements is different. The following will explain these two cases separately.
[0072] like Figure 4 This is a schematic diagram illustrating a data processing flow based on an adjusted execution order, as exemplified in an embodiment of this application.
[0073] First, determine whether the target query statement contains single-group aggregation.
[0074] If it is a single-group aggregation, the following steps are performed: First, determine whether there is any remaining unprocessed data in the data group. If there is no unprocessed data, it means that all data in the current data group has been processed, and the current data retrieval task will be terminated.
[0075] The second step is to determine whether the current target aggregation function has found a random value if there is any remaining unprocessed data. If a random value has been found, it means that the target aggregation function has completed the task of obtaining the value of the target data and no longer needs to use the target scalar function to process the target data.
[0076] Third, if the target aggregation function has not yet found a random value, the target scalar function will be pushed up, meaning the execution order of the target scalar function and the target aggregation function will be optimized and adjusted to be executed synchronously. With synchronous execution, the target aggregation function does not need to wait for the target scalar function to finish processing all target data in the data group to obtain the required random value. Specifically, the target scalar function processes any target data in the grouped data to obtain a processing result, then determines whether the result is empty. If so, it returns to continue executing the first step until all target data in the data group has participated in the execution and there is no remaining unexecuted target data. If the execution result is a non-empty value, this non-empty value can be used as the random value for the target aggregation function. This means that the target aggregation function has completed its value-finding task, and the target scalar function no longer needs to perform redundant data processing tasks, effectively improving the efficiency of random value selection for the target aggregation function.
[0077] The fourth step, if it's a multi-group aggregation, will process each group one by one. Specifically, it will determine if there is any unprocessed data remaining in the target data group currently being processed. If there is no unprocessed data, it means that all data in the current target data group has been processed. Then, it will further determine if there are any other data groups. If there are no other data groups, the current data retrieval task will end.
[0078] The fifth step is to determine whether the target aggregation function has found a random value if there is any remaining unprocessed data or if there are other data groups with unprocessed data. If a random value has been found, it means that the target aggregation function has completed the task of obtaining the value of the target data and no longer needs to use the target scalar function to process the target data.
[0079] Step 6: If the target aggregation function has not yet found a random value, the target scalar function will be pushed up, meaning the execution order of the target scalar function and the target aggregation function will be optimized and adjusted to synchronous execution. With synchronous execution, the target aggregation function does not need to wait for the target scalar function to complete its data processing task on all target data in the data group to obtain the required random value. Specifically, the target scalar function processes any target data in the current data group to obtain a processing result, then determines whether the result is empty. If so, it returns to step 4 and continues until all target data in the data group has participated in the execution, and there is no remaining unexecuted target data. If the execution result is a non-empty value, this non-empty value can be used as the random value for the current target data group. This means that the target aggregation function has completed the task of obtaining a value for one data group and needs to continue executing steps 4, 5, and 6 to obtain values for other groups. After completing the task of obtaining values for all data groups, the task of obtaining values for the target aggregation function can be terminated. When performing tasks for each group, the target scalar function no longer needs to perform redundant data processing tasks, which can effectively improve the efficiency of random value selection by the target aggregation function.
[0080] To facilitate understanding, specific examples will be provided below.
[0081] Suppose there is a target query statement 1: Select json_extract(content,'$.country')from(select project,arbitrary(content)FROM new_table). Since there is no groupby keyword in this statement, it can be known that the target query statement is a single-group aggregation, that is, all data are in a default global group, which is equivalent to randomly taking one data from all target data as the random value of the target aggregation function.
[0082] Now suppose there is a target query statement 2: Select project, json_extract(content, '$.country') from(select project, arbitrary(content) FROM new_table group by project). Since this statement contains the keyword group by, we know that this target query statement is a multi-group aggregation, that is, all data are in different data groups. It is equivalent to randomly selecting a data from different groups as the random value of the target aggregation function in that group.
[0083] For details on the value determination process, please refer to [link / reference]. Figure 4 The corresponding embodiments will not be repeated here.
[0084] In one or more embodiments of this application, after receiving the data processing request, the method further includes:
[0085] The target query statement is analyzed and processed to generate an abstract syntax tree;
[0086] Based on the nested function types in the target query statement, the abstract syntax tree is converted into a logical execution plan containing multiple logical nodes;
[0087] Based on the storage relationship between the target data to be processed and the data nodes corresponding to the target query statement, the logical execution plan is optimized to obtain the optimized target logical execution plan.
[0088] After determining the target query, Antlr4 can be used to perform lexical and syntactic analysis on the target query to obtain an Abstract Syntax Tree (AST). Following this, semantic analysis and validity checks are performed on the AST to verify the existence of corresponding metadata for tables, columns, functions, and expressions. The AST is then converted into a logical execution plan, which is a relational algebra expression. Specifically, after obtaining the AST, further processing is needed based on the types of nested functions in the target query (e.g., nested target scalar functions, nested subqueries). This process transforms the AST for different nested function types, resulting in a logical execution plan containing multiple logical nodes.
[0089] It should be noted that in this scheme, logical nodes are used to determine the relationship between the target aggregate function and the target scalar function, as well as the relationship between the target scalar function and other functions, in order to determine whether the execution optimization rules are met.
[0090] By analyzing and processing the target query statement in the above manner, a logical execution plan containing multiple logical nodes is obtained. This logical execution plan can be used to analyze the relationship between the target aggregate function and the target scalar function in the target query statement, thereby providing a reference for whether the execution order of the target aggregate function and the target scalar function can be optimized.
[0091] Furthermore, the obtained logical execution plan can be further optimized, specifically through the following steps:
[0092] Based on the storage relationship between the target data and the data nodes, the logical nodes are divided into local computing stages and global computing stages;
[0093] Based on the storage relationship between the target data and the data nodes, the multiple data nodes are divided into local computing nodes and global computing nodes;
[0094] The target logic execution plan is generated based on the correspondence between the local computing nodes and the local computing stages, and the correspondence between the global computing nodes and the global computing stages.
[0095] In practical applications, when the database contains a large amount of data, database sharding is often used to meet the requirements of fast data processing response and stability. This means that the target data is stored on different nodes. It's easy to understand that transferring target data between different data nodes consumes network resources and introduces latency, affecting processing efficiency. This impact is even more pronounced when the target data volume is large.
[0096] Therefore, we can further analyze and determine the data nodes where each target data resides. Multiple target data within the same data group may be stored in different data nodes. When partitioning data nodes, we don't need to be constrained by the data grouping; instead, we should partition them to improve data processing efficiency. For example, multiple target data from different data groups can be assigned to the same local computing node.
[0097] After determining the storage relationship between the target data and the data nodes, the logical execution plan can be optimized based on the storage relationship. In other words, the logical nodes in the logical execution plan can be divided into local computation stages and global computation stages.
[0098] The computational tasks of the local computation phase are executed on at least one local computing node, while the global computation task is executed on the global computing node. Since there are multiple local computing nodes in the distributed system, parallel execution can be used to perform local computation tasks on multiple local computing nodes, effectively improving the processing efficiency of the target data. The output of the local computation phase is used as the input of the global computation phase, and the corresponding global computation task is executed using the global computing node. This approach effectively improves the efficiency of the distributed system in parallel execution of the task of randomly selecting values for the target aggregation function.
[0099] For example, such as Figure 5 This is a schematic diagram illustrating the generation of a target logic execution plan, as exemplified in an embodiment of this application. From Figure 5As can be seen, the logical execution plan contains multiple logical nodes, each corresponding to a target aggregate function or a target scalar function. The analysis also reveals the relationships between target aggregate functions and target scalar functions, as well as the relationships between target scalar functions and other functions. The system relationship tree diagram shows the relationships between data nodes in the distributed system and the target data stored in each data node. Furthermore, a correspondence can be established between local computation stages and local computation nodes, and between global computation stages and global computation nodes. Based on these correspondences, the logical execution plan can be optimized to obtain the target logical execution plan. This effectively improves the efficiency of the distributed system in parallel execution of tasks involving random value selection of target aggregate functions.
[0100] In one or more embodiments of this application, dividing the multiple data nodes into local computing nodes and global computing nodes according to the storage relationship between the target data and data nodes includes:
[0101] Based on the sharding rules of the target data, determine the data nodes corresponding to each of the target data.
[0102] The logical node used to process the target data located near the data node is submitted to the local computing node; and,
[0103] The calculation results of multiple local computing nodes are sent to the global computing node for global calculation.
[0104] In practical applications, when the amount of target data is large, it can be stored in chunks. That is, the entire target data is divided into multiple smaller data blocks, which are stored on different data nodes. Multiple data nodes work together to complete data storage and assist in subsequent data processing tasks. By using chunking, data processing efficiency can be effectively improved.
[0105] The target data, after being sharded, is distributed across different data nodes. A master node (or coordinating node) can coordinate and manage these data nodes to achieve load balancing. The storage relationships between each target data node and the data nodes can be determined through the data index relationships established in the coordinating node. Preferably, the data node containing the target data can be used as a local computing node to execute the computational tasks of the corresponding logical node. Of course, it's possible that the amount of target data stored in a data node exceeds its processing capacity, requiring assistance from other data nodes. Therefore, the data node closest to the target data can be understood as the data node containing the target data, or it can be another data node determined based on affinity rules; in other words, data nodes with lower data transfer costs are selected as nodes closer to the target data. This approach can further improve the data processing efficiency of local computing while allowing multiple local computing nodes to execute local computational tasks in parallel.
[0106] In one or more embodiments of this application, the step of converting the abstract syntax tree into a logical execution plan containing multiple logical nodes based on the nested function types in the target query statement includes:
[0107] When the nested function type in the target aggregate function in the target query statement is a subquery function, the abstract syntax tree is converted into a logical execution plan containing multiple logical nodes;
[0108] When the nested parameter in the target aggregate function is a scalar function, the scalar function is converted into a subquery statement;
[0109] Based on the target aggregation function containing the nested subquery statements, the abstract syntax tree is converted into a logical execution plan containing multiple logical nodes.
[0110] For example, there exists a target query statement: "select project,arbitrary(json_extract(content,
[0111] The query "'$.country'))FROM new_table group by project" contains a nested target scalar function json_extract(content,'$.country'). This function cannot be directly used by the SQL analysis engine and needs to be converted into a nested subquery, such as: "select project,arbitrary(country)from(select project,json_extract(content,'$.country')as country FROM new_table)temp group by project".
[0112] After the above transformation, we can obtain the abstract syntax tree corresponding to the target query statement and the logical execution plan containing multiple logical nodes. This transformation method converts the target aggregate function nested with different types of functions into nested subqueries, unifying nested functions and facilitating the generation of logical execution plans with identical structures.
[0113] In one or more embodiments of this application, data processing tasks are executed based on an adjusted execution order, including:
[0114] Determine the local computation tasks corresponding to the local computation stages of the target scalar function for the multiple target data processing tasks;
[0115] Multiple local computation tasks are executed in parallel to obtain multiple local computation results;
[0116] Determine the global computation task corresponding to the global computation stage of the task of randomly selecting values from multiple local computation results by the target aggregation function;
[0117] The global computation task is executed to obtain random value data corresponding to the target query statement.
[0118] In practical applications, as a preferred solution, the target data in the data group can be stored in at least one data node, which serves as a local computing node for executing local computation phase tasks. The target scalar function corresponding to the logical node is executed on this local computing node. If the target scalar function obtains a non-null value when performing computation on the target data in this local computing node, this non-null value can be provided to the target aggregation function corresponding to the global computing node as the result of the random value selection task of the target aggregation function.
[0119] By using the above method, when executing the random value selection task of the target aggregation function, by establishing a correspondence between the target scalar function and the target aggregation function after adjusting the execution order and the data nodes in the distributed system, it is possible to achieve synchronous execution of the target scalar function and the target aggregation function, as well as parallel execution of multiple local computing nodes. This can effectively improve the efficiency of the aggregation function in executing the random value selection task from a large amount of target data.
[0120] To facilitate understanding, the implementation process of the technical solution of this application will be illustrated below through specific embodiments. Figure 6 This is a schematic diagram illustrating the random value selection process of an aggregation function, as exemplified in an embodiment of this application.
[0121] This scheme for executing random value selection tasks can be divided into four parts: logical plan node construction, logical phase division, scalar function push-up (adjusting the execution order of scalar functions and aggregate functions), and physical execution plan generation. In this embodiment, the target aggregate function is the arbitrary function as an example for illustration.
[0122] Part 1: Logical Plan Node Construction, used to construct the corresponding logical plan nodes. SQL statements cannot be directly executed by a computer; therefore, they need to be analyzed and transformed to obtain an Abstract Syntax Tree (AST) that can be recognized by a computer.
[0123] Specifically, the Antlr4 tool is used to perform lexical and syntactic analysis on the SQL statement to obtain an Abstract Syntax Tree (AST). Then, based on the types of aggregate functions and nested functions, it is transformed into different logical plan nodes. The AST obtained in this Antlr4 step is the most basic and original node, while the final logical node is the node after AST optimization (transformation).
[0124] Nested aggregate functions can be divided into two categories: one is nested aggregate functions with scalar functions. For example: "select project,arbitrary(json_extract(content,'$.country'))FROM new_table group byproject".
[0125] Another type is nested subqueries using aggregate functions. For example: "select project,arbitrary(country)from(select project,json_extract(content,'$.country')as country FROM new_table)temp group by project".
[0126] Part Two: Logical Plan Decomposition, used to split the logical execution plan into local computation phases and global computation phases. It should be noted that distributed systems with multiple data nodes can perform logical plan decomposition, while single-machine database systems cannot.
[0127] To fully utilize the hardware resources of the distributed system, local computation phases are performed on data nodes closer to the target data shards (e.g., directly accessing the target data on the local data node to avoid unnecessary network overhead) according to the sharding rules for large amounts of target data. The results of the local computation phases are then used as input for the global computation phase. This avoids processing all target data on the same data node, and aims to improve the parallel efficiency of local computations on the target data while ensuring correctness.
[0128] Part 3: Scalar function push-up, which pushes up columns in nested subqueries that belong to the parameters of the arbitrary function.
[0129] Analyze the relationships between nodes before and after the logical execution plan to confirm whether the type of the argument function parameter is an identifier. If the data corresponding to this identifier is obtained by the target data through the execution of the target scalar function, and this scalar function is not referenced by other functions, then this target scalar function can be used to replace the parameter of the argument function.
[0130] Part Four: Physical Execution Plan Generation. This part generates a physical execution plan that executes a random function and an embedded scalar function synchronously.
[0131] If the current target query does not have a `groupby` clause, it indicates that the target query is a single-group aggregation. The system then iterates through the data groups of the target data. If a random value exists, the process ends. If no random value exists, the nested scalar function is executed. If the scalar function's result is not null, this result is used as the random value for the aggregation function. If the scalar function's result is null, execution continues.
[0132] If the current target query does not have a `groupby` clause, it indicates that the target query is a multi-group aggregation. The dataset is then iterated through. If a random value for the current data group exists, execution continues. If a random value for the current data group does not exist, the nested scalar function is executed. If the scalar function's result is not empty, this result is used as the random value for this group. If the scalar function's result is empty, execution continues.
[0133] Based on the same idea, this embodiment also provides a data processing apparatus. For example... Figure 7 This is a schematic diagram of a data processing apparatus provided in an embodiment of this application. Figure 7 As can be seen, the device, applied to the server side, includes:
[0134] The determination module 71 is used to determine the target aggregate function and target scalar function contained in the target query statement in response to a data processing request.
[0135] The judgment module 72 is used to determine whether the relationship between the parameter types contained in the target aggregation function and the target scalar function conforms to the execution optimization rules.
[0136] The adjustment module 73 is used to adjust the execution order of the target aggregate function and the target scalar function in the target query statement if the execution optimization rules are met, so as to execute the data processing task based on the adjusted execution order.
[0137] Optionally, the determination module 72 is used to determine the parameter type of the parameters in the target aggregation function;
[0138] When the parameter type is an identifier, determine whether the identifier is obtained by processing the target data by the target scalar function;
[0139] If the determination result is yes, then determine whether the target scalar function is referenced by other functions besides the target aggregate function;
[0140] If the judgment result is negative, then the association relationship is determined to conform to the execution optimization rule.
[0141] The adjustment module 73 is used to determine the grouping aggregation statement type in the target query statement when the relationship between the parameter types contained in the target aggregation function and the target scalar function conforms to the execution optimization rules.
[0142] If the grouping aggregation statement type is a single grouping aggregation statement, then if the target aggregation function does not have a determined random value, the execution result of the scalar function will be used as the task result of the target aggregation function's value assignment task.
[0143] The adjustment module 73 is used to determine any single-group aggregation statement currently being processed from multiple groups if the group aggregation statement type is a multi-group aggregation statement;
[0144] If the target aggregation function of any single-group aggregation statement does not have a randomly determined value, the execution result of the scalar function is taken as the task result of the target aggregation function's value assignment task.
[0145] Optionally, it also includes a generation module 74 for analyzing and processing the target query statement to generate an abstract syntax tree;
[0146] Based on the nested function types in the target query statement, the abstract syntax tree is converted into a logical execution plan containing multiple logical nodes;
[0147] Based on the storage relationship between the target data to be processed and the data nodes corresponding to the target query statement, the logical execution plan is optimized to obtain the optimized target logical execution plan.
[0148] Optionally, the generation module 74 is used to divide the logical node into a local computing stage and a global computing stage according to the storage relationship between the target data and the data node;
[0149] Based on the storage relationship between the target data and the data nodes, the multiple data nodes are divided into local computing nodes and global computing nodes;
[0150] The target logic execution plan is generated based on the correspondence between the local computing nodes and the local computing stages, and the correspondence between the global computing nodes and the global computing stages.
[0151] Optionally, the generation module 74 is used to determine the data nodes corresponding to each of the target data according to the sharding rules of the target data;
[0152] The logical node used to process the target data located near the data node is submitted to the local computing node; and,
[0153] The calculation results of multiple local computing nodes are sent to the global computing node for global calculation.
[0154] Optionally, the generation module 74 is used to convert the abstract syntax tree into a logical execution plan containing multiple logical nodes when the nested function type in the target aggregate function in the target query statement is a subquery function;
[0155] When the nested parameter in the target aggregate function is a scalar function, the scalar function is converted into a subquery statement;
[0156] Based on the target aggregation function containing the nested subquery statements, the abstract syntax tree is converted into a logical execution plan containing multiple logical nodes.
[0157] Optionally, it also includes an execution module 75, used to determine the local computation tasks corresponding to the local computation stages of the target scalar function execution for the multiple target data processing tasks;
[0158] Multiple local computation tasks are executed in parallel to obtain multiple local computation results;
[0159] Determine the global computation task corresponding to the global computation stage of the task of randomly selecting values from multiple local computation results by the target aggregation function;
[0160] The global computation task is executed to obtain random value data corresponding to the target query statement.
[0161] Based on the same idea, this embodiment also provides a data processing system. For example... Figure 8 This is a schematic diagram of the structure of a data processing system provided in an embodiment of this application. Figure 8 As can be seen, the system includes: multiple data nodes 81 and a server 82.
[0162] Data node 81 is used to store at least one data packet containing the target data and to perform computation tasks.
[0163] Server 82 is used to determine the target aggregate function and target scalar function contained in the target query statement in response to a data processing request;
[0164] Determine whether the relationship between the parameter types contained in the target aggregation function and the target scalar function conforms to the execution optimization rules;
[0165] If the execution optimization rules are met, the execution order of the target aggregate function and the target scalar function in the target query statement is adjusted so that the data processing task is executed based on the adjusted execution order.
[0166] Server 82 is used to determine the parameter type of the parameters in the target aggregation function;
[0167] When the parameter type is an identifier, determine whether the identifier is obtained by processing the target data by the target scalar function;
[0168] If the determination result is yes, then determine whether the target scalar function is referenced by other functions besides the target aggregate function;
[0169] If the judgment result is negative, then the association relationship is determined to conform to the execution optimization rule.
[0170] Server 82 is used to determine the grouping aggregation statement type in the target query statement when the relationship between the parameter types contained in the target aggregation function and the target scalar function conforms to the execution optimization rules.
[0171] If the grouping aggregation statement type is a single grouping aggregation statement, then if the target aggregation function does not have a determined random value, the execution result of the scalar function will be used as the task result of the target aggregation function's value assignment task.
[0172] Server 82 is configured to determine any single-group aggregation statement currently being processed from multiple groups if the group aggregation statement type is a multi-group aggregation statement.
[0173] If the target aggregation function of any single-group aggregation statement does not have a randomly determined value, the execution result of the scalar function is taken as the task result of the target aggregation function's value assignment task.
[0174] Server 82 is used to analyze and process the target query statement and generate an abstract syntax tree;
[0175] Based on the nested function types in the target query statement, the abstract syntax tree is converted into a logical execution plan containing multiple logical nodes;
[0176] Based on the storage relationship between the target data to be processed and the data nodes corresponding to the target query statement, the logical execution plan is optimized to obtain the optimized target logical execution plan.
[0177] Server 82 is used to divide the logical node into a local computing stage and a global computing stage according to the storage relationship between the target data and the data node;
[0178] Based on the storage relationship between the target data and the data nodes, the multiple data nodes are divided into local computing nodes and global computing nodes;
[0179] The target logic execution plan is generated based on the correspondence between the local computing nodes and the local computing stages, and the correspondence between the global computing nodes and the global computing stages.
[0180] Server 82 is used to determine the data nodes corresponding to each piece of target data according to the sharding rules of the target data;
[0181] The logical node used to process the target data located near the data node is submitted to the local computing node; and,
[0182] The calculation results of multiple local computing nodes are sent to the global computing node for global calculation.
[0183] Server 82 is configured to convert the abstract syntax tree into a logical execution plan containing multiple logical nodes when the nested function type in the target aggregate function in the target query statement is a subquery function.
[0184] When the nested parameter in the target aggregate function is a scalar function, the scalar function is converted into a subquery statement;
[0185] Based on the target aggregation function containing the nested subquery statements, the abstract syntax tree is converted into a logical execution plan containing multiple logical nodes.
[0186] Server 82 is configured to determine the local computation tasks corresponding to the local computation stages of the target scalar function execution for the multiple target data processing tasks, respectively.
[0187] Multiple local computation tasks are executed in parallel to obtain multiple local computation results;
[0188] Determine the global computation task corresponding to the global computation stage of the task of randomly selecting values from multiple local computation results by the target aggregation function;
[0189] The global computation task is executed to obtain random value data corresponding to the target query statement.
[0190] One embodiment of this application also provides an electronic device. This electronic device is a master node electronic device in a computing unit. For example... Figure 9 This is a schematic diagram of an electronic device provided in an embodiment of this application. The electronic device includes a memory 901, a processor 902, and a communication component 903; wherein,
[0191] The memory 901 is used to store programs;
[0192] The processor 902, coupled to the memory, is configured to execute the program stored in the memory for:
[0193] In response to a data processing request, determine the target aggregate function and target scalar function contained in the target query statement;
[0194] Determine whether the relationship between the parameter types contained in the target aggregation function and the target scalar function conforms to the execution optimization rules;
[0195] If the execution optimization rules are met, the execution order of the target aggregate function and the target scalar function in the target query statement is adjusted so that the data processing task is executed based on the adjusted execution order.
[0196] Processor 902 is used to determine the parameter type of the parameters in the target aggregation function;
[0197] When the parameter type is an identifier, determine whether the identifier is obtained by processing the target data by the target scalar function;
[0198] If the determination result is yes, then determine whether the target scalar function is referenced by other functions besides the target aggregate function;
[0199] If the judgment result is negative, then the association relationship is determined to conform to the execution optimization rule.
[0200] Processor 902 is used to determine the grouping aggregation statement type in the target query statement when the relationship between the parameter types contained in the target aggregation function and the target scalar function conforms to the execution optimization rule;
[0201] If the grouping aggregation statement type is a single grouping aggregation statement, then if the target aggregation function does not have a determined random value, the execution result of the scalar function will be used as the task result of the target aggregation function's value assignment task.
[0202] Processor 902 is configured to determine any single-group aggregation statement currently being processed from multiple groups if the grouped aggregation statement type is a multi-group aggregation statement;
[0203] If the target aggregation function of any single-group aggregation statement does not have a randomly determined value, the execution result of the scalar function is taken as the task result of the target aggregation function's value assignment task.
[0204] Processor 902 is used to analyze and process the target query statement and generate an abstract syntax tree;
[0205] Based on the nested function types in the target query statement, the abstract syntax tree is converted into a logical execution plan containing multiple logical nodes;
[0206] Based on the storage relationship between the target data to be processed and the data nodes corresponding to the target query statement, the logical execution plan is optimized to obtain the optimized target logical execution plan.
[0207] The processor 902 is used to divide the logical node into a local computing stage and a global computing stage according to the storage relationship between the target data and the data node;
[0208] Based on the storage relationship between the target data and the data nodes, the multiple data nodes are divided into local computing nodes and global computing nodes;
[0209] The target logic execution plan is generated based on the correspondence between the local computing nodes and the local computing stages, and the correspondence between the global computing nodes and the global computing stages.
[0210] Processor 902 is configured to determine the data nodes corresponding to each piece of target data according to the sharding rules of the target data;
[0211] The logical node used to process the target data located near the data node is submitted to the local computing node; and,
[0212] The calculation results of multiple local computing nodes are sent to the global computing node for global calculation.
[0213] Processor 902 is configured to convert the abstract syntax tree into a logical execution plan containing multiple logical nodes when the nested function type in the target aggregate function in the target query statement is a subquery function;
[0214] When the nested parameter in the target aggregate function is a scalar function, the scalar function is converted into a subquery statement;
[0215] Based on the target aggregation function containing the nested subquery statements, the abstract syntax tree is converted into a logical execution plan containing multiple logical nodes.
[0216] Processor 902 is configured to determine the local computation tasks corresponding to the local computation stages of the target scalar function for the execution of multiple target data processing tasks;
[0217] Multiple local computation tasks are executed in parallel to obtain multiple local computation results;
[0218] Determine the global computation task corresponding to the global computation stage of the task of randomly selecting values from multiple local computation results by the target aggregation function;
[0219] The global computation task is executed to obtain random value data corresponding to the target query statement.
[0220] The aforementioned memory 901 can be configured to store various other data to support operation on the electronic device. Examples of such data include instructions for any application or method used to operate on the electronic device. The memory can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory.
[0221] (SRAM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Erasable Programmable Read-Only Memory (EPROM), Programmable Read-Only Memory (PROM), Read-Only Memory (ROM), Magnetic Storage, Flash Memory, Disk or Optical Disk.
[0222] Furthermore, the processor 902 in this embodiment may specifically be a programmable switching processing chip, which is configured with a data copying engine and can copy the received data.
[0223] When the processor 902 executes the program in memory, in addition to the functions described above, it can also perform other functions, as detailed in the descriptions of the preceding embodiments. Furthermore, as... Figure 9As shown, the electronic device also includes other components such as the power supply component 904.
[0224] This application also provides a non-transitory machine-readable storage medium storing executable code. When the executable code is executed by a processor of an electronic device, the processor performs... Figure 1 The method described in the corresponding embodiment.
[0225] Based on the above embodiments, the distributed system includes multiple data nodes and stores multiple rows of target data. After determining the target query statement and target data to be processed, the target query statement is processed to generate an abstract syntax tree containing multiple nodes. The target aggregate function and target scalar function contained in the target query statement are further determined. When the matching relationship between the parameter types in the target aggregate function and the scalar function conforms to the execution optimization rules, the parameters in the target aggregate function are replaced with the target scalar function according to the previously obtained execution optimization rules. Based on the replaced target aggregate function and target scalar function, the task of retrieving the target data is executed synchronously, which can effectively reduce the number of scalar function loop calculations during the sequential execution of the scalar function and the target aggregate function. This can improve the data processing efficiency in the distributed system.
[0226] In addition, when users need to modify application-related functions (for example, when users find that the application has a fault), they can send the modification request to the corresponding console, and then the modification request can be met by modifying the adaptation parameters, without having to adjust the underlying code of the application or cloud operating system, which can effectively improve the application maintenance efficiency.
[0227] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.
[0228] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.
[0229] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.
Claims
1. A data processing method, applied on a server side, the method comprising: In response to a data processing request, determine the random value function and the target scalar function contained in the target query statement; After generating the logical execution plan corresponding to the target query statement, and before generating the physical execution plan corresponding to the target query statement, it is determined whether the parameter type contained in the random value function is an identifier, and whether the target scalar function is not referenced by any other function besides the random value function; If the parameter type in the random value function is an identifier, and the target scalar function is not referenced by any other function besides the random value function, then the parameter of the random value function is replaced by the target scalar function. When generating the physical execution plan corresponding to the target query statement, the following physical execution plan is generated: If the grouping aggregation statement type in the target query statement is a single grouping aggregation statement, then it is determined whether the execution result obtained by the target scalar function in processing any target data in the data group is a non-null value. If it is a non-null value, then the non-null value is used as the random value of the random value function.
2. The method according to claim 1, wherein the physical execution plan further comprises: And if the grouping aggregation statement type is a multi-group aggregation statement, then it is determined whether the execution result obtained by the target scalar function in processing any target data in any data group among the multiple data groups is a non-null value. If it is a non-null value, then the non-null value is used as the random value of the random value function in any data group.
3. The method according to claim 1 or 2, further comprising, after receiving the data processing request: The target query statement is analyzed and processed to generate an abstract syntax tree; Based on the nested function types in the target query statement, the abstract syntax tree is converted into a logical execution plan containing multiple logical nodes; Based on the storage relationship between the target data to be processed and the data nodes corresponding to the target query statement, the logical execution plan is optimized to obtain the optimized target logical execution plan.
4. The method according to claim 3, wherein optimizing the logical execution plan based on the storage relationship between the target data to be processed and the data nodes corresponding to the target query statement to obtain an optimized target logical execution plan includes: Based on the storage relationship between the target data and the data nodes, the logical nodes are divided into local computing stages and global computing stages; Based on the storage relationship between the target data and the data nodes, the multiple data nodes are divided into local computing nodes and global computing nodes; The target logic execution plan is generated based on the correspondence between the local computing nodes and the local computing stages, and the correspondence between the global computing nodes and the global computing stages.
5. The method according to claim 4, wherein dividing the plurality of data nodes into local computing nodes and global computing nodes according to the storage relationship between the target data and data nodes includes: Based on the sharding rules of the target data, determine the data nodes corresponding to each of the target data. The logical node used to process the target data located near the data node is submitted to the local computing node; as well as, The calculation results of multiple local computing nodes are sent to the global computing node for global calculation.
6. The method according to claim 3, wherein converting the abstract syntax tree into a logical execution plan containing multiple logical nodes based on the nested function types in the target query statement includes: When the nested function type in the random value selection function in the target query statement is a subquery function, the abstract syntax tree is converted into a logical execution plan containing multiple logical nodes; When the nested parameter in the random value selection function is a scalar function, the scalar function is converted into a subquery statement; Based on the random value function containing the nested subquery statements, the abstract syntax tree is converted into a logical execution plan containing multiple logical nodes.
7. The method according to claim 4, wherein the data processing task is executed based on the adjusted execution order, comprising: Determine the local computation tasks corresponding to the local computation stages of the target scalar function for the multiple target data processing tasks; Multiple local computation tasks are executed in parallel to obtain multiple local computation results; Determine the global computation task corresponding to the global computation stage of the task where the random value selection function randomly selects values from multiple local computation results; The global computation task is executed to obtain random value data corresponding to the target query statement.
8. A data processing apparatus, the apparatus comprising: The determination module is used to determine the random value function and the target scalar function contained in the target query statement in response to a data processing request; The judgment module is used to determine, after generating the logical execution plan corresponding to the target query statement and before generating the physical execution plan corresponding to the target query statement, whether the parameter type contained in the random value selection function is an identifier, and whether the target scalar function is referenced by other functions besides the random value selection function; The adjustment module is configured to replace the parameters of the random value function with the target scalar function if the parameter type of the random value function is an identifier and the target scalar function is not referenced by any other function besides the random value function. Furthermore, when generating the physical execution plan corresponding to the target query statement, the module generates a physical execution plan as follows: if the grouping aggregation statement type in the target query statement is a single-group aggregation statement, it determines whether the execution result obtained by the target scalar function in processing any target data in the data group is a non-null value. If it is a non-null value, it uses this non-null value as the random value of the random value function.
9. A data processing system, the system comprising: Data nodes are used to store at least one data packet containing the target data and to perform computational tasks. A server for performing the method according to any one of claims 1 to 7.
10. An electronic device, comprising a memory and a processor; wherein, The memory is used to store programs; The processor, coupled to the memory, is configured to execute the program stored in the memory for implementing the method of any one of claims 1 to 7.
11. A non-transitory machine-readable storage medium storing executable code that, when executed by a processor of an electronic device, causes the processor to perform the method as described in any one of claims 1 to 7.