Data processing method and apparatus, log query method and apparatus, and electronic device

Abstract

The present disclosure relates to the technical field of data processing. Provided are a data processing method and apparatus, a log query method and apparatus, and an electronic device. The data processing method comprises: firstly, acquiring a data processing request for a database, and determining a filtering node and a mapping node which correspond to the data processing request; then, on the basis of the filtering node and the mapping node, generating a binary bytecode; and finally, on the basis of the binary bytecode, processing data in the database, so as to obtain a processing result corresponding to the data processing request.

Description

[0001]TECHNICAL FIELD The present disclosure relates to the field of data processing technology, and more particularly to a data processing method, log query method, device, and electronic device. Background: Filtering and mapping operators are fundamental tools for implementing database queries. By combining them, various complex query operations can be implemented. These operators are the two most frequently used computational operators in databases, and column-based processing offers inherent advantages over row-based processing in the field of distributed online analytical processing (OLAP). Currently, computations using these operators involve numerous virtual function calls, which typically require access to a virtual function table (a table that stores virtual function addresses). This process incurs additional overhead and compromises computing performance. To avoid virtual function calls, a string concatenation-based code generation approach is employed, generating code at compile time rather than executing operations through virtual function calls at runtime. This approach avoids the overhead associated with virtual function calls, but requires compilation and packaging for execution, resulting in a complex compilation process that cannot meet real-time requirements. SUMMARY OF THE INVENTION Embodiments of the present disclosure provide a data processing method, a log query method, an apparatus, and an electronic device to achieve real-time performance while improving computing performance. In a first aspect, embodiments of the present disclosure provide a data processing method, comprising: obtaining a data processing request for a database and determining a filter node and a mapping node corresponding to the data processing request; generating binary bytecode based on the filter node and the mapping node; and processing data in the database based on the binary bytecode to obtain a processing result corresponding to the data processing request. In a second aspect, embodiments of the present disclosure provide a log query method, comprising: obtaining a log query request for a distributed database and determining a filter node and a mapping node corresponding to the log query request; generating binary bytecode based on the filter node and the mapping node; and querying log data in the distributed database based on the binary bytecode to obtain a query result corresponding to the log query request. In a third aspect, an embodiment of the present disclosure provides a data processing device, comprising: a determination module configured to obtain a data processing request for a database and determine a filter node and a mapping node corresponding to the data processing request; a generation module configured to generate a binary bytecode based on the filter node and the mapping node; and a processing module configured to process the data in the database according to the binary bytecode to obtain a processing result corresponding to the data processing request.In a fourth aspect, embodiments of the present disclosure provide an electronic device comprising a memory, a processor, and a computer program stored in the memory. When the processor executes the computer program, it implements any of the aforementioned methods. In a fifth aspect, embodiments of the present disclosure provide a computer-readable storage medium storing the computer program. When the processor executes the computer program, it implements any of the aforementioned methods. In a sixth aspect, embodiments of the present disclosure provide a computer program product comprising the computer program. When the processor executes the computer program, it implements any of the aforementioned methods. Compared to the prior art, the present disclosure has the following advantages: The present disclosure provides a data processing method, apparatus, and electronic device. First, a data processing request for a database is obtained, and the filter node and mapping node corresponding to the data processing request are determined; then, binary bytecode is generated based on the filter node and mapping node; finally, data in the database is processed based on the binary bytecode to obtain a processing result corresponding to the data processing request. In an embodiment of the present disclosure, binary bytecode is generated based on filter nodes and mapping nodes, and data in a database is processed based on the binary bytecode. This binary bytecode-based code generation effectively avoids virtual function calls, eliminates the need for overall compilation and packaging, and improves real-time performance. This disclosure provides a log query method. First, a log query request for a distributed database is obtained, and the filter nodes and mapping nodes corresponding to the log query request are determined. Then, binary bytecode is generated based on the filter nodes and mapping nodes. Finally, the log data in the distributed database is queried based on the binary bytecode to obtain query results corresponding to the log query request. In this embodiment, binary bytecode is generated based on filter nodes and mapping nodes, and log data in the database is queried based on the binary bytecode. This binary bytecode-based code generation effectively avoids virtual function calls, eliminates the need for overall compilation and packaging, and improves the real-time performance of log analysis services. The above description is merely an overview of the technical solutions of the present disclosure. To better understand the technical solutions of the present disclosure, implementation is possible in accordance with the contents of this specification. To further enhance the understanding of the aforementioned and other objectives, features, and advantages of the present disclosure, specific embodiments of the present disclosure are described below. In the accompanying drawings, unless otherwise specified, identical reference numerals throughout the various figures denote identical or similar components or elements. The drawings are not necessarily drawn to scale. It should be understood that these drawings depict only some embodiments of the present disclosure and should not be construed as limiting the scope of the present disclosure. Figure 1 is a flow chart illustrating an application scenario of the data processing method provided by the present disclosure.Figure 2 is a flowchart of a data processing method according to an embodiment of the present disclosure. Figure 3 is a flowchart of a log query method according to an embodiment of the present disclosure. Figure 4 is a block diagram of a data processing device according to an embodiment of the present disclosure. Figure 5 is a block diagram of an electronic device used to implement an embodiment of the present disclosure. DETAILED DESCRIPTION OF THE EMBODIMENTS The following briefly describes certain exemplary embodiments. As those skilled in the art will appreciate, the described embodiments may be modified in various ways without departing from the spirit or scope of the present disclosure. Therefore, the drawings and descriptions are to be considered illustrative in nature, rather than restrictive. To facilitate understanding of the technical solutions of the embodiments of the present disclosure, the following describes related technologies of the embodiments of the present disclosure. The following related technologies are optional solutions that can be combined with the technical solutions of the embodiments of the present disclosure in any manner and fall within the scope of protection of the embodiments of the present disclosure. It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for analysis, storage, and display, etc.) involved in this disclosure are all information and data authorized by the user or fully authorized by all parties. The collection, use, and processing of the relevant data must comply with the relevant laws, regulations, and standards of the relevant countries and regions, and corresponding operation portals are provided for users to choose to authorize or reject. Figure 1 is a flowchart of an application scenario of the data processing method provided by the present disclosure. In this embodiment, the data processing request is a Structured Query Language (SQL) request. As shown in Figure 1, the data processing method includes: Step S101: Obtaining a SQL request and performing grammatical analysis on the SQL request to obtain an Abstract Syntax Tree (AST). Specifically, obtaining a SQL request for a database, performing lexical and syntactic analysis on the SQL request using a language recognition tool, and obtaining an AST. An AST is a tree representation of the abstract syntax structure of source code written in a programming language. Each node in the tree represents a structure in the source code, such as an expression, a declaration, or a control flow statement. Step S102: Generating filter nodes and mapping nodes based on the AST. Specifically, determining a logical plan corresponding to the SQL request based on the AST. The logical plan includes various types of nodes, such as filter nodes, mapping nodes, and aggregation nodes. Extracting the filter nodes and mapping nodes from the logical plan.In database queries, filtering operations typically correspond to conditional expressions in the source code, such as a comparison operation (a > b) or a logical operation (a && b). In an abstract syntax tree, conditional expressions are typically represented as specific types of nodes, such as a comparison node or a logical node, both of which are filter nodes. Mapping operations typically correspond to function calls or arithmetic expressions in the source code, such as an addition operation (a + b) or a function call (f(a)). In an abstract syntax tree, these operations are typically represented as specific types of nodes, such as an addition node or a function call node, both of which are map nodes. Step S103: Determine multiple local computation phases and multiple global computation phases. The logical plan is split into multiple local computation phases and multiple global computation phases based on the characteristics of the computation steps. Local computation phases include computation steps that can be executed independently on a single node or data partition without exchanging information with other nodes. Global computation phases include computation steps that require cross-node collaboration. These steps may involve data redistribution, cross-node aggregation, or sorting. In actual computing, to fully utilize the hardware resources of a distributed system, the database data is sharded and stored on multiple first physical nodes. The second physical node closest to each first physical node is determined. For the data stored in any first physical node, a local computation phase is performed on the corresponding second physical node. The results of these local computation phases are then used as input for the global computation phase, which is then executed. This prevents all data from being processed on the same physical node, improving parallelism while ensuring correctness. During the execution of multiple local and global computation phases, it is necessary to determine whether the filter nodes and mapping nodes are continuous or independent. If the filter nodes and mapping nodes are continuous, step S104 is executed; otherwise, step S105 is executed. In step S104, the filter nodes are converted into filter operators, the mapping nodes are converted into mapping operators, and the filter and mapping operators are merged to obtain a merged operator. If the filter nodes and mapping nodes are continuous, that is, during the execution of the logical plan, the filter nodes and mapping nodes are executed sequentially. Convert a filter node into a filter operator. Specifically, parse the filter node to obtain the filter condition, and construct a filter operator based on the filter condition. The filter operator is used to implement the filter operation. Convert a map node into a map operator. Specifically, parse the map node to determine the specific content of the map operation, and construct a map operator based on the specific content of the map operation. The map operator is used to implement the map operation.Then, the filter operator and the mapping operator are merged to obtain a merged operator. Merging consecutive filter and mapping operators avoids the need for intermediate result construction, thereby improving execution efficiency. In step S105, the filter node is converted into a filter operator, and the mapping node is converted into a mapping operator. If the filter node and the mapping node are independent, the filter node and the mapping node are converted into corresponding operators, respectively. Steps S104 and S105 above describe the process of converting logical nodes into physical operators. In step S106, binary bytecode is generated based on the merged operator, the filter operator, and the mapping operator. If the filter node and the mapping node are consecutive, binary bytecode is generated for the merged operator. If the filter node and the mapping node are independent, binary bytecode is generated for the filter operator and the mapping operator, respectively. Optionally, binary bytecode generation technology, ASM (Assembly), can be used to generate binary bytecode. ASM is a Java bytecode manipulation and analysis framework that can be used to modify existing classes or dynamically generate new classes. Because filtering and mapping have a certain degree of randomness, bytecode generation technology (ASM) is employed to generate bytecode based on the expressions of the filter and mapping operators and dynamically load it into the virtual machine. This enhances flexibility while avoiding virtual function calls. Furthermore, the binary bytecode can be cached regularly, avoiding repeated compilation and loading of expressions while ensuring memory utilization, thus meeting the real-time requirements of the distributed system. In step S107, if the binary bytecode determines that the mapping and filtering processes are preceded by a preceding one, filtering is performed on the data in the database, and mapping is performed on the filtered data. If filtering occurs before mapping, the database data set is traversed to determine whether the current row meets the filtering conditions. If so, the row number is recorded in a preset array, and mapping is performed based on the row number recorded in the preset s array. Rows that do not meet the filtering conditions are not mapped. If the filtering operators are independent, the filtering results are directly output. If the mapping operators are independent, the mapping results are directly output. During the calculation process, if the data column being processed is a regular column (non-dictionary column), the expression corresponding to the binary bytecode is calculated for each row of data, and the result is stored in the result column corresponding to the expression. If the data column being processed is a dictionary column, the valid dictionary row is determined based on the dictionary column. Based on the binary bytecode and the valid dictionary row, the processing result corresponding to the SQL request is determined. Dictionary columns are a column-level data compression technology that uses a dictionary to store all unique values ​​in the column and replaces the original values ​​in the data column with the indexes of these unique values.Because indexes typically occupy less space than raw values ​​and are more suitable for fast lookup and comparison operations, valid dictionary rows are data rows that require further calculation, for example, rows in a dictionary column that meet the filter criteria. This includes the following scenarios: If the data column to be processed is a single dictionary column, valid dictionary rows are first calculated, and the calculation results are cached in the one-dimensional result array (dict inner result array). The calculation results for each row can be obtained through two indexing operations. Two-indexing operation means first obtaining the row number of the valid dictionary row corresponding to the current row, and then obtaining the corresponding expression result based on the row number of the valid dictionary row. The corresponding expression calculation is only performed on valid dictionary rows, reducing the amount of data processing and thus improving performance. If the data column to be processed consists of multiple dictionary columns, all of which are dictionary columns. First, the product of the number of valid dictionary rows for all columns is calculated to see if it is less than a preset threshold (for example, the number of rows on the page containing the valid dictionary rows multiplied by 70%). If so, the result corresponding to the Cartesian product of the valid dictionary rows for all columns is cached in the multidimensional result array dict inner result array. The result for each row can be retrieved through multiple indexing. Executing the corresponding expression on the Cartesian product of the valid dictionary rows can reduce the amount of data processed, thereby improving performance. If the data to be processed consists of multiple columns and the product of the number of valid dictionary rows for all columns is greater than the preset threshold, or if some columns are dictionary columns and some are non-dictionary columns, the maximum subexpression corresponding to the dictionary columns is first calculated. Then, for each dictionary column, the dictionary value of the maximum subexpression corresponding to the valid dictionary rows is pre-calculated. Finally, the complete expression value is calculated using the pre-calculated dictionary value of the maximum subexpression. Specifically, the maximum subexpression is replaced with its dictionary value in the complete expression. For the remaining portion of the complete expression, the dictionary value corresponding to the dictionary column or the value of the non-dictionary column is calculated. Compared to directly computing the full expression, this approach reduces data processing and improves performance. For each dictionary column, query expressions involving that column are extracted. These query expressions are the largest subexpressions corresponding to the dictionary column—that is, expressions that can be calculated independently without considering other columns.For example, the expressions obtained according to the SQL request are regexp_like(server_protocol, '\d+') and regexp_like(lower(level), '.*(info)|(error).*') and date_trunc('day', time - 2023-09-28 00:00:00', where serverjprotocol and level are dictionary columns. The maximum subexpressions extracted are regexp_like(server_protocol, '\d+') and regexp_like(lower(level), '.*(info)|(error).*'). The dictionary values ​​of regexp_like(serverj)protocol, '\d+') and regexp_like(lower(level), ' . * (info) | (error).*') are calculated. Using the dictionary values, the expression is calculated as regexp_like(serverj)protocol, '\d+') and regexp_like(lower(level), '.*(info)|(error).*') and date_trunc('day', > time - 2023-09-28 00:00:00'. In this embodiment, code generation using bytecode mode effectively avoids virtual function calls, does not require overall compilation and packaging, can be dynamically loaded, and has good real-time performance. Continuous filter operators and mapping operators are merged to avoid intermediate result construction and improve execution efficiency. Valid data in dictionary columns is processed to avoid repeated processing of the same data and error processing of invalid data, thereby shortening processing time. An embodiment of the present disclosure provides a data processing method, which can be applied to a computing device, and the computing device can include: a server, a user terminal, etc. FIG2 is a flowchart of a data processing method according to an embodiment of the present disclosure, comprising: Step S201, obtaining a data processing request for a database, and determining a filter node and a mapping node corresponding to the data processing request. Step S202, generating a binary bytecode based on the filter node and the mapping node. Step S203, according to the binary bytecode, Process the data in the database to obtain a processing result corresponding to the data processing request. The data processing request for the database may include, but is not limited to, a request to implement functions such as query, insert, delete, or update, such as an SQL request.A filter node is a logical node that implements a filtering operation, while a mapping node is a logical node that implements a mapping operation. The data processing method provided in the disclosed embodiments first obtains a data processing request for a database and determines the filter node and mapping node corresponding to the data processing request. Then, binary bytecode is generated based on the filter node and mapping node. Finally, data in the database is processed based on the binary bytecode to obtain a processing result corresponding to the data processing request. In the disclosed embodiments, binary bytecode is generated based on the filter node and mapping node, and data in the database is processed based on the binary bytecode. Code generation in the binary bytecode model effectively avoids virtual function calls, eliminates the need for overall compilation and packaging, and improves real-time performance. The specific implementation process of each of the above steps is described below using multiple implementations. In one implementation, step S201 determines the filter node and mapping node corresponding to the data processing request, including: the data processing request includes a Structured Query Language (SQL) request, performs syntax analysis on the SQL request to obtain an abstract syntax tree, and determines the filter node and mapping node based on the abstract syntax tree. In practical applications, the logical plan corresponding to an SQL request is determined based on the abstract syntax tree. The logical plan includes various types of nodes, such as filter nodes, map nodes, and aggregate nodes. Filter nodes and map nodes are then extracted from the plan. In database queries, filter operations typically correspond to conditional expressions in the source code, such as a comparison operation (a > b) or a logical operation (a && b). In the abstract syntax tree, conditional expressions are typically represented as specific types of nodes, such as a comparison node or a logical node, both of which are filter nodes. Map operations typically correspond to function calls or arithmetic expressions in the source code, such as an addition operation (a + b) or a function call (f(a)). In the abstract syntax tree, these operations are typically represented as specific types of nodes, such as an addition node or a function call node, both of which are map nodes. In one implementation, step S202 generates binary bytecode based on the filter node and the map node. This includes: if the filter node and the map node are consecutive, converting the filter node into a filter operator, converting the map node into a map operator, merging the filter operator and the map operator, and generating binary bytecode based on the merged operator. If the filter node and the map node are consecutive, that is, during the execution of the logical plan, the filter node and the map node are executed sequentially.The filter node is converted into a filter operator. Specifically, the filter node is parsed to obtain a filter condition. A filter operator is constructed based on the filter condition. The filter operator is used to implement the filter operation. The mapping node is converted into a mapping operator. Specifically, the mapping node is parsed to determine the specific content of the mapping operation. A mapping operator is constructed based on the specific content of the mapping operation. The mapping operator is used to implement the mapping operation. The filter operator and the mapping operator are then merged to obtain a merged operator. In this embodiment, the merging of consecutive filter and mapping operators avoids the need for constructing intermediate results, thereby improving execution efficiency. In one implementation, step S203 processes the data in the database based on the binary bytecode to obtain a processing result corresponding to the data processing request. This includes: if the binary bytecode determines that a filtering process precedes the mapping process, filtering the data in the database and recording the identifiers of the data obtained by the filtering process in a preset array; and performing a mapping process based on the identifiers to obtain a processing result corresponding to the data processing request. In practical applications, if the filter node and the mapping node are continuous and have a preceding and following relationship, that is, the mapping process precedes the filtering process, the data set is traversed to determine whether the current row meets the filtering conditions. If so, the row number is recorded in a preset array, and the mapping process is performed based on the row number recorded in the preset array. For rows that do not meet the filtering conditions, no mapping process is performed. In one implementation, step S202 generates binary bytecode based on the filter node and the mapping node, including: if the filter node and the mapping node are independent, converting the filter node into a filter operator, converting the mapping node into a mapping operator, and generating binary bytecode corresponding to the filter operator and the mapping operator, respectively. If the filter node and the mapping node are independent, converting the filter node and the mapping node into corresponding operators, respectively. In one implementation, step S203 processes the data in the database according to the binary bytecode to obtain a processing result corresponding to the data processing request, including: performing filtering or mapping processing on the data in the database according to the binary bytecode; and determining the filtered or mapped data as the processing result corresponding to the data processing request. If the binary bytecode implements a filtering operation and the filtering operator is independent, the filtering result is directly output. If the binary bytecode implements a mapping operation and the mapping operator is independent, the mapping result is directly output.In one implementation, step S203 processes the data in the database based on the binary bytecode to obtain a processing result corresponding to the data processing request. This includes: if the data column corresponding to the binary bytecode in the database is a dictionary column, determining valid dictionary rows based on the dictionary column; and determining the processing result corresponding to the data processing request based on the binary bytecode and the valid dictionary rows. The data column corresponding to the binary bytecode in the database is the data column to be processed, i.e., the data column to be filtered or mapped. During the calculation process, if the data column to be processed is a normal column (non-dictionary column), the expression corresponding to the binary bytecode is calculated for each row of data, and the result is stored in the result column corresponding to the expression. If the data column to be processed is a dictionary column, valid dictionary rows are determined based on the dictionary column (valid dictionary rows are data rows that require further calculation, for example, data rows that meet the filtering conditions), and calculation is performed on the valid dictionary rows. In this embodiment, processing valid data in the dictionary column avoids repeated processing of identical data and erroneous processing of invalid data, thereby shortening processing time. In one implementation, determining a processing result corresponding to a data processing request based on a binary bytecode and a valid dictionary row includes: If the number of dictionary columns is one, processing the valid dictionary row using the binary bytecode to obtain a processing result corresponding to the data processing request. Determining the data column to be processed and the operation to be performed on the processed data column based on the binary bytecode. If the data column to be processed is one column and this column is a dictionary column, first calculating the valid dictionary row, then caching the calculation result in a one-dimensional result array. The calculation result for each row can be obtained through double indexing. Double indexing refers to first obtaining the row number of the valid dictionary row corresponding to the current row, and then obtaining the corresponding expression result based on the row number of the valid dictionary row. The corresponding expression calculation is performed only on the valid dictionary row, reducing the amount of data processing and thereby improving performance. In one implementation, determining a processing result corresponding to a data processing request based on binary bytecodes and valid dictionary rows includes: if there are multiple dictionary columns and the product of the valid dictionary rows is less than a preset threshold, calculating the Cartesian product of the valid dictionary rows; and processing the Cartesian product using the binary bytecode to obtain a processing result corresponding to the data processing request. Determining the data column to be processed and the operation to be performed on the processed data column based on the binary bytecode. If there are multiple columns to be processed, all of which are dictionary columns, first calculate whether the product of the number of valid dictionary rows for all columns is less than a preset threshold. If so, cache the calculation result corresponding to the Cartesian product of the valid dictionary rows for all columns in a multidimensional result array. The calculation result for each row can be obtained by multiple indexing locations.Executing the corresponding expression on the Cartesian product of valid dictionary rows can reduce the amount of data processed, thereby improving performance. In one implementation, determining the processing result corresponding to the data processing request based on the binary bytecode and valid dictionary rows includes: if there are multiple dictionary columns and the product of the valid dictionary rows is greater than or equal to a preset threshold, determining the dictionary value of the maximum subexpression corresponding to the valid dictionary row based on the maximum subexpression corresponding to the dictionary column; and determining the processing result corresponding to the data processing request based on the binary bytecode and the dictionary value of the maximum subexpression corresponding to the valid dictionary row. Determining the data column to be processed and the operation to be performed on the data column to be processed based on the binary bytecode. If there are multiple columns to be processed and the product of the number of valid dictionary rows for all columns is greater than a preset threshold (for example, the number of rows on the page where the valid dictionary row is located multiplied by 70%). oFirst, the maximum subexpression corresponding to the dictionary column is calculated. Then, for each dictionary column, the dictionary value of the maximum subexpression corresponding to the valid dictionary row is pre-calculated. Finally, the complete expression value is calculated using the pre-calculated dictionary value of the maximum subexpression. Specifically, the maximum subexpression is replaced with its dictionary value in the complete expression. For the remaining portion of the complete expression, the dictionary value corresponding to the dictionary column is calculated. Compared with directly calculating the complete expression value, this reduces the amount of data processing and achieves improved performance. In one implementation, before processing data in a database according to the binary bytecode to obtain a processing result corresponding to the data processing request, the data processing method further includes: determining a local computation phase and a global computation phase based on the data processing request; sharding the data in the database, storing the sharded data in multiple first physical nodes, and determining the second physical node closest to each of the multiple first physical nodes; and processing the data in the database according to the binary bytecode to obtain a processing result corresponding to the data processing request, including: executing a local computation phase in a second physical node corresponding to any first physical node according to the binary bytecode for the data stored in the first physical node; executing a global computation phase based on the computation results of the multiple local computation phases, and using the computation result of the global computation phase as the processing result corresponding to the data processing request. During the computation process, the logical plan is split into multiple local computation phases and multiple global computation phases. To fully utilize the hardware resources of a distributed system, the data in the database is sharded and stored on multiple first physical nodes. The second physical node closest to each first physical node is determined. For the data stored in any first physical node, a local computation phase is performed on the corresponding second physical node. The results of the multiple local computation phases are then used as input for the global computation phase, which is then executed. This prevents all data from being processed on the same physical node, improving parallelism while ensuring correctness. Applying the above data processing method to log query scenarios, embodiments of the present disclosure provide a log query method. The method in this embodiment can be applied to computing devices, such as servers and user terminals. Figure 3 is a flowchart of the log query method according to one embodiment of the present disclosure, which includes the following steps: Step S301: Obtain a log query request for a distributed database and determine the filter node and mapping node corresponding to the log query request. Step S302: Generate binary bytecode based on the filter node and mapping node.In step S303, log data in the distributed database is queried based on the binary bytecode to obtain a query result corresponding to the log query request. The specific implementation process of each of the above steps is similar to the process in the aforementioned data processing method embodiment and will not be repeated here. In this embodiment, binary bytecode is generated based on the filter nodes and mapping nodes, and log data in the database is queried based on the binary bytecode. Code generation in the binary bytecode model effectively avoids virtual function calls, eliminates the need for overall compilation and packaging, and improves the real-time performance of the log analysis service. Corresponding to the application scenarios and methods provided by the embodiments of the present disclosure, the embodiments of the present disclosure also provide a data processing device. Figure 4 shows a block diagram of the data processing device according to one embodiment of the present disclosure. The device includes: a determination module 401, configured to obtain a data processing request for a database and determine the filter nodes and mapping nodes corresponding to the data processing request; a generation module 402, configured to generate binary bytecode based on the filter nodes and mapping nodes; and a processing module 403, configured to process the data in the database based on the binary bytecode to obtain a processing result corresponding to the data processing request. The data processing device provided in the embodiments of the present disclosure first obtains a data processing request for a database and determines the filter node and mapping node corresponding to the data processing request. Then, based on the filter node and mapping node, binary bytecode is generated. Finally, the data in the database is processed based on the binary bytecode to obtain a processing result corresponding to the data processing request. In the embodiments of the present disclosure, binary bytecode is generated based on the filter node and mapping node, and the database data is processed based on the binary bytecode. Code generation in binary bytecode mode effectively avoids virtual function calls, eliminates the need for overall compilation and packaging, and improves real-time performance. In one implementation, generation module 402 is configured to: if a filter node and a mapping node are consecutive, convert the filter node into a filter operator, convert the mapping node into a mapping operator, merge the filter operator and the mapping operator, and generate binary bytecode based on the merged operators. In one implementation, the processing module 403 is configured to: if it is determined based on the binary bytecode that filtering processing exists before the mapping processing, perform filtering processing on the data in the database, and record the identifier of the data obtained by the filtering processing into a preset array; and perform mapping processing based on the identifier to obtain a processing result corresponding to the data processing request.In one implementation, generation module 402 is configured to: if a filter node and a mapping node are independent, convert the filter node into a filter operator, convert the mapping node into a mapping operator, and generate binary bytecodes corresponding to the filter operator and the mapping operator, respectively. In one implementation, processing module 403 is configured to: perform filtering or mapping processing on data in a database based on the binary bytecode; and determine the filtered or mapped data as a processing result corresponding to a data processing request. In one implementation, processing module 403 is configured to: if the data column corresponding to the binary bytecode in the database is a dictionary column, determine a valid dictionary row based on the dictionary column; and determine a processing result corresponding to the data processing request based on the binary bytecode and the valid dictionary row. In one implementation, when determining the processing result corresponding to the data processing request based on the binary bytecode and the valid dictionary row, processing module 403 is configured to: if there is only one dictionary column, process the valid dictionary row using the binary bytecode to obtain a processing result corresponding to the data processing request. In one implementation, when determining the processing result corresponding to the data processing request based on the binary bytecode and valid dictionary rows, processing module 403 is configured to: if there are multiple dictionary columns and the product of the valid dictionary rows is less than a preset threshold, calculate the Cartesian product of the valid dictionary rows; and process the Cartesian product using the binary bytecode to obtain the processing result corresponding to the data processing request. In one implementation, when determining the processing result corresponding to the data processing request based on the binary bytecode and valid dictionary rows, processing module 403 is configured to: if there are multiple dictionary columns and the product of the valid dictionary rows is greater than or equal to a preset threshold, determine the dictionary value of the maximum sub-expression corresponding to the valid dictionary row based on the maximum sub-expression corresponding to the dictionary column; and determine the processing result corresponding to the data processing request based on the binary bytecode and the dictionary value of the maximum sub-expression corresponding to the valid dictionary row.In one implementation, the apparatus is further configured to: determine a local computation phase and a global computation phase based on the data processing request before processing the data in the database according to the binary bytecode to obtain a processing result corresponding to the data processing request; shard the data in the database, store the sharded data in multiple first physical nodes, and determine the second physical node closest to each of the multiple first physical nodes; and the processing module 403 is configured to: perform a local computation phase on the data stored in any first physical node according to the binary bytecode in the second physical node corresponding to the first physical node; perform a global computation phase based on the computation results of the multiple local computation phases, and use the computation result of the global computation phase as the processing result corresponding to the data processing request. In one implementation, when determining the filter node and mapping node corresponding to the data processing request, the determination module 401 is configured to: if the data processing request includes a structured query language (SQL) request, perform syntax analysis on the SQL request to obtain an abstract syntax tree; and determine the filter node and mapping node based on the abstract syntax tree. The functions of each module in the embodiments of the present disclosure can be found in the corresponding descriptions of the above-mentioned methods, and they have corresponding beneficial effects, so they will not be repeated here. Figure 5 is a block diagram of an electronic device used to implement the embodiments of the present disclosure. As shown in Figure 5, the electronic device includes a memory 510 and a processor 520. The memory 510 stores a computer program that can be executed by the processor 520. When the processor 520 executes the computer program, the method in the above-mentioned embodiment is implemented. The number of memory 510 and processor 520 can be one or more. The electronic device also includes a communication interface 530 for communicating with external devices and exchanging data. If the memory 510, processor 520, and communication interface 530 are implemented independently, the memory 510, processor 520, and communication interface 530 can be interconnected via a bus to enable communication between them. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, for example. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of illustration, FIG5 shows only one thick line, but this does not mean that there is only one bus or one type of bus.Optionally, in a specific implementation, if the memory 510, processor 520, and communication interface 530 are integrated on a single chip, the memory 510, processor 520, and communication interface 530 can communicate with each other via an internal interface. Embodiments of the present disclosure provide a computer-readable storage medium storing a computer program. When executed by a processor, the program implements the methods provided in the embodiments of the present disclosure. Embodiments of the present disclosure provide a computer program product comprising a computer program. When executed by a processor, the computer program implements the methods provided in the embodiments of the present disclosure. Embodiments of the present disclosure also provide a chip comprising a processor configured to retrieve and execute instructions stored in a memory, thereby enabling a communication device equipped with the chip to perform the methods provided in the embodiments of the present disclosure. Embodiments of the present disclosure also provide a chip comprising an input interface, an output interface, a processor, and a memory. The input interface, the output interface, the processor, and the memory are connected via an internal connection path. The processor is configured to execute code in the memory. When the code is executed, the processor performs the methods provided in the embodiments of the present disclosure. It should be understood that the processor described above may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor. It is worth noting that the processor may be a processor supporting the Advanced RISC Machines (ARM) architecture. Furthermore, optionally, the memory described above may include read-only memory and random access memory. The memory may be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory.Non-volatile memory may include read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory may include random access memory (RAM), which serves as an external cache. By way of example and not limitation, many forms of RAM are available. For example, static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link DRAM (SLDRAM) and direct rambus RAM (DR RAM). oThe above embodiments can be implemented in whole or in part via software, hardware, firmware, or any combination thereof. When implemented using software, they can be implemented in whole or in part in the form of a computer program product. A computer program product comprises one or more computer instructions. When the computer program instructions are loaded and executed on a computer, they fully or partially generate the processes or functions according to the present disclosure. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transferred from one computer-readable storage medium to another. Throughout this specification, references to terms such as "one embodiment," "some embodiments," "examples," "specific examples," or "some examples" indicate that the specific features, structures, materials, or characteristics described in conjunction with that embodiment or example are included in at least one embodiment or example of the present disclosure. Furthermore, the specific features, structures, materials, or characteristics described can be combined in any suitable manner in any one or more embodiments or examples. Furthermore, those skilled in the art may combine and assemble the different embodiments or examples, and features of different embodiments or examples, described in this specification, without conflict. Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, features specified as "first" or "second" may explicitly or implicitly include at least one of such features. In the description of this disclosure, "plurality" means two or more, unless otherwise specifically defined. Any process or method described in a flowchart or otherwise herein may be understood to represent a module, segment, or portion of code comprising one or more executable instructions for implementing specific logical functions or process steps. Furthermore, the scope of the preferred embodiments of this disclosure includes alternative implementations in which functions may be performed out of the order shown or discussed, including substantially simultaneously or in reverse order depending on the functions involved. The logic and / or steps described in the flowcharts or otherwise described herein can be considered, for example, as a sequenced list of executable instructions for implementing the logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, apparatus, or device (e.g., a computer-based system, a system including a processor, or other system that can fetch and execute instructions from an instruction execution system, apparatus, or device), or in conjunction with such an instruction execution system, apparatus, or device. It should be understood that various aspects of the present disclosure can be implemented using hardware, software, firmware, or a combination thereof.In the above embodiments, multiple steps or methods can be implemented using software or firmware stored in a memory and executed by a suitable instruction execution system. All or part of the steps of the above method embodiments can be performed by instructing the relevant hardware using a program. This program can be stored in a computer-readable storage medium. When executed, this program includes one or a combination of the steps of the method embodiments. Furthermore, the functional units in the various embodiments of the present disclosure can be integrated into a single processing module, each unit can exist physically separately, or two or more units can be integrated into a single module. These integrated modules can be implemented in hardware or as software functional modules. If these integrated modules are implemented as software functional modules and sold or used as standalone products, they can also be stored in a computer-readable storage medium. This storage medium can be a read-only memory, a magnetic disk, or an optical disk, for example. The above are merely exemplary embodiments of the present disclosure, but the scope of protection of the present disclosure is not limited thereto. Anyone skilled in the art can readily conceive of various variations or substitutions within the technical scope of the present disclosure, and such variations or substitutions are intended to be encompassed by the scope of protection of the present disclosure. Therefore, the scope of protection of this disclosure should be based on the scope of protection of the claims. Industrial Applicability: This disclosure provides a log query method. First, a log query request for a distributed database is obtained and the filter node and mapping node corresponding to the log query request are determined. Then, binary bytecode is generated based on the filter node and mapping node. Finally, based on the binary bytecode, the log data in the distributed database is queried to obtain a query result corresponding to the log query request. In this embodiment, binary bytecode is generated based on the filter node and mapping node, and the log data in the database is queried based on the binary bytecode. Code generation in a binary bytecode mode effectively avoids virtual function calls, eliminates the need for overall compilation and packaging, and improves the real-time performance of log analysis services.

Claims

Claims 1. A data processing method, comprising: Obtaining a data processing request for a database, and determining a filtering node and a mapping node corresponding to the data processing request; Generate binary bytecode based on the filtering node and the mapping node; The data in the database is processed according to the binary bytecode to obtain a processing result corresponding to the data processing request.

2. The method according to claim 1, wherein: The generating of binary bytecode based on the filter node and the mapping node includes: if the filter node and the mapping node are continuous, converting the filter node into a filter operator, converting the mapping node into a mapping operator, merging the filter operator and the mapping operator, and generating binary bytecode according to the merged operator.

3. The method according to claim 2, wherein: The processing of the data in the database according to the binary bytecode to obtain a processing result corresponding to the data processing request includes: if filtering processing exists before mapping processing according to the binary bytecode, filtering processing is performed on the data in the database, and recording an identifier of the data obtained by the filtering processing in a preset array; and performing mapping processing based on the identifier to obtain a processing result corresponding to the data processing request.

4. The method according to claim 1, wherein: The generating of binary bytecode based on the filter node and the mapping node includes: if the filter node and the mapping node are independent, converting the filter node into a filter operator, converting the mapping node into a mapping operator, and generating binary bytecodes corresponding to the filter operator and the mapping operator respectively.

5. The method according to claim 4, wherein: The processing of the data in the database according to the binary bytecode to obtain a processing result corresponding to the data processing request includes: performing filtering processing or mapping processing on the data in the database according to the binary bytecode; and determining the data after the filtering processing or the mapping processing as the processing result corresponding to the data processing request.

6. The method according to claim 1, wherein: The processing of the data in the database according to the binary bytecode to obtain a processing result corresponding to the data processing request includes: if a data column corresponding to the binary bytecode in the database is a dictionary column, determining a valid dictionary row based on the dictionary column; and determining the processing result corresponding to the data processing request according to the binary bytecode and the valid dictionary row.

7. The method according to claim 6, wherein: The determining, based on the binary bytecode and the valid dictionary row, a processing result corresponding to the data processing request includes: if the number of dictionary columns is one, processing the valid dictionary row using the binary bytecode to obtain a processing result corresponding to the data processing request.

8. The method according to claim 6, wherein: The determining, based on the binary bytecode and the valid dictionary rows, a processing result corresponding to the data processing request includes: if the number of dictionary columns is multiple and the product of the valid dictionary rows is less than a preset threshold, calculating a Cartesian product of the valid dictionary rows; and processing the Cartesian product using the binary bytecode to obtain a processing result corresponding to the data processing request.

9. The method according to claim 6, wherein: The determining, based on the binary bytecode and the valid dictionary row, a processing result corresponding to the data processing request includes: if the number of the dictionary columns is multiple and the product of the valid dictionary rows is greater than or equal to a preset threshold, determining, based on the maximum subexpression corresponding to the dictionary column, a dictionary value of the maximum subexpression corresponding to the valid dictionary row; and determining, based on the binary bytecode and the dictionary value of the maximum subexpression corresponding to the valid dictionary row, a processing result corresponding to the data processing request.

10. The method according to claim 1, wherein: Before processing the data in the database according to the binary bytecode to obtain the processing result corresponding to the data processing request, the method further includes: determining a local calculation stage and a global calculation stage based on the data processing request; sharding the data in the database, storing the sharded data in multiple first physical nodes, and determining the second physical node closest to each of the multiple first physical nodes; processing the data in the database according to the binary bytecode to obtain the processing result corresponding to the data processing request includes: executing the local calculation stage in the second physical node corresponding to any first physical node according to the binary bytecode for the data stored in the first physical node; executing the global calculation stage based on the calculation results of multiple local calculation stages, and using the calculation result of the global calculation stage as the processing result corresponding to the data processing request.

11. A log query method, the method comprising: Obtaining a log query request for a distributed database, and determining a filtering node and a mapping node corresponding to the log query request; Generate binary bytecode based on the filtering node and the mapping node; According to the binary bytecode, the log data in the distributed database is queried to obtain a query result corresponding to the log query request.

12. A data processing device, comprising: a determination module configured to obtain a data processing request for a database and determine a filtering node and a mapping node corresponding to the data processing request; a generation module configured to generate binary bytecode based on the filtering node and the mapping node; The processing module is configured to process the data in the database according to the binary bytecode to obtain a processing result corresponding to the data processing request.

13. An electronic device comprising a memory, a processor, and a computer program stored in the memory, wherein the processor implements the method according to any one of claims 1 to 10 when executing the computer program.

14. A computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor, implements the method according to any one of claims 1 to 11.

15. A computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the computer program implements the method according to any one of claims 1 to 11. 16

Images