Data processing method and device, computer equipment and storage medium

Abstract

The invention relates to the technical field of distributed databases, and discloses a data processing method and device, computer equipment and a storage medium, applied to a distributed database, the method comprises the steps of obtaining Pivot clauses of to-be-processed original data in the distributed database; performing content analysis on the Pivot clause, extracting metadata in the Pivot clause, packaging the extracted metadata into a Pivot semantic object, and storing the Pivot semantic object into a query plan; reading the Pivot semantic object in the query plan, converting the metadata in the Pivot semantic object into an equivalent CASE WHEN expression, replacing a Pivot operator in the query plan with the CASE WHEN expression, and generating an execution plan; and according to the execution plan, processing the original data, and generating processed target data. Therefore, the processing efficiency of the Pivot operation in the distributed database can be effectively improved.

Description

Technical Field

[0001] This invention relates to the field of distributed database technology, and more specifically to a data processing method, apparatus, computer equipment, and computer-readable storage medium. Background Technology

[0002] Currently, with the explosive growth of data volume and the increasing complexity of business operations, traditional centralized database systems are struggling to meet the demands of modern data processing. Distributed database systems have emerged to address this, offering greater scalability, fault tolerance, and processing efficiency by distributing data across multiple nodes. However, distributed databases also face numerous challenges in data processing, especially when handling complex queries and data transformation operations. Currently, in data analysis and report generation, the Pivot operation (row-to-column conversion) is a common data transformation technique. It allows users to convert data from row-based to column-based formats, enabling more intuitive data display and analysis. For example, time-series data can be expanded into independent columns by quarter or month, facilitating aggregation calculations and multi-dimensional analysis. However, existing technologies for implementing Pivot operations in distributed databases suffer from the following problems: 1. Manual Conversion: Users typically convert Pivot clauses manually into equivalent CASE WHEN expressions or other SQL statements. This method is not only tedious but also inefficient and error-prone, especially when dealing with complex Pivot operations.

[0003] 2. Query Optimizer: Existing query optimizers typically convert the Pivot clause into an intermediate form when processing it, but this conversion process lacks intelligent optimization and may result in inefficient execution plans.

[0004] Therefore, how to provide a data processing method, apparatus, computer equipment, and computer-readable storage medium that can effectively improve the processing efficiency of Pivot operations in distributed databases is a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0005] In view of the shortcomings of the prior art, the purpose of this invention is to provide a data processing method, apparatus, computer device and computer-readable storage medium, aiming to solve the problem of how to effectively improve the processing efficiency of Pivot operations in distributed databases.

[0006] To achieve the above objectives, the present invention adopts the following technical solution: In a first aspect, the present invention provides a data processing method applied to a distributed database, wherein the method includes: The Pivot clause retrieves the raw data to be processed from the distributed database; The Pivot clause is parsed to extract metadata, which is then encapsulated into a Pivot semantic object and stored in the query plan. Read the Pivot semantic object in the query plan, convert the metadata therein into an equivalent CASE WHEN expression, and replace the Pivot operator in the query plan with the CASE WHEN expression to generate an execution plan; According to the execution plan, the raw data is processed to generate processed target data.

[0007] Secondly, the present invention provides a data processing apparatus applied to a distributed database, wherein the apparatus comprises: The acquisition module is used to acquire the Pivot clause of the raw data to be processed in the distributed database; The parsing module is used to parse the content of the Pivot clause, extract the metadata therein, encapsulate the extracted metadata into a Pivot semantic object and store it in the query plan; The transformation module is used to read the Pivot semantic object in the query plan, convert the metadata therein into an equivalent CASE WHEN expression, and replace the Pivot operator in the query plan with the CASE WHEN expression to generate an execution plan; The processing module is used to process the raw data according to the execution plan and generate processed target data.

[0008] Thirdly, the present invention provides a computer device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the data processing method described above.

[0009] Fourthly, the present invention provides a computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor, implements the data processing method described above.

[0010] Compared to existing technologies, this invention provides a data processing method, apparatus, computer device, and computer-readable storage medium applied to a distributed database. The method includes: obtaining a Pivot clause from the raw data to be processed in the distributed database; parsing the Pivot clause to extract metadata, encapsulating the extracted metadata into a Pivot semantic object and storing it in a query plan; reading the Pivot semantic object from the query plan, converting the metadata therein into an equivalent CASE WHEN expression, and replacing the Pivot operator in the query plan with the CASE WHEN expression to generate an execution plan; and processing the raw data according to the execution plan to generate processed target data. Therefore, this invention can effectively improve the processing efficiency of Pivot operations in a distributed database. Attached Figure Description

[0011] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0012] Figure 1 This is a schematic diagram of the application environment of a data processing method provided in an embodiment of the present invention.

[0013] Figure 2 This is a flowchart illustrating a data processing method according to an embodiment of the present invention.

[0014] Figure 3 This is a schematic diagram of a program module of a data processing device provided in an embodiment of the present invention.

[0015] Figure 4 This is a schematic diagram of the structure of a computer device provided in an embodiment of the present invention.

[0016] Figure 5 This is another structural schematic diagram of a computer device provided in an embodiment of the present invention. Detailed Implementation

[0017] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0018] It should be understood that, when used in this specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or collections thereof.

[0019] It should also be understood that the term “and / or” as used in this specification and the appended claims refers to any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0020] As used in this specification and the appended claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection." Similarly, the phrase "if determined" or "if [the described condition or event] is detected" may be interpreted, depending on the context, as meaning "once determined," "in response to determination," "once [the described condition or event] is detected," or "in response to detection of [the described condition or event]."

[0021] Furthermore, in the description of this invention and the appended claims, the terms "first," "second," "third," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0022] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of the invention include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0023] It should be understood that the sequence number of each step in the following embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0024] To illustrate the technical solution of the present invention, specific embodiments are described below.

[0025] An embodiment of the present invention provides a data processing method that can be applied to, for example... Figure 1In the application environment shown, the client and server communicate via a network. The client includes, but is not limited to, handheld computers, desktop computers, laptops, ultra-mobile personal computers (UMPCs), netbooks, cloud computing devices, and personal digital assistants (PDAs). The server can be a standalone server or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms.

[0026] Please see Figure 2 An embodiment of the present invention provides a data processing method applied to a distributed database, wherein the method includes the following steps: S100: Obtain the Pivot clause for the raw data to be processed in the distributed database; S200. Parse the content of the Pivot clause, extract the metadata therein, encapsulate the extracted metadata into a Pivot semantic object and store it in the query plan; S300: Read the Pivot semantic object in the query plan, convert the metadata therein into an equivalent CASE WHEN expression, and replace the Pivot operator in the query plan with the CASE WHEN expression to generate an execution plan; S400. According to the execution plan, the original data is processed to generate processed target data.

[0027] In practical implementation, the data processing method of this embodiment effectively improves the processing efficiency of Pivot operations in a distributed database through a series of carefully designed steps. A detailed analysis follows: 1. Intelligent parsing and packaging (S100 and S200) By acquiring and parsing the Pivot clause, extracting its key metadata, and encapsulating it into Pivot semantic objects, this method can quickly understand and organize the core information of Pivot operations. This intelligent parsing and encapsulation process improves the accuracy of data processing. By encapsulating metadata into structured semantic objects, query plans can manage and use this information more efficiently, thereby reducing the complexity of query optimization and execution.

[0028] 2. Dynamic Conversion and Optimization (S300) The metadata in the Pivot semantic object is converted into an equivalent CASE WHEN expression, which replaces the Pivot operator in the query plan, generating an execution plan. This process leverages the flexibility of CASE WHEN expressions and the advantages of native database support, enabling more efficient execution of Pivot operations. Simultaneously, dynamic transformation and optimization mechanisms generate the optimal execution plan based on specific query requirements, further improving query performance. In this way, the method not only solves the compatibility issues of the Pivot clause in distributed databases but also reduces computational resource consumption by optimizing the execution plan.

[0029] 3. Efficient execution and target data generation (S400) Based on the generated execution plan, the raw data is processed to produce the processed target data. This step ensures the efficiency and consistency of the entire data processing flow. By efficiently executing the optimized query plan in a distributed environment, this method can significantly reduce data processing time and improve system response speed. Furthermore, the generated target data can be directly used for data analysis and report generation, further enhancing the overall efficiency of data processing.

[0030] Through the synergistic effect of the above steps, this method not only improves the parsing and transformation efficiency of Pivot operations, but also significantly enhances the processing efficiency of Pivot operations in distributed databases through dynamic optimization and efficient execution mechanisms. This method effectively addresses the complexity and performance bottlenecks of Pivot operations in distributed databases, providing users with faster and more accurate data processing capabilities, thus playing a crucial role in large-scale data analysis and report generation scenarios.

[0031] Furthermore, in one embodiment, the data processing method, wherein the Pivot clause for obtaining the raw data to be processed in the distributed database specifically includes: The client receives SQL query requests for the distributed database submitted by the user. Parse the content of the SQL query request and extract the Pivot clause concerning the raw data to be processed in the distributed database.

[0032] In practice, the specific implementation process of this embodiment is roughly as follows: 1. Receive SQL query requests submitted by users. Client Interface: Provides a user-friendly client interface (such as a web interface, command-line tool, or graphical interface) that allows users to input or upload SQL query requests.

[0033] Request reception: The client sends the user's SQL query request to the server, and the server receives the request through the network interface.

[0034] Request caching: Caches received SQL query requests in memory or temporary storage for later processing.

[0035] 2. Preliminary verification of SQL query requests Syntax check: Performs a preliminary syntax check on the SQL query request to ensure that the SQL statement is structurally correct.

[0036] Error message: If a syntax error is found, return detailed error information to the user and prompt the user to make corrections.

[0037] Security check: Check whether SQL query requests contain potential security risks, such as SQL injection attacks.

[0038] 3. Parse the SQL query request content Parse the SQL query request and extract the Pivot clause containing the raw data to be processed in the distributed database.

[0039] Through the above process, user-submitted SQL query requests can be received, verified, and parsed efficiently, and the Pivot clause concerning the raw data to be processed in the distributed database can be extracted.

[0040] Furthermore, in one embodiment, the data processing method, wherein parsing the content of the Pivot clause and extracting its metadata specifically includes: Using an SQL syntax parser, the Pivot clause is analyzed to construct an abstract syntax tree; The abstract syntax tree is recursively traversed to extract metadata from the Pivot clause.

[0041] Furthermore, in the data processing method, the metadata includes aggregation functions, aggregation target columns, pivot reference columns, and a set of column value-alias mappings.

[0042] Furthermore, the data processing method, wherein encapsulating the extracted metadata into a Pivot semantic object and storing it in the query plan, specifically includes: Identify the type and structure of the extracted metadata, and encapsulate the metadata into a Pivot semantic object based on the identification result; The Pivot semantic object is stored in the context information of the query plan.

[0043] In practice, the specific implementation process of this embodiment is roughly as follows: 1. Perform syntax analysis using an SQL syntax parser. Choose a parser: Select an advanced SQL syntax parser that supports multiple SQL dialects, such as ANTLR, JSQLParser, etc.

[0044] Parsing the Pivot clause: The Pivot clause is input into the parser, which performs syntactic analysis on the Pivot clause according to the SQL syntax rules.

[0045] Constructing the AST: The parser outputs an abstract syntax tree (AST) of the Pivot clause. The AST is a tree structure that represents the grammatical structure of the Pivot clause.

[0046] 2. Recursively traverse the abstract syntax tree to extract metadata. Initialization traversal: Starting from the root node of the AST, recursively traverse each node.

[0047] Extracting aggregate functions: Identify and extract aggregate functions (such as SUM, AVG, etc.) from the Pivot clause.

[0048] Extract the target column of the aggregation: Identify and extract the target column (the field affected by the aggregation function) of the aggregation function.

[0049] Extract pivot baseline column: Identify and extract the pivot baseline column (the field specified in the FOR clause, which serves as the dimension baseline for row-to-column transformation).

[0050] Extract column value-alias mapping set: Identify and extract the column value-alias mapping set (the mapping relationship between the original values ​​defined in the IN clause and the target column names).

[0051] 3. Identify the type and structure of metadata Type identification: The extracted metadata is type-identified, for example, the aggregate function is identified as SUM, the aggregate target column is amount, and the pivot base column is quarter.

[0052] Structural analysis: Analyze the structural relationships between metadata, such as the association between aggregate functions and aggregate target columns, and the association between pivot base columns and column value-alias mapping sets.

[0053] 4. Encapsulate metadata into Pivot semantic objects Create a semantic object: Create a Pivot semantic object, which is a structured data structure used to store the metadata of the Pivot clause.

[0054] Populate metadata: Populate the extracted metadata into the Pivot semantic object.

[0055] 5. Store the Pivot semantic object in the context information of the query plan. Store in context: Store the Pivot semantic object in the context information of the query plan.

[0056] Associate contextual information: Ensure that the Pivot semantic object is consistent with other parts of the query plan (such as the FROM clause, WHERE clause, etc.) so that it can be used correctly in subsequent query optimization and execution.

[0057] Through the above process, the Pivot clause can be efficiently parsed, metadata extracted, and encapsulated into a Pivot semantic object, which is then stored in the context information of the query plan. This process not only ensures the correct parsing and encapsulation of the Pivot clause but also provides a solid foundation for subsequent query optimization and execution.

[0058] Further, in one embodiment, the data processing method, wherein reading the Pivot semantic object in the query plan, converting the metadata therein into an equivalent CASE WHEN expression, and replacing the Pivot operator in the query plan with the CASE WHEN expression to generate an execution plan, specifically includes: Read the Pivot semantic object from the context information of the query plan; According to a preset conversion strategy, the metadata in the Pivot semantic object is converted into an equivalent CASEWHEN expression; The CASE WHEN expression is used to replace the Pivot operator in the query plan to generate an initial execution plan. The initial execution plan is then optimized to obtain the final execution plan.

[0059] In practice, the specific implementation process of this embodiment is roughly as follows: 1. Read the Pivot semantic object from the context information of the query plan. Access Context Information: Access the portion that stores Pivot semantic objects by querying the context information interface of the plan.

[0060] Read semantic objects: Extract Pivot semantic objects from the context, ensuring their integrity and consistency.

[0061] Verify version information: Check the version information of the Pivot semantic object to ensure that the latest data version is being used.

[0062] Error detection: During the reading process, possible errors or inconsistencies are automatically detected and corrected.

[0063] 2. Based on the preset conversion strategy, convert the metadata in the Pivot semantic object into an equivalent CASEWHEN expression. Load Transformation Strategy: Loads a preset transformation strategy that defines how to convert metadata in a Pivot semantic object into a CASE WHEN expression.

[0064] Parse metadata: Extract metadata from the Pivot semantic object, including aggregate functions, aggregate target columns, pivot baseline columns, and column value-alias mapping sets.

[0065] Generate CASE WHEN expression: Generate an equivalent CASE WHEN expression based on the extracted metadata.

[0066] 3. Replace the Pivot operator in the query plan with the CASE WHEN expression to generate the initial execution plan. Locate the Pivot operator: Locate the position of the Pivot operator in the query plan.

[0067] Replace operator: Replace the Pivot operator with the generated CASE WHEN expression.

[0068] Generate initial execution plan: Generate an initial execution plan based on the replaced query plan.

[0069] Verify the plan logic: Ensure that the logic of the replaced initial execution plan is correct and free of syntax or semantic errors.

[0070] 4. Optimize the initial execution plan. Query optimization: Use the query optimizer to optimize the initial execution plan, such as adjusting the query partitioning strategy and optimizing the distribution and computation of data.

[0071] Resource allocation optimization: Dynamically allocate computing resources based on data volume and computational complexity to ensure efficient query execution.

[0072] Fault tolerance mechanism: Implement a fault tolerance mechanism in the execution plan to ensure that the system can automatically recover and continue execution when some nodes fail.

[0073] Performance monitoring: Implement performance monitoring during the optimization process to ensure that the optimized execution plan has the best performance in the distributed environment.

[0074] 5. Generate the final execution plan. Record optimization results: Record the results of the optimization process, including the adjusted partitioning strategy, resource allocation, etc.

[0075] Generate final plan: Based on the optimization results, generate the final execution plan.

[0076] Through the above process, Pivot semantic objects can be read efficiently, their metadata converted into CASEWHEN expressions, and optimized execution plans generated. This process not only ensures the correctness and efficiency of Pivot operations but also improves the performance and reliability of distributed databases in large-scale data processing through dynamic optimization and fault tolerance mechanisms.

[0077] Furthermore, in one embodiment, the data processing method, wherein processing the original data according to the execution plan to generate processed target data specifically includes: Based on the execution plan, the original data is aggregated and transformed to generate intermediate results; Based on the client's configuration information, the intermediate results are formatted to generate target data, which is then returned to the client for display.

[0078] In practice, the specific implementation process of this embodiment is roughly as follows: 1. Perform aggregation and transformation operations Load execution plan: Load the execution plan from the context information of the query plan to ensure its integrity and consistency.

[0079] Distributed execution: In a distributed database environment, data processing tasks are launched according to the execution plan. Distributed computing frameworks (such as Spark, Flink, etc.) are used to process data in parallel, performing aggregation and transformation operations on the raw data to improve processing efficiency.

[0080] Generate intermediate results: Store the aggregated and transformed data as intermediate results for subsequent processing. Intermediate results can be stored in memory or temporary storage to ensure efficient access.

[0081] 2. Read intermediate results Locate intermediate results: Based on the storage location information in the execution plan, locate the storage location of intermediate results.

[0082] Data loading: Load intermediate results from memory or temporary storage to ensure data integrity and consistency.

[0083] Data validation: Validate the loaded intermediate results to ensure that no errors or loss occurred during the aggregation and transformation process.

[0084] 3. Perform formatting based on the client's configuration information. Get client configuration: Obtain configuration information from the client, including data format (such as CSV, JSON, XML), column selection, sorting rules, etc.

[0085] Data sorting: Sort intermediate results according to the sorting rules configured on the client side.

[0086] Column selection: Select the columns to be displayed based on the client configuration, remove unnecessary columns, and reduce data transmission volume.

[0087] Data format adjustment: Convert intermediate results into a data format specified by the client. For example, convert data to JSON format for client processing.

[0088] Generate target data: Assemble the formatted data into target data to ensure data integrity and consistency.

[0089] 4. Return target data Data transmission: The target data is transmitted back to the client via a network interface. Efficient data transmission protocols (such as HTTP / 2 and gRPC) are used to ensure efficient and reliable data transmission.

[0090] Data compression: Compressing data during transmission reduces transmission time and bandwidth usage.

[0091] Secure transmission: The transmitted data is encrypted to ensure its security and confidentiality.

[0092] Client reception: The client receives the target data and performs decompression and decryption to ensure data integrity and consistency.

[0093] Data presentation: The client displays the target data to the user, for example, in a web interface or reporting tool.

[0094] Through the above process, raw data can be efficiently aggregated and transformed according to the execution plan to generate intermediate results. These intermediate results are then formatted according to the client's configuration information, and finally, the target data is returned to the client for display. This process ensures the efficiency and accuracy of data processing and improves the user experience.

[0095] As can be seen from the above method embodiments, the data processing method provided by the present invention is applied to a distributed database. The method includes: obtaining a Pivot clause from the original data to be processed in the distributed database; parsing the content of the Pivot clause, extracting its metadata, encapsulating the extracted metadata into a Pivot semantic object and storing it in a query plan; reading the Pivot semantic object from the query plan, converting the metadata therein into an equivalent CASE WHEN expression, and replacing the Pivot operator in the query plan with the CASE WHEN expression to generate an execution plan; and processing the original data according to the execution plan to generate processed target data. Thus, the method of the present invention can effectively improve the processing efficiency of Pivot operations in a distributed database.

[0096] It should be understood that although this application provides the method operation steps as described in the embodiments or flowcharts, conventional or non-inventive labor may include more or fewer operation steps, and these operation steps are not necessarily executed sequentially according to the order of the embodiments or flowcharts. The order of steps listed in the embodiments or flowcharts is merely one way of executing many steps and does not represent the only execution order. It should be noted that there is no necessary sequential order between the above steps. Those skilled in the art can understand from the description of the embodiments of the present invention that the above steps may have different execution orders in different embodiments, that is, they may be executed in parallel or in exchange, etc. Moreover, at least some steps in the embodiments or flowcharts may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but may be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but may be executed in turn, alternately, or synchronously with other steps or at least a part of the sub-steps or stages of other steps.

[0097] Based on the above method embodiments, please refer to Figure 3 Another embodiment of the present invention also provides a data processing apparatus applied to a distributed database, wherein the apparatus includes: Module 11 is used to obtain the Pivot clause of the raw data to be processed in the distributed database; Parsing module 12 is used to parse the content of the Pivot clause, extract the metadata therein, encapsulate the extracted metadata into a Pivot semantic object and store it in the query plan; The conversion module 13 is used to read the Pivot semantic object in the query plan, convert the metadata therein into an equivalent CASE WHEN expression, and replace the Pivot operator in the query plan with the CASE WHEN expression to generate an execution plan; The processing module 14 is used to process the raw data according to the execution plan to generate processed target data.

[0098] Further, in one embodiment, the data processing apparatus, wherein the Pivot clause for obtaining the raw data to be processed in the distributed database specifically includes: The client receives SQL query requests for the distributed database submitted by the user. Parse the content of the SQL query request and extract the Pivot clause concerning the raw data to be processed in the distributed database.

[0099] Furthermore, in one embodiment, the data processing apparatus, wherein the content parsing of the Pivot clause and extraction of its metadata specifically includes: Using an SQL syntax parser, the Pivot clause is analyzed to construct an abstract syntax tree; The abstract syntax tree is recursively traversed to extract metadata from the Pivot clause.

[0100] Furthermore, in the data processing apparatus, the metadata includes aggregation functions, aggregation target columns, pivot reference columns, and a set of column value-alias mappings.

[0101] Furthermore, in the data processing apparatus, the step of encapsulating the extracted metadata into a Pivot semantic object and storing it in the query plan specifically includes: Identify the type and structure of the extracted metadata, and encapsulate the metadata into a Pivot semantic object based on the identification result; The Pivot semantic object is stored in the context information of the query plan.

[0102] Further, in one embodiment, the data processing apparatus, wherein reading the Pivot semantic object in the query plan, converting the metadata therein into an equivalent CASE WHEN expression, and replacing the Pivot operator in the query plan with the CASE WHEN expression to generate an execution plan, specifically includes: Read the Pivot semantic object from the context information of the query plan; According to a preset conversion strategy, the metadata in the Pivot semantic object is converted into an equivalent CASEWHEN expression; The CASE WHEN expression is used to replace the Pivot operator in the query plan to generate an initial execution plan. The initial execution plan is then optimized to obtain the final execution plan.

[0103] Furthermore, in one embodiment, the data processing apparatus, wherein processing the raw data according to the execution plan to generate processed target data specifically includes: Based on the execution plan, the original data is aggregated and transformed to generate intermediate results; Based on the client's configuration information, the intermediate results are formatted to generate target data, which is then returned to the client for display.

[0104] It should be noted that, in the device embodiments of the present invention, the information interaction and execution process between the above modules are based on the same concept as in the method embodiments of the present invention. For details on their specific functions and the resulting technical effects, please refer to the aforementioned method embodiments section, which will not be repeated here.

[0105] Based on the above method embodiments, another embodiment of the present invention also provides a computer device, which can be a server, and its internal structure diagram can be as follows. Figure 4 As shown. The computer device includes a processor, memory, network interface, and database connected via a system bus. The processor provides computing and control capabilities. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores an operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The network interface is used to communicate with external terminals via a network connection. When the computer program is executed by the processor, it implements the functions or steps of the data processing method on the server side as described in any of the above method embodiments.

[0106] Based on the above method embodiments, another embodiment of the present invention also provides a computer device, which can be a client, and its internal structure diagram can be as follows. Figure 5As shown, the computer device includes a processor, memory, network interface, display screen, and input device connected via a system bus. The processor provides computing and control capabilities. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores an operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The network interface is used to communicate with external terminals via a network connection. When executed by the processor, the computer program implements the functions or steps of the data processing method on the client side as described in any of the above method embodiments.

[0107] Those skilled in the art will understand that Figure 4 and Figure 5 The structural schematic diagram shown is only a schematic diagram of a part of the structure related to the present invention and does not constitute a limitation on the computer device on which the present invention is applied. The specific computer device may include more components than shown in the figure, or combine certain components, or have different component arrangements.

[0108] The processor referred to herein can be a CPU, but it can also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor.

[0109] The memory includes readable storage media, internal memory, etc., where internal memory can be the RAM of a computer device. Internal memory provides an environment for the operation of the operating system and computer-readable instructions stored in the readable storage media. The readable storage media can be the hard drive of the computer device, or in other embodiments, it can be an external storage device of the computer device, such as a plug-in hard drive, Smart Media Card (SMC), Secure Digital (SD) card, or Flash Card. Furthermore, the memory can include both internal storage units and external storage devices of the computer device. The memory is used to store the operating system, applications, bootloader, data, and other programs, such as program code for computer programs. The memory can also be used to temporarily store data that has been output or will be output.

[0110] Based on the above method embodiments, another embodiment of the present invention provides a computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor, implements the data processing method as described in any of the above method embodiments. The computer-readable storage medium may be non-volatile or volatile.

[0111] It should be noted that the functions or steps that can be achieved by the computer-readable storage medium or computer device, and the technical effects brought about by the functions / steps, can be referred to the relevant descriptions in the foregoing method embodiments. To avoid repetition, they will not be described one by one here.

[0112] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc. The disclosed memory components or memories of the operating environment described herein are intended to include one or more of these and / or any other suitable types of memory.

[0113] Those skilled in the art will understand that, for the sake of convenience and brevity, the embodiments of the device of the present invention are only illustrated by the division of the above-mentioned functional units and modules. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of the present invention. The specific working process of the units and modules in the above device can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here. If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium.

[0114] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0115] In the embodiments provided by this invention, it should be understood that the disclosed apparatus / computer devices and methods can be implemented in other ways. For example, the apparatus / computer device embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0116] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0117] It should be noted that if any software tools or components not belonging to this company appear in the embodiments of this application, they are merely illustrative examples and do not represent actual use. The above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.

Claims

1. A data processing method applied to a distributed database, characterized in that, The method includes: The Pivot clause retrieves the raw data to be processed from the distributed database; The Pivot clause is parsed to extract metadata, which is then encapsulated into a Pivot semantic object and stored in the query plan. Read the Pivot semantic object in the query plan, convert the metadata therein into an equivalent CASEWHEN expression, and replace the Pivot operator in the query plan with the CASEWHEN expression to generate an execution plan; According to the execution plan, the raw data is processed to generate processed target data.

2. The data processing method according to claim 1, characterized in that, The Pivot clause for retrieving the raw data to be processed from the distributed database includes: The client receives SQL query requests for the distributed database submitted by the user. Parse the content of the SQL query request and extract the Pivot clause concerning the raw data to be processed in the distributed database.

3. The data processing method according to claim 1, characterized in that, The process of parsing the Pivot clause and extracting its metadata includes: Using an SQL syntax parser, the Pivot clause is analyzed to construct an abstract syntax tree; The abstract syntax tree is recursively traversed to extract metadata from the Pivot clause.

4. The data processing method according to claim 1, characterized in that, The step of encapsulating the extracted metadata into a Pivot semantic object and storing it in the query plan includes: Identify the type and structure of the extracted metadata, and encapsulate the metadata into a Pivot semantic object based on the identification result; The Pivot semantic object is stored in the context information of the query plan.

5. The data processing method according to claim 4, characterized in that, The process of reading the Pivot semantic object from the query plan, converting its metadata into an equivalent CASE WHEN expression, and replacing the Pivot operator in the query plan with the CASE WHEN expression to generate an execution plan includes: Read the Pivot semantic object from the context information of the query plan; According to a preset conversion strategy, the metadata in the Pivot semantic object is converted into an equivalent CASE WHEN expression; The CASE WHEN expression is used to replace the Pivot operator in the query plan to generate an initial execution plan. The initial execution plan is then optimized to obtain the final execution plan.

6. The data processing method according to claim 1, characterized in that, The step of processing the raw data according to the execution plan to generate processed target data includes: Based on the execution plan, the original data is aggregated and transformed to generate intermediate results; Based on the client's configuration information, the intermediate results are formatted to generate target data, which is then returned to the client for display.

7. The data processing method according to any one of claims 1-6, characterized in that, The metadata includes aggregate functions, aggregate target columns, pivot base columns, and a set of column value-alias mappings.

8. A data processing apparatus, applied to a distributed database, characterized in that, The device includes: The acquisition module is used to acquire the Pivot clause of the raw data to be processed in the distributed database; The parsing module is used to parse the content of the Pivot clause, extract the metadata therein, encapsulate the extracted metadata into a Pivot semantic object and store it in the query plan; The transformation module is used to read the Pivot semantic object in the query plan, convert the metadata therein into an equivalent CASE WHEN expression, and replace the Pivot operator in the query plan with the CASE WHEN expression to generate an execution plan; The processing module is used to process the raw data according to the execution plan and generate processed target data.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the data processing method as described in any one of claims 1-7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the data processing method as described in any one of claims 1-7.

Images