Data processing method and device based on dynamic code generation

Abstract

The application provides a data processing method and device based on dynamic code generation, and relates to the technical field of data processing. The data processing method based on dynamic code generation comprises the following steps: analyzing a cross-source data processing request to determine type information and data processing syntax of the data; generating a data processing physical plan according to the type information and the data processing syntax; replacing built-in operators corresponding to the data processing physical plan with predefined operation operators; and generating a corresponding first format code according to the predefined operation operators, so as to execute the data processing request based on the first format code and determine a data processing result. In the application, the federated data computing logic is dynamically generated into efficient C++ code in a large-scale data and high-concurrency scenario, and is executed locally to bypass the performance bottleneck of JVM, while fully utilizing the advantages of Spark in distributed computing, task scheduling and resource management, thereby greatly improving the data processing efficiency.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and for example to a data processing method and apparatus based on dynamic code generation. Background Technology

[0002] With the rapid development of big data technology, distributed data analysis has become a core means of processing massive amounts of data. However, in data-driven applications, privacy protection and data security have become critical issues that cannot be ignored. Especially in fields involving sensitive data, such as finance, healthcare, and government affairs, how to achieve efficient data sharing and analysis in a distributed environment while ensuring data privacy and security has become a major challenge facing current technology. Although traditional distributed computing frameworks perform well in processing large-scale data, their design has not fully considered the needs of privacy-preserving computation. In the process of cross-node data sharing and collaborative computation, data usually needs to be transmitted between nodes, which may lead to the leakage of sensitive information. Although existing encryption technologies and multi-party secure computation can alleviate the problem of sensitive information leakage to some extent, these methods are often accompanied by a significant decrease in computational efficiency, making it difficult to meet the performance requirements of large-scale data processing while ensuring privacy.

[0003] Among related technologies, existing distributed computing frameworks such as Spark are based on the Scala (Scalable Language, a multi-paradigm programming language) and run on the JVM (Java Virtual Machine). While they offer good cross-platform compatibility and ease of development, they have inherent limitations in execution efficiency. The JVM's garbage collection mechanism and runtime interpreted execution method limit its performance when handling complex computational tasks, especially in scenarios involving large-scale data and high concurrency, where data processing efficiency is low. Summary of the Invention

[0004] This application aims to provide a data processing method and apparatus based on dynamic code generation.

[0005] According to one aspect of this application, a data processing method based on dynamic code generation is proposed, comprising: parsing a cross-source data processing request to determine the data type information and data processing syntax; generating a data processing physical plan based on the type information and data processing syntax; replacing the built-in operator corresponding to the data processing physical plan with a predefined operation operator; generating a corresponding first format code based on the predefined operation operator, executing the data processing request based on the first format code, and determining the data processing result.

[0006] According to one aspect of this application, a data processing apparatus based on dynamic code generation is proposed, comprising:

[0007] The request parsing module is used to parse cross-origin data processing requests to determine the data type information and data processing syntax;

[0008] The plan generation module is used to generate a data processing physical plan based on type information and data processing syntax.

[0009] The operator replacement module is used to replace the built-in operators corresponding to the data processing physical plan with predefined operation operators;

[0010] The task execution module is used to generate corresponding first format code based on predefined operation operators, execute data processing requests based on the first format code, and determine the data processing results.

[0011] According to one aspect of this application, an electronic device is provided, comprising: a processor; and a memory storing a computer program that, when executed by the processor, causes the processor to perform the method described above.

[0012] According to one aspect of this application, a non-transitory computer-readable medium is proposed, on which readable instructions are stored, which, when executed by a processor, cause the processor to perform the method described above.

[0013] It should be understood that the above general description and the following detailed description are merely exemplary and do not limit this application.

[0014] Beneficial effects:

[0015] The embodiments provided in this application, by dynamically generating efficient C++ code from federated data computation logic and executing it locally, bypasses the performance bottleneck of the JVM. Simultaneously, they fully leverage Spark's advantages in distributed computing, task scheduling, and resource management, significantly improving data processing efficiency. The use of dynamic code generation technology, combined with Spark's distributed architecture, enables privacy-preserving computation tasks to execute efficiently in parallel, significantly reducing computational latency and supporting high-performance federated computing needs in large-scale data scenarios. Through targeted modifications to operators, making them fully adaptable to the Spark environment, both Spark's ease of use and scalability are retained, while higher-performance privacy-preserving computation is achieved through underlying C++ optimizations, providing an efficient and secure solution for complex data collaboration scenarios. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings, without exceeding the scope of protection claimed by this application.

[0017] Figure 1 This is a schematic diagram of the module architecture of a data processing system based on dynamic code generation provided in an embodiment of this application;

[0018] Figure 2 A flowchart of a data processing method based on dynamic code generation provided in an embodiment of this application;

[0019] Figure 3 A flowchart illustrating the federal data connection provided in this application embodiment;

[0020] Figure 4 A block diagram of a data processing apparatus based on dynamic code generation provided in an embodiment of this application;

[0021] Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0022] Exemplary embodiments will now be described more fully with reference to the accompanying drawings. However, these exemplary embodiments can be implemented in many forms and should not be construed as limited to the embodiments set forth herein; rather, they are provided so that this application will be thorough and complete, and will fully convey the concept of the exemplary embodiments to those skilled in the art. The same reference numerals in the drawings denote the same or similar parts, and therefore repeated descriptions of them will be omitted.

[0023] Furthermore, the described features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. Numerous specific details are provided in the following description to give a thorough understanding of embodiments of this application. However, those skilled in the art will recognize that the technical solutions of this application can be practiced without one or more of the specific details, or other methods, components, apparatuses, steps, etc., can be employed. In other instances, well-known methods, apparatuses, implementations, or operations are not shown or described in detail to avoid obscuring various aspects of this application.

[0024] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities can be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0025] The flowcharts shown in the accompanying drawings are merely illustrative and do not necessarily include all content and operations / steps, nor do they necessarily have to be performed in the described order. For example, some operations / steps can be broken down, while others can be combined or partially combined; therefore, the actual execution order may change depending on the specific circumstances.

[0026] It should be understood that although the terms first, second, third, etc., may be used herein to describe various components, these components should not be limited by these terms. These terms are used to distinguish one component from another. Therefore, the first component discussed below may be referred to as the second component without departing from the teachings of this application. As used herein, the term "and / or" includes all combinations of any one and more of the associated listed items.

[0027] Figure 1 This is a schematic diagram of the module architecture of a data processing system based on dynamic code generation, provided in an embodiment of this application. Figure 1 This is a complete system architecture diagram corresponding to federated data connectivity, such as... Figure 1 As shown, the system can include a data acquisition and parsing module (jydataSQL), a plan generation and optimization module (SparkSQL), a code generation module (jydata code generator), and an operator invocation module (jydata engine). The jydataSQL module provides an entry point for users or other programs to access and manipulate data through a programming interface, supporting cross-source data retrieval and schema parsing. It can acquire data from multiple different data sources (such as Hive, MySQL, etc.), breaking the limitations of data storage location and achieving centralized data aggregation, providing a data foundation for subsequent processing. Simultaneously, the jydataSQL module parses the data's schema information (such as column names, data types, etc.) to ensure the system can correctly process and use the data.

[0028] The SparkSQL module can generate and optimize query plans. Based on the input query request and schema information, SparkSQL can generate a preliminary logical plan and improve query execution efficiency by optimizing the operator order and selecting better algorithms. For the optimized logical plan, considering the distributed nature of the cluster, a physical plan is generated. In the physical plan, when federated data join operations are involved, SparkSQL can replace built-in operators with custom federated data join operators to support privacy protection and efficient computation.

[0029] The jydata code generator module can deeply analyze the query plan generated by SparkSQL (i.e., the physical plan after replacing operators) and dynamically generate efficient first-format code. In this application, C++ code can be used as the first-format code. The generated code is optimized for specific data sources, fields, and query conditions, avoiding redundant calculations and performance bottlenecks in general-purpose code.

[0030] The jydata engine module leverages a high-performance C++ operator library and combines it with Spark's advantages in distributed computing, task scheduling, and resource management to achieve efficient data processing and computation. By dynamically loading and executing optimized first-format code, the jydata engine significantly improves computational performance while avoiding the performance bottlenecks of traditional distributed runtime environments. Utilizing Spark's elastic scheduling and efficient parallel computing capabilities, this module ensures efficient execution of computational tasks and optimal resource utilization, providing strong support for large-scale distributed data processing.

[0031] For specific implementation details, please refer to the following examples.

[0032] Figure 2 This is a flowchart illustrating a data processing method based on dynamic code generation provided in an embodiment of this application. The method in this embodiment can be applied to a data processing server. Figure 2 As shown, the method includes steps S20, S21, S22 and S23.

[0033] In step S20, the cross-source data processing request is parsed to determine the data type information and data processing syntax.

[0034] In this application, data processing requests may involve multiple devices, and the data to be processed may involve multiple data sources. Such cross-source data processing requests can be input by the user or by other programs accessing the data, and may include data type information and data processing syntax. The format of the data processing request can be an SQL statement, such as `SELECT * FROM HiveTable JOIN MySQLTable ON HiveTable.id = MySQLTable.id`. First, the schema information of the table (such as field names, data types, etc.) is retrieved from Hive and MySQL, and this schema information is parsed and managed to determine the type information and data processing syntax. The data processing syntax is the specific syntax used in subsequent data processing operations such as queries.

[0035] In some implementations, data processing syntax and type information can be extracted directly from the data processing request.

[0036] In step S21, a data processing physical plan is generated based on the type information and data processing syntax.

[0037] In this application, the plan generation module can be the SparkSQL module, which has the ability to generate a data processing physical plan based on the parsed type information and data processing syntax.

[0038] In step S22, the built-in operator corresponding to the data processing physical plan is replaced with a predefined operation operator.

[0039] This application can pre-construct predefined operators compatible with Spark's native operations by introducing technologies such as Extended Oblivious Transfer (OT), Lightweight Oblivious Pseudorandom Function (OPRF), and parallelized federated alignment protocols. Different built-in operators have corresponding predefined operators. The required built-in operators can be determined through the data processing physical plan, and these built-in operators can be replaced with predefined operators.

[0040] In step S23, a corresponding first format code is generated according to a predefined operation operator, and a data processing request is executed based on the first format code to determine the data processing result.

[0041] This application allows for the pre-configuration of a jydata code generator. This module can generate first-format code corresponding to predefined operation operators; in this application, the first-format code is C++ code. Guided by the C++ code, the corresponding operations for the data processing request can be executed to obtain the data processing results.

[0042] This application bypasses the JVM performance bottleneck by dynamically generating efficient C++ code for federated data computation logic and executing it locally. It also fully leverages Spark's advantages in distributed computing, task scheduling, and resource management, significantly improving data processing efficiency. The use of dynamic code generation technology, combined with Spark's distributed architecture, enables privacy-preserving computation tasks to execute in parallel and efficiently, significantly reducing computation latency and supporting the high-performance federated computing needs of large-scale data scenarios. Through targeted modifications to operators, it fully adapts to the Spark environment, retaining Spark's ease of use and scalability while achieving higher-performance privacy-preserving computation through underlying C++ optimizations, providing an efficient and secure solution for complex data collaboration scenarios.

[0043] According to some embodiments, in the process of parsing a data processing request, specifically, the data processing request can be parsed to determine the data source, data location, and data processing syntax of the corresponding data to be processed; and the data location of the data source can be located to determine the type information.

[0044] This application can directly extract the data source, data location, and data processing syntax of the data to be processed from the data processing request. Then, the preset type information corresponding to the data source can be found at the corresponding data location.

[0045] This application automatically identifies the data source, data location, and processing syntax of the data to be processed by parsing data processing requests, ensuring the accuracy of data acquisition, avoiding manual configuration errors, and improving the reliability of the data processing workflow. It supports unified parsing and location of different data sources, enabling efficient integration of heterogeneous data and reducing the complexity of multi-source data collaborative processing.

[0046] According to some embodiments, a data processing physical plan can be generated by generating an optimized logical plan. Specifically, an optimized logical plan is generated based on type information, data processing syntax, and a pre-defined Spark distributed computing framework; a data processing physical plan is then generated based on the cluster's distributed execution characteristics and the optimized logical plan.

[0047] This application first leverages the capabilities of the Spark distributed computing framework itself to generate a logical plan based on type information and data processing syntax, disregarding certain characteristics. This plan is then optimized using preset optimization methods to obtain an optimized logical plan. Next, considering the characteristics of clustered distributed execution, an execution strategy is selected to adjust the optimized logical plan into a physical data processing plan.

[0048] This application optimizes the logical plan using the Spark distributed computing framework, reducing redundant data scanning and computation, and lowering I / O and CPU overhead. It also dynamically adjusts the physical plan based on cluster resources to avoid data skew and computational hotspots, thereby improving task parallelism.

[0049] According to some embodiments, a data processing request includes a data processing objective. During the generation of an optimized logical plan, the Spark distributed computing framework can be controlled to generate a preliminary logical plan corresponding to the data processing request based on type information and data processing syntax; the Spark distributed computing framework can also be controlled to determine the target optimization algorithm and operator optimization order based on the data processing objective, so as to generate an optimized logical plan based on the target optimization algorithm, operator optimization order, and the preliminary logical plan.

[0050] In this application, the data processing objective can be used to characterize the desired effect during data processing, such as reducing computational or transmission volume.

[0051] In some implementations, the Spark distributed computing framework can first be controlled to generate a preliminary logical plan corresponding to the data processing request based on the type information and data processing syntax. The Spark distributed computing framework can then select the preset target optimization algorithm and operator optimization order corresponding to the data processing target. The preliminary logical plan can then be optimized according to the target optimization algorithm, and the order of the operators can be adjusted to obtain the optimized logical plan.

[0052] This application automatically parses type information and data processing syntax to generate a preliminary logical plan, reducing manual intervention and avoiding the writing of inefficient queries. Based on the target optimization algorithm and operator optimization order, it dynamically optimizes the execution logic, significantly reducing computation and I / O overhead and improving overall performance.

[0053] According to some embodiments, when the predefined operation operator is a predefined federated data connection operator, the corresponding operation execution information and operation execution parameters can be determined according to the operator type of the federated data connection operator; a first format code is generated according to the preset code generation method, operation execution information and operation execution parameters; a data processing request is executed based on the first format code, and the data processing result is determined.

[0054] When the predefined operation operator is a predefined federated data connection operator, for related requests to the federated data connection, the corresponding operation execution information and parameters can be found first based on the operator type. Following a preset code generation method, a first-format code containing the operation execution information and parameters is automatically generated. The data processing request is then executed using this first-format code to obtain the data processing result.

[0055] This application dynamically generates optimized first-format code (such as C++), significantly improving the execution efficiency of complex privacy-preserving computation tasks such as cryptographic computation and secure multi-party computation, achieving performance improvements several times compared to pure JVM implementations. It automatically matches operation execution information and parameters based on operator type, supporting diverse privacy-preserving computation scenarios. By dynamically generating efficient native code, it bypasses the interpretation overhead of the Spark JVM, shortening the time interval from initiation to receiving the final result for computationally intensive tasks such as cryptographic operations and large-scale data connections.

[0056] According to some embodiments, when the predefined operation operator is a predefined federated data connection operator, the first format code can also be converted into binary code according to a preset compiler toolchain.

[0057] In some implementations, the primary format code can be converted into optimized binary code using a pre-defined, efficient compiler toolchain, thereby guiding the execution process.

[0058] This application improves performance in computationally intensive tasks by compiling the first format code into highly optimized binary machine code, which is directly executed by the CPU, completely bypassing the overhead of JVM interpretation or JIT compilation. The native binary code avoids the indirect costs of JVM garbage collection and memory management, compressing end-to-end processing latency from seconds to milliseconds, thus meeting real-time requirements.

[0059] According to some embodiments, when the predefined operation operator is a predefined federated data connection operator, the execution of the data processing request may specifically involve calling the federated data connection operator corresponding to the basic connection requirement from the preset first format code detail operator library according to the basic connection requirement corresponding to the first format code; and processing the corresponding data to be processed based on the federated data connection operator to generate the data processing result.

[0060] Requests related to federated data connectivity are often complex. The first-format code might only provide logical guidance; for example, it might involve more detailed calculations by specific federated data connectivity operators. In such complex cases, the basic connectivity requirements corresponding to the first-format code can be obtained. These basic connectivity requirements can be represented by a general interface. Through this general interface, the federated data connectivity operators corresponding to the basic connectivity requirements in a pre-defined first-format code detail operator library can be invoked. For example, the federated data connectivity operators can process the corresponding data to be processed to generate the data processing results. In some implementations, the first-format code detail operator library is a C++ detail operator library.

[0061] This application utilizes a predefined C++ federated data joiner library to directly call optimized privacy-preserving computation operators, significantly improving the computational efficiency of cross-agency data collaboration while ensuring data privacy and security. It intelligently matches the optimal operator based on fundamental joiner requirements (such as privacy intersection, encrypted JOIN, and federated statistics), supports dynamic loading of different privacy protocols, and meets diverse data compliance requirements.

[0062] According to other embodiments, the overall process for working with federal data connectivity can be referred to Figure 3 .

[0063] 1) Sending a Request: The user submits a cross-source SQL query request, such as `SELECT * FROM HiveTable JOIN MySQLTable ON HiveTable.id=MySQLTable.id`. The core of this step is that the system needs to obtain table schema information (such as field names, data types, and other metadata) from two different databases, Hive and MySQL, and then parse and manage this metadata. For example, the system will pull the table schema information from the MySQL table and manage it in a unified manner with the HiveTable schema information, providing basic support for subsequent query plan generation.

[0064] 2) Request Parsing: After receiving the request, the jydataSQL module parses the SQL statement by combining it with the schema information of the parsed MySQL table. The system analyzes the syntax structure of the SQL query, as well as metadata information such as field names and data types, clarifying the field information and join conditions of the two tables. This step ensures that the table structures of different data sources are managed uniformly, laying the foundation for the next step of generating a query plan. For example, during the parsing process, the system determines that the id field of the Hive table and the id field of the MySQL table are join conditions and verifies data type compatibility, providing necessary metadata support for subsequent query optimization and execution.

[0065] 3) Query Plan Generation and Optimization: The SparkSQL module generates a preliminary logical plan based on the parsed schema and query syntax, and optimizes the physical plan. Based on the user-submitted query, the system generates a logical plan describing JOIN operations and filtering conditions. During the physical plan generation phase, the system iterates through the query execution plan, extracts key information from the data sources, such as table names, database names, and owner information, and performs cross-source checks. When the query involves multiple data owners, the system automatically enables federated computing mode to ensure the security of cross-data source computation. During the physical plan optimization phase, the system adjusts the execution logic according to query requirements, rewrites computation tasks involving cross-source data joins, and replaces them with optimization schemes suitable for federated computing, ensuring both high efficiency in data processing and privacy protection.

[0066] 4) Dynamic Code Generation: In the dynamic code generation phase, after receiving the optimized physical execution plan, the system first deeply analyzes the federated computing nodes, extracting key metadata such as data source type, field attributes, join conditions, and partitioning strategies. For different data source types, the system generates dedicated data reading logic. For example, for Hive data sources, it builds a processing flow including data parsing and validity validation, while for MySQL data sources, it generates processing logic including connection configuration retrieval and result set transformation. After generating the data reading code, the system further constructs a data preprocessing flow, including format conversion, condition filtering, and column extraction, preparing standardized input for subsequent federated computing. Finally, the system generates the calling code for the federated join operators, automatically injecting data partitioning and communication settings to ensure that the generated C++ code meets functional requirements while fully leveraging distributed computing performance. The entire code generation process strictly follows the execution order of the query plan, and through multi-stage code assembly and optimization, it ultimately outputs highly customized, high-performance execution code.

[0067] 5) Compilation and Execution: The system first transforms the dynamically generated C++ code into optimized binary modules using a high-efficiency compiler toolchain. The jydata engine module is responsible for loading these compiled binary codes and calling the built-in high-performance C++ operator library to execute the actual computation tasks. It works deeply with Spark's distributed scheduling mechanism, fully utilizing Spark's native task scheduling, resource management, and fault tolerance mechanisms to execute federated computing tasks and efficiently handle data shards from heterogeneous data sources such as Hive and MySQL. When executing federated data join tasks, the engine reads data shards from heterogeneous data sources such as Hive and MySQL in parallel, and completes secure data alignment and join computation across nodes by calling the function calls of the federated data join dynamic link library. The entire process retains Spark's excellent characteristics in distributed computing while avoiding JVM performance bottlenecks through native C++ execution, ultimately achieving high-performance federated data join computation.

[0068] 6) Generate results: Obtain the output of the federated data connection operator, format the results, and output them to the console or storage system in a user-friendly manner.

[0069] Through the collaborative design of the above system modules and software processes, this application realizes an efficient and reliable federated data connection mechanism, which solves the shortcomings of existing technologies in terms of privacy protection and execution efficiency, and provides a practical solution for privacy protection and efficient computing in a distributed environment.

[0070] The following describes an apparatus embodiment of this application, which can be used to perform the method embodiment of this application. For details not disclosed in the apparatus embodiment of this application, please refer to the method embodiment of this application.

[0071] Figure 4 A block diagram of a data processing apparatus based on dynamic code generation provided in an embodiment of this application. (See diagram below.) Figure 4 As shown, the data processing device 400 based on dynamic code generation includes a request parsing module 401, a plan generation module 402, an operator replacement module 403, and a task execution module 404.

[0072] The request parsing module 401 is used to parse cross-origin data processing requests to determine the data type information and data processing syntax;

[0073] The plan generation module 402 is used to generate a data processing physical plan based on type information and data processing syntax;

[0074] Operator replacement module 403 is used to replace the built-in operators corresponding to the data processing physical plan with predefined operation operators;

[0075] The task execution module 404 is used to generate corresponding first format code according to predefined operation operators, execute data processing requests based on the first format code, and determine the data processing results.

[0076] Optionally, the request parsing module 401 is specifically used for:

[0077] Parse the data processing request to determine the data source, data location, and data processing syntax of the corresponding data to be processed;

[0078] Locate the data source to determine the type information.

[0079] Optionally, the planning generation module 402 is specifically used for:

[0080] Based on type information, data processing syntax, and the pre-defined Spark distributed computing framework, an optimized logical plan is generated.

[0081] Based on the distributed execution characteristics of the cluster and the optimized logical plan, a physical plan for data processing is generated.

[0082] Optionally, the data processing request includes a data processing target; the plan generation module 402, in generating an optimized logical plan based on type information, data processing syntax, and a preset Spark distributed computing framework, is specifically used for:

[0083] The Spark distributed computing framework is controlled to generate a preliminary logical plan corresponding to the data processing request based on the type information and data processing syntax.

[0084] The Spark distributed computing framework is controlled based on the data processing objective, which determines the target optimization algorithm and the order of operator optimization. Based on the target optimization algorithm, the order of operator optimization, and the preliminary logical plan, an optimized logical plan is generated.

[0085] Optionally, if the predefined operation operator is a predefined federated data join operator, the task execution module 404 is specifically used for:

[0086] Based on the operator type of the federated data connection operator, determine the corresponding operation execution information and operation execution parameters;

[0087] Generate first-format code based on the preset code generation method, operation execution information, and operation execution parameters;

[0088] The data processing request is executed based on the first format code, and the data processing result is determined.

[0089] Optionally, the data processing device 400 based on dynamically generated code further includes a format conversion module 405, used for:

[0090] Based on the preset compiler toolchain, the first-format code is converted into binary code.

[0091] Optionally, the task execution module 404, when executing the data processing request based on the first format code and determining the data processing result, is specifically used for:

[0092] Based on the basic connection requirements corresponding to the first format code, the federated data connection operator corresponding to the basic connection requirements is called from the preset first format code detail operator library;

[0093] The corresponding data to be processed is processed based on the federated data connection operator to generate data processing results.

[0094] The device performs functions similar to those described above; other functions are described in the preceding descriptions and will not be repeated here.

[0095] Figure 5 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application, such as... Figure 5 As shown, the electronic device 500 of this embodiment may include a memory 501 and a processor 502.

[0096] The memory 501 stores a computer program, which, when executed by the processor 502, causes the processor 502 to perform the method described in the above embodiments.

[0097] The processor 502 and the memory 501 are connected, for example, via a bus.

[0098] Optionally, the electronic device 500 may also include a transceiver. It should be noted that in practical applications, the transceiver is not limited to one, and the structure of the electronic device 500 does not constitute a limitation on the embodiments of this application.

[0099] Processor 502 may be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It can implement or execute the various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of this application. Processor 502 may also be a combination that implements computational functions, such as including one or more microprocessor combinations, a combination of a DSP and a microprocessor, etc.

[0100] A bus can include a pathway for transmitting information between the aforementioned components. The bus can be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, only one thick line is used in the diagram, but this does not imply that there is only one bus or one type of bus.

[0101] The memory 501 may be a ROM (Read Only Memory) or other type of static storage device capable of storing static information and instructions, RAM (Random Access Memory) or other type of dynamic storage device capable of storing information and instructions, or an EEPROM (Electrically Erasable Programmable Read Only Memory), CD-ROM (Compact Disc Read Only Memory) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), magnetic disk storage media or other magnetic storage devices, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but not limited thereto.

[0102] The memory 501 is used to store application code that executes the solution of this application, and its execution is controlled by the processor 502. The processor 502 is used to execute the application code stored in the memory 501 to implement the content shown in the foregoing method embodiments.

[0103] Electronic devices include, but are not limited to: mobile terminals such as mobile phones, laptops, digital radio receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), and in-vehicle terminals (such as in-vehicle navigation terminals), as well as fixed terminals such as digital TVs and desktop computers. Servers can also be included. Figure 5 The electronic device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of this application.

[0104] The electronic device in this embodiment can be used to execute the method of any of the above embodiments, and its implementation principle and technical effect are similar, so they will not be described again here.

[0105] This application also provides a non-transitory computer-readable storage medium storing computer-readable instructions thereon, which, when executed by a processor, cause the processor to perform the method as described in the above embodiments.

[0106] Those skilled in the art will understand that all or part of the steps of the above-described method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a non-transitory computer-readable storage medium. When executed, the program performs the steps of the above-described method embodiments; and the aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

[0107] The embodiments of this application have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the embodiments above are only for the purpose of helping to understand the method and core ideas of this application. Furthermore, any changes or modifications made by those skilled in the art based on the ideas of this application, and on the specific implementation methods and application scope of this application, are all within the scope of protection of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A data processing method based on dynamic code generation, characterized in that, include: Parse cross-origin data processing requests to determine the data type information and data processing syntax; Generate a data processing physical plan based on the type information and the data processing syntax; Replace the built-in operators corresponding to the data processing physical plan with predefined operation operators; The first format code is generated according to the predefined operation operator to execute the data processing request based on the first format code and determine the data processing result; wherein, the first format code is C++ code.

2. The method according to claim 1, characterized in that, The parsing of cross-source data processing requests to determine data type information and data processing syntax includes: The data processing request is parsed to determine the data source of the corresponding data to be processed, the data location of the data to be processed, and the data processing syntax. Locate the data location of the data source to determine the type information.

3. The method according to claim 1, characterized in that, The step of generating a data processing physical plan based on the type information and the data processing syntax includes: Based on the type information, the data processing syntax, and the preset Spark distributed computing framework, an optimized logical plan is generated; Based on the cluster's distributed execution characteristics and the optimized logical plan, the data processing physical plan is generated.

4. The method according to claim 3, characterized in that, The data processing request includes the data processing target; The step of generating an optimized logic plan based on the type information, the data processing syntax, and the preset Spark distributed computing framework includes: The Spark distributed computing framework is controlled to generate a preliminary logical plan corresponding to the data processing request based on the type information and the data processing syntax. The Spark distributed computing framework is controlled to determine the target optimization algorithm and operator optimization order based on the data processing target, so as to generate the optimization logic plan according to the target optimization algorithm, the operator optimization order and the preliminary logic plan.

5. The method according to claim 1, characterized in that, When the predefined operation operator is a predefined federated data connection operator, the step of generating a corresponding first format code based on the predefined operation operator, executing the data processing request based on the first format code, and determining the data processing result includes: Based on the operator type of the federated data connection operator, determine the corresponding operation execution information and operation execution parameters; The first format code is generated based on the preset code generation method, the operation execution information, and the operation execution parameters; The data processing request is executed based on the first format code, and the data processing result is determined.

6. The method according to claim 5, characterized in that, Also includes: Based on the preset compiler toolchain, the first format code is converted into binary code.

7. The method according to claim 5, characterized in that, The step of executing the data processing request based on the first format code and determining the data processing result includes: Based on the basic connection requirements corresponding to the first format code, the federated data connection operator corresponding to the basic connection requirements is called from the preset first format code detail operator library; The data to be processed is processed based on the federated data connection operator to generate the data processing result.

8. A data processing device based on dynamic code generation, characterized in that, include: The request parsing module is used to parse cross-origin data processing requests to determine the data type information and data processing syntax; The plan generation module is used to generate a data processing physical plan based on the type information and the data processing syntax. The operator replacement module is used to replace the built-in operators corresponding to the data processing physical plan with predefined operation operators; The task execution module is used to generate corresponding first format code according to the predefined operation operator, to execute the data processing request based on the first format code, and to determine the data processing result; wherein, the first format code is C++ code.

9. An electronic device, characterized in that, include: processor; A memory storing a computer program that, when executed by the processor, causes the processor to perform the method as described in any one of claims 1-7.

10. A non-transitory computer-readable storage medium, characterized in that, It stores computer-readable instructions that, when executed by a processor, cause the processor to perform the method as described in any one of claims 1-7.

Images