Data processing method and device based on dynamic code generation

Abstract

The invention provides a data processing method and device based on dynamic code generation, and relates to the technical field of data processing. The data processing method based on dynamic code generation comprises the following steps: analyzing a cross-source data processing request to determine type information and data processing grammar of data; generating a data processing physical plan according to the type information and the data processing grammar; replacing a built-in operator corresponding to the data processing physical plan with a predefined operation operator; and generating a corresponding first format code according to the predefined operator, executing the data processing request based on the first format code, and determining a data processing result. According to the method and the device, in a scene related to large-scale data and high concurrency, federal data calculation logic is dynamically generated into efficient C + + codes, the efficient C + + codes are sunk to the local for execution, the performance bottleneck of JVM is bypassed, meanwhile, the advantages of Spark in the aspects of distributed calculation, task scheduling and resource management are fully utilized, and the data processing efficiency is greatly improved.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a data processing method and device based on dynamic code generation. Background Art

[0002] With the rapid development of big data technology, distributed data analysis has become a core means of processing massive amounts of data. However, in data-driven applications, privacy protection and data security have become critical issues that cannot be ignored. Especially in fields involving sensitive data, such as finance, healthcare, and government affairs, achieving efficient data sharing and analysis in a distributed environment while ensuring data privacy and security has become a major challenge facing current technologies. While traditional distributed computing frameworks perform well in processing large-scale data, their designs do not fully consider the needs of privacy-preserving computing. During cross-node data sharing and collaborative computing, data often needs to be transmitted between nodes, which can lead to the leakage of sensitive information. Although existing encryption technologies and multi-party secure computation can alleviate the problem of sensitive information leakage to some extent, these methods are often accompanied by a significant decrease in computational efficiency, making it difficult to meet the performance requirements of large-scale data processing while ensuring privacy.

[0003] Existing distributed computing frameworks, such as Spark, are based on the Scala language (a multi-paradigm programming language) and run on the JVM (Java Virtual Machine). While these frameworks offer excellent cross-platform support and development convenience, they suffer from inherent performance limitations. The JVM's garbage collection mechanism and interpreted execution approach limit its performance when handling complex computing tasks, particularly in scenarios involving large-scale data and high concurrency, resulting in low data processing efficiency. Summary of the Invention

[0004] The present application aims to provide a data processing method and device based on dynamic code generation.

[0005] According to one aspect of the present application, a data processing method based on dynamic code generation is proposed, including: parsing a cross-source data processing request to determine the type information and data processing syntax of the data; generating a data processing physical plan based on the type information and data processing syntax; replacing the built-in operator corresponding to the data processing physical plan with a predefined operation operator; generating a corresponding first format code based on the predefined operation operator, so as to execute the data processing request based on the first format code and determine the data processing result.

[0006] According to one aspect of the present application, a data processing device based on dynamic code generation is proposed, comprising: The request parsing module is used to parse cross-source data processing requests to determine the data type information and data processing syntax; A plan generation module is used to generate a data processing physical plan based on type information and data processing syntax; Operator replacement module, used to replace the built-in operators corresponding to the data processing physical plan with predefined operation operators; The task execution module is used to generate a corresponding first format code according to a predefined operation operator, to execute a data processing request based on the first format code, and to determine a data processing result.

[0007] According to one aspect of the present application, an electronic device is provided, comprising: a processor; and a memory storing a computer program, wherein when the computer program is executed by the processor, the processor executes the method as described above.

[0008] According to one aspect of the present application, a non-transitory computer-readable medium is provided, on which readable instructions are stored. When the instructions are executed by a processor, the processor is caused to execute the method described above.

[0009] It should be understood that the foregoing general description and the following detailed description are merely illustrative and are not restrictive of the present application.

[0010] Beneficial effects: Through the above-mentioned embodiments provided by this application, by dynamically generating the federated data computing logic into efficient C++ code and sinking it to local execution, the performance bottleneck of the JVM is bypassed, while making full use of Spark's advantages in distributed computing, task scheduling and resource management, greatly improving data processing efficiency. The use of dynamic code generation technology, combined with Spark's distributed architecture, enables privacy computing tasks to be executed efficiently in parallel, significantly reducing computing latency and supporting high-performance federated computing requirements in large-scale data scenarios. Through targeted transformation of operators, it is fully adapted to the Spark environment, retaining Spark's ease of use and scalability, and achieving higher-performance privacy computing through C++ underlying optimization, providing an efficient and secure solution for complex data collaboration scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without exceeding the scope of protection required by this application.

[0012] Figure 1A schematic diagram of the module architecture of a data processing system based on dynamic code generation provided in an embodiment of the present application; Figure 2 A flowchart of a data processing method based on dynamic code generation provided in an embodiment of the present application; Figure 3 A flowchart of a federated data connection provided in an embodiment of the present application; Figure 4 A block diagram of a data processing device based on dynamic code generation provided in an embodiment of the present application; Figure 5 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0013] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be embodied in many forms and should not be construed as limited to the embodiments set forth herein; rather, these embodiments are provided so that this disclosure will be thorough and complete and will fully convey the concepts of the example embodiments to those skilled in the art. Like reference numerals in the drawings represent like or similar parts, and thus repetitive description thereof will be omitted.

[0014] In addition, described feature, structure or characteristic can be combined in one or more embodiments in any suitable manner.In the following description, many specific details are provided so as to provide a full understanding of the embodiments of the present application. However, it will be appreciated by those skilled in the art that the technical scheme of the present application can be put into practice without one or more of the specific details, or other methods, components, devices, steps etc. can be adopted. In other cases, known methods, devices, implementations or operations are not shown or described in detail to avoid blurring the various aspects of the application.

[0015] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically separate entities. That is, these functional entities may be implemented in software, in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices.

[0016] The flowcharts shown in the accompanying drawings are for illustrative purposes only and do not necessarily include all contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps may be decomposed, while others may be combined or partially combined. Therefore, the actual execution order may vary depending on the actual situation.

[0017] It should be understood that although the terms first, second, third, etc. may be used herein to describe various components, these components should not be limited by these terms. These terms are used to distinguish one component from another. Thus, the first component discussed below could be referred to as the second component without departing from the teachings of the present invention. As used herein, the term "and / or" includes any one and all combinations of one or more of the associated listed items.

[0018] Figure 1 A schematic diagram of the module architecture of a data processing system based on dynamic code generation provided in an embodiment of the present application. Figure 1 This is a complete system architecture diagram corresponding to the federated data connection, such as Figure 1 As shown, the system can include a data acquisition and parsing module (i.e., jydataSQL), a plan generation and optimization module (i.e., SparkSQL), a code generation module (i.e., jydata code generator), and an operator invocation module (i.e., jydata engine). The jydataSQL module provides users or other programs with access and operation access through a programming interface, supporting cross-source data pulling and schema parsing. It can acquire data from multiple different data sources (such as Hive, MySQL, etc.), breaking the limitations of data storage locations, enabling centralized data aggregation, and providing a data foundation for subsequent processing. Furthermore, the jydataSQL module parses the data's schema information (such as column names and data types) to ensure that the system can correctly process and use the data.

[0019] The SparkSQL module can generate and optimize query plans. Based on the query input and database schema information, SparkSQL generates a preliminary logical plan and improves query execution efficiency by optimizing operator order and selecting more optimal algorithms. Based on the optimized logical plan, a physical plan is generated, taking into account the distributed nature of the cluster. In the physical plan, when federated data joins are involved, SparkSQL can replace built-in operators with custom federated data join operators to support privacy protection and efficient computation.

[0020] The jydata code generator module deeply analyzes SparkSQL-generated query plans (i.e., the physical plan after replacing operators) and dynamically generates efficient first-format code. In this application, C++ code can be used as the first-format code. The generated code is optimized for specific data sources, fields, and query conditions, avoiding the redundant calculations and performance bottlenecks of general-purpose code.

[0021] The jydata engine module leverages a high-performance C++ operator library and combines Spark's strengths in distributed computing, task scheduling, and resource management to achieve efficient data processing and computation. By dynamically loading and executing optimized first-format code, the jydata engine significantly improves computing performance while avoiding the performance bottlenecks of traditional distributed runtime environments. Leveraging Spark's flexible scheduling and efficient parallel computing capabilities, the module ensures efficient execution of computing tasks and optimal resource utilization, providing strong support for large-scale distributed data processing.

[0022] For specific implementation methods, please refer to the following embodiments.

[0023] Figure 2 This is a flow chart of a data processing method based on dynamic code generation provided in an embodiment of the present application. The method of this embodiment can be applied to a data processing server. Figure 2 As shown, the method includes: step S20, step S21, step S22 and step S23.

[0024] In step S20, the cross-source data processing request is parsed to determine the data type information and data processing syntax.

[0025] In this application, data processing requests may involve multiple devices, and the data to be processed may involve multiple data sources. This cross-source data processing request can be input by the user, or input when accessed by other programs, and may include data type information and data processing syntax. The format of the data processing request can be an SQL statement, such as SELECT*FROM HiveTable JOIN MySQLTable ON HiveTable.id = MySQLTable.id. First, the schema information of the table (such as field name, data type, etc.) is pulled from Hive and MySQL, and the schema information is parsed and managed to determine the type information and data processing syntax. Among them, the data processing syntax is the specific syntax for subsequent data processing operations such as queries.

[0026] In some implementations, data processing syntax and type information can be extracted directly from the data processing request.

[0027] In step S21, a data processing physical plan is generated according to the type information and the data processing syntax.

[0028] In the present application, the plan generation module may be a SparkSQL module, which has the function of generating a data processing physical plan based on the parsed type information and data processing syntax.

[0029] In step S22, the built-in operators corresponding to the data processing physical plan are replaced with predefined operation operators.

[0030] This application can pre-introduce technologies such as the extended Oblivious Transfer (OT) protocol, lightweight Oblivious Pseudorandom Function (OPRF), and a parallelized federated alignment protocol to build predefined operators compatible with Spark's native architecture. Different built-in operators have corresponding predefined operators. The physical processing plan can be used to determine which built-in operators are needed and replace them with predefined operators.

[0031] In step S23, a corresponding first format code is generated according to the predefined operation operator, so as to execute the data processing request based on the first format code and determine the data processing result.

[0032] This application can pre-set the jydata code generator, which can generate the first format code corresponding to the predefined operation operator. The first format code in this application is C++ code. Under the guidance of the C++ code, the operation corresponding to the corresponding data processing request can be executed to obtain the data processing result.

[0033] This application bypasses the performance bottleneck of the JVM by dynamically generating the federated data computing logic into efficient C++ code and sinking it to local execution. At the same time, it fully utilizes Spark's advantages in distributed computing, task scheduling, and resource management to significantly improve data processing efficiency. The use of dynamic code generation technology, combined with Spark's distributed architecture, enables privacy computing tasks to be executed efficiently in parallel, significantly reducing computing latency and supporting high-performance federated computing requirements in large-scale data scenarios. Through targeted modifications to operators, it is fully adapted to the Spark environment, retaining Spark's ease of use and scalability, while achieving higher-performance privacy computing through C++ underlying optimization, providing an efficient and secure solution for complex data collaboration scenarios.

[0034] According to some embodiments, during the process of parsing a data processing request, the data processing request can be parsed to determine the data source of the corresponding data to be processed, the data location of the data to be processed, and the data processing syntax; and the data location of the data source can be found to determine the type information.

[0035] This application can directly extract the data source, data location and data processing syntax of the data to be processed from the data processing request, and then find the preset type information corresponding to the data source in the corresponding data location.

[0036] This application automatically identifies the data source, location, and processing syntax of the data to be processed by parsing data processing requests, ensuring accurate data acquisition, avoiding manual configuration errors, and improving the reliability of the data processing process. It supports unified parsing and positioning of different data sources, enabling efficient integration of heterogeneous data and reducing the complexity of collaborative processing of multi-source data.

[0037] According to some embodiments, a data processing physical plan can be generated by generating an optimized logical plan. Specifically, the optimized logical plan is generated based on type information, data processing syntax, and a preset Spark distributed computing framework; and the data processing physical plan is generated based on the cluster distributed execution characteristics and the optimized logical plan.

[0038] This application first leverages the capabilities of the Spark distributed computing framework to generate a logical plan based on type information and data processing syntax, disregarding some of the features. This plan is then optimized using a pre-defined optimization method to obtain an optimized logical plan. The optimized logical plan is then adjusted to a physical data processing plan by selecting an execution strategy based on the distributed execution characteristics of the cluster.

[0039] This application uses the Spark distributed computing framework to optimize logical plans, reduce redundant data scanning and calculations, and lower I / O and CPU overhead. It also dynamically adjusts physical plans based on cluster resources to avoid data skew and computational hotspots, and improve task parallelism.

[0040] According to some embodiments, a data processing request includes a data processing target. During the process of generating an optimized logical plan, the Spark distributed computing framework can be controlled to generate a preliminary logical plan corresponding to the data processing request based on the type information and the data processing syntax. The Spark distributed computing framework can also be controlled to determine a target optimization algorithm and an operator optimization sequence based on the data processing target, so as to generate an optimized logical plan based on the target optimization algorithm, the operator optimization sequence, and the preliminary logical plan.

[0041] In this application, the data processing goal can be used to characterize the effect that is desired to be achieved during data processing, such as reducing the amount of calculation or transmission.

[0042] In some implementations, the Spark distributed computing framework can be controlled to generate a preliminary logical plan corresponding to the data processing request based on the type information and the data processing syntax. The Spark distributed computing framework can then select a preset target optimization algorithm and operator optimization sequence corresponding to the data processing target. The Spark distributed computing framework can then optimize the preliminary logical plan according to the target optimization algorithm and adjust the sequence of operators therein to obtain an optimized logical plan.

[0043] This application automatically parses type information and data processing syntax to generate a preliminary logical plan, reducing manual intervention and avoiding inefficient manual query writing. Based on the target optimization algorithm and operator optimization sequence, it dynamically optimizes the execution logic, significantly reducing computation and I / O overhead and improving overall performance.

[0044] According to some embodiments, when the predefined operation operator is a predefined federated data connection operator, the corresponding operation execution information and operation execution parameters can be determined specifically based on the operator type of the federated data connection operator; a first format code can be generated based on a preset code generation method, operation execution information, and operation execution parameters; a data processing request is executed based on the first format code, and a data processing result is determined.

[0045] If the predefined operation operator is a predefined federated data connection operator, for requests related to the federated data connection, the corresponding operation execution information and operation execution parameters can be first searched based on the operator type. A first-format code containing the operation execution information and operation execution parameters is automatically generated according to a preset code generation method. The data processing request is executed using the first-format code to obtain a data processing result.

[0046] This application dynamically generates optimized first-format code (e.g., C++), significantly improving the execution efficiency of complex privacy-focused computing tasks such as cryptographic operations and secure multi-party computation, achieving several times the performance improvement compared to pure JVM. It automatically matches operation execution information and parameters based on operator type, supporting diverse privacy-focused computing scenarios. By dynamically generating efficient native code, it bypasses the interpretation and execution overhead of Spark JVM, shortening the time between initiation and final result reception for computationally intensive tasks such as cryptographic operations and large-scale data joins.

[0047] According to some embodiments, when the predefined operation operator is a predefined federated data connection operator, the first format code may be converted into binary code according to a preset compiler tool chain.

[0048] In some implementations, the first format code may be converted into optimized binary code through a preset efficient compiler tool chain to guide the execution process.

[0049] This application improves performance in compute-intensive tasks by compiling the first-format code into highly optimized binary machine code that is executed directly by the CPU, completely bypassing the overhead of JVM interpretation or JIT compilation. Native binary code avoids the JVM's garbage collection and memory management overhead, reducing end-to-end processing latency from seconds to milliseconds, meeting real-time requirements.

[0050] According to some embodiments, when the predefined operation operator is a predefined federated data connection operator, executing the data processing request can specifically call the federated data connection operator corresponding to the basic connection requirement from the preset first format code detail operator library according to the basic connection requirement corresponding to the first format code; and process the corresponding data to be processed based on the federated data connection operator to generate a data processing result.

[0051] Requests related to federated data connections are relatively complex, and the first format code may only be a logical guide, for example, which federated data connection operators need to be involved in a more detailed calculation process. In this complex situation, the basic connection requirements corresponding to the first format code can be obtained, and the basic connection requirements can be a related universal interface. Through this universal interface, the federated data connection operators corresponding to the basic connection requirements in the preset first format code detail operator library can be called, for example, the federated data connection operators process the corresponding data to be processed to generate data processing results. In some implementations, the first format code detail operator library is a C++ detail operator library.

[0052] This application leverages a predefined C++ federated data connection operator library to directly call optimized privacy-preserving computational operators, significantly improving the computational efficiency of cross-institutional data collaboration while ensuring data privacy and security. It intelligently matches the optimal operator based on basic connection requirements (such as privacy intersection, encrypted join, and federated statistics), supports dynamic loading of different privacy protocols, and meets diverse data compliance requirements.

[0053] According to other embodiments, the overall process of the federated data connection work can refer to Figure 3 .

[0054] 1) Sending a Request: The user submits a cross-source SQL query request, such as SELECT * FROM HiveTable JOIN MySQLTable ON HiveTable.id = MySQLTable.id. The core of this step is that the system needs to obtain table schema information (such as field names, data types, and other metadata) from two different databases, Hive and MySQL, and parse and manage this metadata. For example, the system will pull the table schema information from the MySQL table and manage it together with the HiveTable schema information, providing basic support for subsequent query plan generation.

[0055] 2) Request Parsing: After receiving the request, the jydataSQL module parses the SQL statement based on the parsed MySQL table schema information. The system analyzes the SQL query's syntax and metadata, such as field names and data types, to identify the field information and join conditions for the two tables. This step ensures unified management of table structures across different data sources, laying the foundation for query plan generation. For example, during parsing, the system identifies the Hive table's id field and the MySQL table's id field as the join condition and verifies data type compatibility, providing the necessary metadata support for subsequent query optimization and execution.

[0056] 3) Query plan generation and optimization: The SparkSQL module generates a preliminary logical plan based on the parsed schema and query syntax, and optimizes the physical plan. Based on the query submitted by the user, the system generates a logical plan that describes the JOIN operation and filter conditions. In the physical plan generation phase, the system traverses the query execution plan, extracts key information of the data source, such as table name, database name and owner information, and performs cross-source judgment. When it is determined that the query involves multiple data owners, the system automatically enables the federated computing mode to ensure the security of cross-data source computing. In the physical plan optimization phase, the system adjusts the execution logic according to the query requirements, rewrites the computing tasks involving cross-source data connections, and replaces them with optimization solutions suitable for federated computing to ensure the efficiency of data processing and privacy protection capabilities.

[0057] 4) Dynamic Code Generation: During the dynamic code generation phase, after receiving the optimized physical execution plan, the system first deeply analyzes the federated compute nodes, extracting key metadata such as data source type, field attributes, join conditions, and partitioning strategies. For different data source types, the system generates dedicated data reading logic. For example, for Hive data sources, a processing flow is constructed that includes data parsing and validity verification, while for MySQL data sources, processing logic is generated that includes connection configuration acquisition and result set conversion. After completing data reading code generation, the system further constructs a data preprocessing flow, including operations such as format conversion, condition filtering, and column extraction, to prepare standardized input for subsequent federated computations. Finally, the system generates the calling code for the federated join operator, automatically injecting data partitioning and communication settings, ensuring that the generated C++ code meets functional requirements while fully leveraging distributed computing performance. The entire code generation process strictly adheres to the query plan execution order. Through multi-stage code assembly and optimization, the final output is highly customized, high-performance execution code.

[0058] 5) Compilation and Execution: The system first converts the dynamically generated C++ code into an optimized binary module through an efficient compiler toolchain. The jydata engine module is responsible for loading these compiled binary codes and calling the built-in high-performance C++ operator library to perform the actual computing tasks. It deeply collaborates with Spark's distributed scheduling mechanism, making full use of Spark's native task scheduling, resource management, and fault-tolerance mechanisms to execute federated computing tasks and efficiently process data shards from heterogeneous data sources such as Hive and MySQL. When executing federated data connection tasks, the engine reads data shards from heterogeneous data sources such as Hive and MySQL in parallel, and completes cross-node secure data alignment and connection calculations by calling the functional functions of the federated data connection dynamic link library. The entire process not only retains Spark's excellent features in distributed computing, but also circumvents the performance bottleneck of the JVM through native C++ execution, ultimately achieving high-performance federated data connection calculations.

[0059] 6) Generate results: Get the output of the federated data connection operator, format the results, and output them to the console or storage system in a user-friendly manner.

[0060] Through the collaborative design of the above-mentioned system modules and software processes, this application realizes an efficient and reliable federated data connection mechanism, solves the shortcomings of existing technologies in privacy protection and execution efficiency, and provides a practical solution for privacy protection and efficient computing in distributed environments.

[0061] The following describes an apparatus embodiment of the present application, which can be used to perform the method embodiment of the present application. For details not disclosed in the apparatus embodiment of the present application, reference can be made to the method embodiment of the present application.

[0062] Figure 4 This is a block diagram of a data processing device based on dynamic code generation provided in an embodiment of the present application. Figure 4 As shown, the data processing device 400 based on dynamic code generation includes a request parsing module 401, a plan generating module 402, an operator replacing module 403 and a task executing module 404.

[0063] Request parsing module 401, used to parse the cross-source data processing request to determine the data type information and data processing syntax; A plan generation module 402 is used to generate a data processing physical plan based on the type information and the data processing syntax; Operator replacement module 403, used to replace the built-in operator corresponding to the data processing physical plan with a predefined operation operator; The task execution module 404 is configured to generate a corresponding first format code according to the predefined operation operator, execute the data processing request based on the first format code, and determine a data processing result.

[0064] Optionally, the request parsing module 401 is specifically configured to: Parse the data processing request to determine the data source, data location, and data processing syntax of the corresponding data to be processed; Look up the data location of the data source to determine the type information.

[0065] Optionally, the plan generation module 402 is specifically configured to: Generate an optimized logical plan based on type information, data processing syntax, and the preset Spark distributed computing framework; Generate a data processing physical plan based on the cluster's distributed execution characteristics and optimized logical plan.

[0066] Optionally, the data processing request includes a data processing target; when the plan generation module 402 generates an optimized logical plan based on the type information, the data processing syntax, and the preset Spark distributed computing framework, it is specifically configured to: Control the Spark distributed computing framework to generate a preliminary logical plan corresponding to the data processing request based on type information and data processing syntax; The Spark distributed computing framework is controlled to determine the target optimization algorithm and operator optimization sequence based on the data processing target, so as to generate an optimized logical plan according to the target optimization algorithm, operator optimization sequence and preliminary logical plan.

[0067] Optionally, when the predefined operation operator is a predefined federated data connection operator, the task execution module 404 is specifically configured to: Determine the corresponding operation execution information and operation execution parameters based on the operator type of the federated data connection operator; Generate a first format code according to a preset code generation method, operation execution information, and operation execution parameters; The data processing request is executed based on the first format code, and a data processing result is determined.

[0068] Optionally, the data processing device 400 based on dynamic code generation further includes a format conversion module 405, which is used to: The first format code is converted into binary code according to a preset compiler tool chain.

[0069] Optionally, when executing the data processing request based on the first format code and determining the data processing result, the task execution module 404 is specifically configured to: According to the basic connection requirement corresponding to the first format code, calling the federated data connection operator corresponding to the basic connection requirement from the preset first format code detail operator library; The corresponding data to be processed is processed based on the federated data connection operator to generate data processing results.

[0070] The device performs functions similar to the method provided above. For other functions, please refer to the previous description and will not be repeated here.

[0071] Figure 5 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application, such as Figure 5 As shown, the electronic device 500 of this embodiment may include: a memory 501 and a processor 502.

[0072] The memory 501 stores a computer program. When the computer program is executed by the processor 502 , the processor 502 executes the method in the above embodiment.

[0073] The processor 502 and the memory 501 are connected, for example, via a bus.

[0074] Optionally, the electronic device 500 may further include a transceiver. It should be noted that in actual applications, the number of transceivers is not limited to one, and the structure of the electronic device 500 does not constitute a limitation on the embodiments of the present application.

[0075] Processor 502 may be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It may implement or execute the various exemplary logic blocks, modules, and circuits described in conjunction with the present disclosure. Processor 502 may also be a combination that implements computing functions, such as a combination of one or more microprocessors, or a combination of a DSP and a microprocessor.

[0076] A bus includes a path that transmits information between the components mentioned above. Examples include a PCI (Peripheral Component Interconnect) bus and an EISA (Extended Industry Standard Architecture) bus. Buses can be categorized as address buses, data buses, and control buses. For ease of illustration, the diagram uses a single thick line, but this does not imply a single bus or type of bus.

[0077] The memory 501 may be a ROM (Read Only Memory) or other type of static storage device that can store static information and instructions, a RAM (Random Access Memory) or other type of dynamic storage device that can store information and instructions, or an EEPROM (Electrically Erasable Programmable Read Only Memory), a CD-ROM (Compact Disc Read Only Memory) or other optical disk storage, optical disk storage (including compact discs, laser discs, optical discs, digital versatile discs, Blu-ray discs, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto.

[0078] The memory 501 is used to store application code for executing the solution of the present application, and the execution is controlled by the processor 502. The processor 502 is used to execute the application code stored in the memory 501 to implement the content shown in the above method embodiment.

[0079] Electronic devices include, but are not limited to, mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), and in-vehicle terminals (e.g., in-vehicle navigation terminals), as well as fixed terminals such as digital TVs and desktop computers. They may also include servers, etc. Figure 5 The electronic device shown is merely an example and should not limit the functions and scope of use of the embodiments of the present application.

[0080] The electronic device of this embodiment can be used to execute the method of any of the above embodiments. Its implementation principles and technical effects are similar and will not be described in detail here.

[0081] The present application also provides a non-transitory computer-readable storage medium having computer-readable instructions stored thereon. When the aforementioned instructions are executed by a processor, the processor executes the method in the above embodiment.

[0082] Those skilled in the art will appreciate that all or part of the steps in the above-described method embodiments can be implemented using hardware associated with program instructions. The aforementioned program can be stored in a non-transitory computer-readable storage medium. When executed, the program performs the steps of the above-described method embodiments. The aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

[0083] The embodiments of the present application are described in detail above. Specific examples are used herein to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only intended to help understand the method and core ideas of the present application. At the same time, changes or modifications made by those skilled in the art based on the ideas of the present application, the specific implementation methods, and the scope of application of the present application, all fall within the scope of protection of the present application. In summary, the contents of this specification should not be construed as limiting the present application.

Claims

1. A data processing method based on dynamic code generation, characterized in that: include: Parse cross-origin data processing requests to determine data type information and data processing syntax; generating a data processing physical plan according to the type information and the data processing syntax; Replacing the built-in operators corresponding to the data processing physical plan with predefined operation operators; A corresponding first format code is generated according to the predefined operation operator, so as to execute the data processing request based on the first format code and determine a data processing result.

2. The method according to claim 1, characterized in that The cross-source data processing request is parsed to determine the data type information and data processing syntax, including: Parsing the data processing request to determine a data source of corresponding to-be-processed data, a data location of the to-be-processed data, and the data processing syntax; The data location of the data source is searched to determine the type information.

3. The method according to claim 1, characterized in that Generating a data processing physical plan according to the type information and the data processing syntax includes: Generate an optimized logical plan based on the type information, the data processing syntax, and a preset Spark distributed computing framework; The data processing physical plan is generated based on the cluster distributed execution characteristics and the optimized logical plan.

4. The method according to claim 3, characterized in that The data processing request includes a data processing target; The step of generating an optimization logic plan based on the type information, the data processing syntax, and the preset Spark distributed computing framework includes: Controlling the Spark distributed computing framework to generate a preliminary logical plan corresponding to the data processing request according to the type information and the data processing syntax; The Spark distributed computing framework is controlled to determine a target optimization algorithm and an operator optimization sequence based on the data processing target, so as to generate the optimized logical plan according to the target optimization algorithm, the operator optimization sequence and the preliminary logical plan.

5. The method according to claim 1, wherein In a case where the predefined operation operator is a predefined federated data connection operator, generating a corresponding first format code according to the predefined operation operator, executing the data processing request based on the first format code, and determining a data processing result includes: Determining corresponding operation execution information and operation execution parameters according to the operator type of the federated data connection operator; Generate the first format code according to a preset code generation method, the operation execution information and the operation execution parameters; The data processing request is executed based on the first format code, and a data processing result is determined.

6. The method according to claim 5, characterized in that Also includes: The first format code is converted into binary code according to a preset compiler tool chain.

7. The method according to claim 5, characterized in that The executing the data processing request based on the first format code and determining a data processing result includes: According to the basic connection requirement corresponding to the first format code, calling the federated data connection operator corresponding to the basic connection requirement from a preset first format code detail operator library; The corresponding data to be processed is processed based on the federated data connection operator to generate the data processing result.

8. A data processing device based on dynamic code generation, characterized in that: include: The request parsing module is used to parse cross-source data processing requests to determine the data type information and data processing syntax; a plan generation module, configured to generate a data processing physical plan based on the type information and the data processing syntax; An operator replacement module, used to replace the built-in operator corresponding to the data processing physical plan with a predefined operation operator; A task execution module is used to generate a corresponding first format code according to the predefined operation operator, to execute the data processing request based on the first format code, and to determine a data processing result.

9. An electronic device, characterized in that: include: processor; A memory storing a computer program, which, when executed by the processor, causes the processor to perform the method according to any one of claims 1 to 7.

10. A non-transitory computer-readable storage medium, characterized in that Computer-readable instructions are stored thereon, and when the instructions are executed by a processor, the processor is caused to perform the method according to any one of claims 1 to 7.

Images