Data processing method, electronic device, and storage medium

Abstract

Embodiments of the present application provide a data processing method, an electronic device and a storage medium. The method comprises: loading a service configuration, generating a data processing flow based on the service configuration, the data processing flow comprising a plurality of steps; parsing the plurality of steps to determine an execution order of the plurality of steps; generating a SQL segment corresponding to each step based on the execution order; combining the plurality of SQL segments to generate a first SQL statement corresponding to the data processing flow; if the service configuration is updated, obtaining a change set of the service configuration, generating an update SQL segment corresponding to at least one step based on the change set; and reorganizing the plurality of SQL segments in the first SQL statement and the at least one update SQL segment to generate a second SQL statement. The step stack SQL generation method of the embodiments of the present application effectively improves the maintainability of the data processing flow, and the incremental update method based on configuration differences effectively improves the update efficiency of the SQL.

Description

Technical Field

[0001] This application relates to the field of data technology, and in particular to a data processing method, electronic device, and storage medium. Background Technology

[0002] Edge computing scenarios involve a wide variety of devices, protocols, and data formats, resulting in highly heterogeneous data sources. Simultaneously, business requirements iterate rapidly, necessitating frequent adjustments to data processing logic. SQL (Structured Query Language) boasts declarative syntax, standardized interfaces, rich optimization techniques, and a mature tool ecosystem, making it a common foundation for building data processing workflows. The SQL statements corresponding to these workflows are typically generated based on business configurations. However, in related technologies, changes in business configurations require the generation of entirely new SQL statements, leading to low update efficiency and poor adaptability to edge computing scenarios. Summary of the Invention

[0003] In view of the above, it is necessary to provide a data processing method, electronic device and storage medium to solve the problem in the above-mentioned related technologies that when the business configuration of data processing scenarios changes, it is necessary to generate new SQL statements in full, resulting in low SQL statement update efficiency and poor adaptability to edge computing scenarios.

[0004] In a first aspect, embodiments of this application provide a data processing method, the method comprising: Load the business configuration, and generate a data processing flow based on the business configuration. The data processing flow includes multiple steps. The multiple steps are analyzed to determine their execution order; Based on the execution order, generate a Structured Query Language (SQL) fragment corresponding to each step; The multiple SQL fragments are combined to generate the first SQL statement corresponding to the data processing flow; If the business configuration is updated, obtain the change set of the business configuration, and generate at least one update SQL fragment corresponding to the change set; The first SQL statement is reorganized by combining multiple SQL fragments and at least one of the update SQL fragments to generate a second SQL statement.

[0005] In one possible implementation, parsing the plurality of steps and determining the execution order of the plurality of steps includes: The multiple steps are analyzed to determine the type of each step; Create a step object for each step based on the type of each step; Determine the dependencies between the multiple steps based on the step object; Based on the dependencies, a directed acyclic graph corresponding to the data processing flow is constructed, and the execution order of the multiple steps is determined based on the directed acyclic graph.

[0006] In one possible implementation, constructing a directed acyclic graph (DAG) corresponding to the data processing flow based on the dependencies, and determining the execution order of the multiple steps based on the DAG, includes: Based on the aforementioned dependencies, construct the adjacency table for the multiple steps; The directed acyclic graph is constructed based on the adjacency list; Perform a topological sorting algorithm on the directed acyclic graph to generate an execution order list of the multiple steps.

[0007] In one possible implementation, generating the Structured Query Language (SQL) fragment for each step includes: If any step object includes a package type field, the first SQL fragment corresponding to the step whose execution order precedes any step shall be used as the inner input. Load the SQL template based on the package type and assign an alias to any of the steps; Based on the SQL template, the first SQL fragment, and the alias, generate the SQL fragment corresponding to any of the steps.

[0008] In one possible implementation, generating the Structured Query Language (SQL) fragment for each step further includes: If the wrapping rule in any of the steps contains multiple nested layers, obtain the inner SQL fragment; Apply the current layer's wrapping rules to embed the inner layer's SQL fragment into the current layer's SQL template, generating the wrapped SQL fragment.

[0009] In one possible implementation, the step of obtaining the change set of the service configuration if the service configuration is updated includes: A difference detection algorithm is performed on the updated business configuration and the business configuration before the update to identify at least one difference configuration item, and the at least one difference configuration item forms the change set.

[0010] In one possible implementation, generating at least one update SQL fragment corresponding to the changes set includes: Determine the SQL fragment corresponding to each difference configuration item, and determine multiple second SQL fragments based on the dependency relationship between the SQL fragment corresponding to each difference configuration item and other SQL fragments. The multiple second SQL fragments are the SQL fragments that need to be changed. Change the cache status of the multiple second SQL fragments; Check the cache status of each SQL fragment in the first SQL statement. If any SQL fragment is not cached, generate an update SQL fragment corresponding to the uncached SQL fragment.

[0011] In one possible implementation, reorganizing multiple SQL fragments and at least one update SQL fragment in the SQL statement to generate a second SQL statement includes: If the check determines that any SQL fragment is cached, read the cached SQL fragment from the cache; The cached SQL fragment and the updated SQL fragment are combined to generate the second SQL statement.

[0012] Secondly, embodiments of this application provide an electronic device, the electronic device including a memory and a processor: wherein the memory is used to store program instructions; the processor is used to read and execute the program instructions stored in the memory, and when the program instructions are executed by the processor, the electronic device performs the above-described data processing method.

[0013] Thirdly, embodiments of this application provide a computer storage medium storing program instructions that, when executed on an electronic device, cause the processor of the electronic device to perform the aforementioned data processing method.

[0014] The data processing method, electronic device, and storage medium provided in this application can generate a data processing flow based on business configuration, and generate SQL statements based on the execution order of multiple steps in the data processing flow, realizing a step-stack SQL generation method. This effectively improves the SQL generation efficiency and the maintainability of the data processing flow. Furthermore, when business configuration is updated, the differences in business configuration are detected, and the SQL statements are incrementally updated based on the differences, eliminating the need for a full update of the SQL statements. This effectively improves the update efficiency of the SQL statements, thereby enhancing adaptability to edge computing scenarios. Attached Figure Description

[0015] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of this application. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0016] Figure 1 This is a schematic diagram of the architecture of a data processing system provided in an embodiment of this application.

[0017] Figure 2 This is a flowchart of a data processing method provided in an embodiment of this application.

[0018] Figure 3 This is a flowchart illustrating the process of determining the execution order of steps, as provided in an embodiment of this application.

[0019] Figure 4 This is a schematic diagram of a step stack data structure provided in an embodiment of this application.

[0020] Figure 5 This is a flowchart of a wrapper-style SQL generation provided in an embodiment of this application.

[0021] Figure 6 This is a schematic diagram illustrating the interaction between a wrapper generator and related components provided in an embodiment of this application.

[0022] Figure 7 This is a schematic diagram of the step stack and wrap-around generation framework provided in an embodiment of this application.

[0023] Figure 8 This is a schematic diagram of configuration difference detection provided in an embodiment of this application.

[0024] Figure 9 This is a schematic diagram of an SQL fragment cache management structure provided in an embodiment of this application.

[0025] Figure 10 This is a schematic diagram of configuration difference-driven SQL fragment reorganization provided in an embodiment of this application.

[0026] Figure 11 This is a schematic diagram of an edge computing data processing flow provided in an embodiment of this application.

[0027] Figure 12 This is a schematic diagram of the structure of a data processing apparatus provided in an embodiment of this application.

[0028] Figure 13 This is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0029] The terms "first" and "second" used in the embodiments of this application are for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include one or more features. In the description of the embodiments of this application, words such as "exemplary" or "for example" are used to indicate examples, illustrations, or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present the relevant concepts in a concrete manner.

[0030] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains. The terminology used in this application's specification is for the purpose of describing particular embodiments only and is not intended to limit the application. It should be understood that, unless otherwise stated, " / " in this application means "or". For example, A / B can mean A or B. "And / or" in this application is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, and B alone. "At least one" refers to one or more. "More than one" refers to two or more. For example, at least one of a, b, or c can represent: a, b, c, a and b, a and c, b and c, and a, b, and c. Where there is no conflict, the following embodiments and features described herein can be combined with each other.

[0031] With the widespread adoption of IoT and edge computing technologies, data processing is migrating from centralized to edge computing. In edge computing scenarios, data sources exhibit high heterogeneity, with diverse device types, protocols, and data formats. Simultaneously, rapidly iterating business requirements necessitate frequent adjustments to data processing logic. Edge nodes, on the other hand, are typically resource-constrained, with weaker CPU, memory, and storage capabilities compared to data centers, and are often deployed in remote or mobile environments. This places extremely high demands on the system's lightweight design, low latency, autonomy, and remote operation and maintenance capabilities. Facing these challenges, SQL, with its declarative syntax, standardized interfaces, rich optimization techniques, and mature tool ecosystem, has become an ideal choice for building edge data processing pipelines. However, efficiently and accurately translating dynamically changing business configurations into executable SQL statements is no easy task. It involves core challenges such as precise mapping of configuration items to SQL syntax trees, correct combination and dependency management of multiple SQL fragments, and incremental updates and state reuse during configuration changes. Designing a lightweight, flexible, and reliable dynamic SQL generation mechanism that enables edge nodes to respond to configuration changes in real time and generate correct and efficient execution plans has become crucial to the practicality and maintainability of edge data processing systems.

[0032] Edge nodes not only need to process massive amounts of device data but also need to meet requirements such as low latency and local autonomy. However, the environment differs greatly from that of data centers, bringing the following challenges: Dynamic configuration: Edge services change frequently, involving adjustments to multiple levels such as data sources (devices, protocols, formats), processing rules (filtering, aggregation), output targets (databases, APIs), and performance parameters (batch size, concurrency), requiring rapid system response. Resource constraints: Edge devices are mostly embedded or industrial-grade hardware, with CPUs, memory, and storage reduced by tens to thousands of times compared to data centers, requiring extremely lightweight and high-efficiency systems. Ease of operation and maintenance: Devices are mostly deployed in unattended environments, requiring simple configuration, self-diagnosis of errors, and support for remote updates to reduce operation and maintenance costs.

[0033] Faced with these challenges, an increasing number of data processing systems are adopting SQL as their core processing language. SQL's advantage lies in its declarative nature: users only need to describe "what data is needed and what transformations are required," without needing to concern themselves with the specific execution steps. This greatly simplifies the expression and maintenance of business logic. Furthermore, as a standardized language, SQL boasts a broad ecosystem of tools and optimization experience, enabling cross-platform reuse and reducing learning costs and migration difficulties.

[0034] In edge computing scenarios, the application of SQL is gradually expanding from single queries to complete data processing pipelines. A typical data pipeline includes multiple steps, such as data cleaning, field transformation, multi-stream aggregation, window computation, and data output. These steps can be expressed using SQL fragments and chained together to form a directed acyclic graph (DAG) processing flow. The input and output of each step are datasets, and the steps are connected through temporary views or tables. This "multi-step SQL" processing approach not only retains the ease of use of SQL but also naturally supports integrated stream and batch processing, perfectly matching the characteristics of variable and multi-source data on the edge.

[0035] However, while applying SQL to dynamically configured edge processing systems brings convenience, it also introduces new technical challenges: how to dynamically and correctly generate executable SQL statements based on configuration, while ensuring the efficiency and reliability of the generation process. For example, there's the mapping issue from business configuration to SQL: business configurations (such as filter conditions, window definitions, multi-table joins, nested logic, and UDFs) need to be accurately mapped to SQL syntax trees. Simple mappings are easy to handle, but complex mappings such as windows, joins, and nesting can easily lead to syntax or semantic errors, requiring a rigorous mapping model. There's also the issue of combining SQL fragments: pipelines contain multiple SQL fragments (such as SELECT, FROM, WHERE), which must be correctly combined in syntactic order, and field alias dependencies and CTE nested scopes must be properly handled; incorrect order will lead to execution failure. Finally, there's the issue of incremental updates for configuration changes: partial configuration modifications (such as adding filters or modifying groups) require a full SQL regeneration, which is inefficient, prone to state loss, and complex to maintain. Incremental updates are needed—modifying only the affected fragments while retaining the unchanged parts, thus improving responsiveness. In summary, building an efficient and reliable dynamic SQL generator is key to achieving flexible, lightweight, and maintainable data processing in edge computing scenarios.

[0036] SQL generation methods in related technologies include string concatenation, ORM frameworks, template engines, and Abstract Syntax Trees (ASTs). String concatenation is simple to implement, but suffers from problems such as unreliable syntax correctness, high SQL injection risk, and maintenance difficulties. ORM frameworks provide type safety through query builders, but have high learning costs, limited flexibility, and heavy dependencies, making them unsuitable for resource-constrained edge environments. Template engines separate structure and data using templates, but template maintenance is complex, and dynamic combination and incremental updates are difficult. AST solutions construct SQL at the syntax tree level, offering powerful functionality but complex implementation and high maintenance costs, making them too cumbersome in edge scenarios. These technologies all have significant shortcomings when facing the requirements of multi-step pipelines in edge computing, incremental updates driven by configuration differences, and intelligent combination of SQL fragments, and their adaptability to edge environments is poor.

[0037] To address the issues in the aforementioned related technologies where changes in business configuration during data processing scenarios necessitate the generation of entirely new SQL statements, leading to low SQL update efficiency and poor adaptability to edge computing scenarios, this application provides a data processing method. This method generates a data processing flow based on business configuration and generates SQL statements based on the execution order of multiple steps within the data processing flow, implementing a step-stack SQL generation method. This effectively improves SQL generation efficiency and the maintainability of the data processing flow. Furthermore, when business configuration is updated, the method detects the differences in business configuration and incrementally updates the SQL statements based on the differences, eliminating the need for a full SQL update and significantly improving SQL update efficiency, thereby enhancing adaptability to edge computing scenarios.

[0038] In a dynamically configured edge computing data processing system, a flexible, efficient, and maintainable dynamic SQL generation scheme is formed based on how to organize SQL fragments in steps (step-based generation), how to achieve local updates based on configuration differences (incremental reorganization), and how to wrap existing SQL statements into subqueries or CTEs to achieve hierarchical combination (wrapping generation). This scheme addresses the needs of multi-step pipelines, frequent configuration changes, and complex query nesting in edge computing scenarios.

[0039] See Figure 1 The diagram shown is an architectural schematic of a data processing system provided in an embodiment of this application. The data processing method provided in this embodiment is applied to systems such as... Figure 1 The data processing system 100 shown includes, but is not limited to, a configuration layer 10, a core engine layer 20, an output layer 30, and a storage layer 40.

[0040] Configuration Layer 10, located at the top layer of the system, is used to receive and parse user business configurations. The configuration uses YAML (YAML is not Markup Language) format, a human-readable data serialization standard that contains complete information such as data source definitions, processing steps, and output targets. The configuration parser converts the YAML file into an internal data structure for use by subsequent modules. The design of the configuration layer allows business users to adjust data processing logic without modifying the code, realizing a configuration-driven design philosophy.

[0041] The core engine layer 20 includes, but is not limited to: a step stack manager 21, a configuration difference detector 22, an SQL fragment generator 23, a wrapper generator 24, and a cache manager 25. The step stack manager 21 maintains the ordered organization and dependencies of steps. The configuration difference detector 22 identifies configuration changes and analyzes their impact. The SQL fragment generator 23 converts individual steps into SQL fragments. The wrapper generator 24 implements hierarchical nested combinations of SQL. The cache manager 25 runs throughout the entire engine layer, providing efficient caching services for all components.

[0042] Output layer 30 is used to output the generated SQL statements and execution plans. The SQL statements can be directly submitted to the database for execution, while the execution plan describes the execution order and dependencies of each step, facilitating debugging and optimization.

[0043] Storage layer 40 is used to persist cached data and configuration version information, supporting rapid system recovery and incremental updates. The storage layer is implemented using a local file system, requiring no external database support, thus meeting the lightweight requirements of edge computing scenarios.

[0044] The data processing system provided in this application embodiment illustrates the data flow and component relationships between four layers. The configuration layer passes YAML configuration to various components of the core engine layer through a configuration parser; the step stack manager and configuration difference detector manage step organization and configuration changes, respectively; the SQL fragment generator and wrapper generator are responsible for the specific SQL generation work; the cache manager coordinates the caching needs of various components; the finally generated SQL statements and execution plans are output through the output layer; and all cached data and configuration version information are persistently saved through the storage layer.

[0045] See Figure 2 The diagram shown is a flowchart of a data processing method provided in an embodiment of this application. The data processing method is applied in an electronic device and includes: S101: Load business configuration and generate a data processing flow based on the business configuration. The data processing flow includes multiple steps.

[0046] In one embodiment of this application, a YAML configuration file is loaded, and an empty step stack container is initialized. For example, the business configuration may include data acquisition, data preprocessing, data aggregation, and data transmission. Multiple steps in the data processing flow generated based on the business configuration may include: acquiring data from sensor devices and IoT gateways, filtering and aligning the acquired data, aggregating the preprocessed data, and uploading the data to a cloud server.

[0047] S102, parse multiple steps and determine the execution order of multiple steps.

[0048] In one embodiment of this application, the step definition is parsed, and branching is performed based on the step type. For example, for ordinary steps, a Step object is directly created; for conditional steps, the conditional expression is evaluated to determine whether to create an object; for wrapping steps, a WrapStep object is created and a wrapping generator is called. Then, dependencies are parsed, a directed acyclic DAG dependency graph is constructed, and topological sorting is performed to determine the execution order of the steps.

[0049] See Figure 3 The diagram shown is a flowchart illustrating the process of determining the execution order of steps according to an embodiment of this application.

[0050] S1021, parse multiple steps and determine the type of each step.

[0051] In one embodiment of this application, the types of steps include, but are not limited to: ordinary steps, conditional steps, and wrapping steps.

[0052] S1022, Create a step object for each step based on the type of each step.

[0053] In one embodiment of this application, the steps are basic data processing units. Each step is represented by a Step object, and the step data structure definition is shown in Table 1.

[0054] Table 1 In this structure, the `name` field serves as a unique identifier for each step and forms the basis for dependency referencing; the `template` field stores the SQL template string, employing a placeholder mechanism for parameterized generation; the `params` field stores template parameters, decoupling configuration from SQL; the `dependencies` field establishes dependencies through step name references; the `condition` field supports conditional execution, enabling flexible business rules; the `config_hash` field is used for version control caching, supporting incremental updates; and the `wrap_type` field identifies the wrapping type, supporting hierarchical composition. This structural design allows step objects to express both SQL fragment content and dependencies between steps, while also supporting conditional execution and wrapping-style generation, forming the data foundation for implementing the step stack SQL generation method.

[0055] The step stack uses a stack data structure as its underlying container, supporting last-in-first-out (LIFO) operations. The core operation primitives provided by the step stack manager are shown in Table 2. Table 2 The `PUSH` operation pushes a new step onto the top of the stack, performing a name uniqueness check beforehand to avoid duplicate definitions; the `POP` operation removes a specified step by name, automatically updating affected dependencies afterward; the `WRAP` operation applies wrapping rules to a specified step, converting it into a `WrapStep` object, supporting hierarchical composition; the `INSERT` operation inserts a new step at a specified position, supporting flexible adjustments; the `REPLACE` operation replaces a specified step with a new step, preserving existing dependencies; and the `TOPO_SORT` operation performs a topological sort on all steps in the stack, ensuring correct dependency order. These operation primitives collectively constitute the API interface of the step stack manager, supporting dynamic addition, deletion, and flexible adjustment of steps, and are the core mechanism for implementing the step stack SQL generation method.

[0056] See Figure 4 The diagram shown is a schematic of a step stack data structure provided in an embodiment of this application. The step stack structure includes an example of a five-step stack structure: data cleaning, field transformation, conditional filtering, aggregation calculation, and result output. These steps are organized in stack order to form a complete data processing pipeline. The internal structure of each step includes six core fields of the step object: name, SQL template, parameter list, dependency list, condition expression, and configuration hash. Figure 4 This demonstrates the data organization method of the step stack method, which is the data structure foundation for implementing step stack SQL generation.

[0057] In one embodiment of this application, the conditional step corresponds to a conditional execution mechanism. Conditional execution supports dynamically determining whether to execute a step based on runtime configuration. Conditional expressions include: Boolean expressions: such as `config.debug == true`, determining whether to execute based on the Boolean value of the configuration item; comparison expressions: such as `config.batch_size>100`, determining whether to execute based on the numerical comparison result of the configuration item; existence checks: such as `config.filter exists`, checking if the configuration item exists; and function calls: such as `is_enabled("feature_x")`, calling a predefined function for complex judgments. The condition evaluator evaluates the conditional expression before step execution: if the condition is true, the step is executed normally, generating an SQL fragment. If the condition is false, the step is skipped, no SQL fragment is generated, but its position in the dependency graph is retained (other steps depending on this step need to handle empty dependencies). The conditional execution mechanism allows the data processing flow to be flexibly adjusted according to different business scenarios, enabling on / off control of functions without modifying the code.

[0058] S1023, Determine the dependencies between multiple steps based on the step object.

[0059] In one embodiment of this application, dependency management is used to ensure the correct execution order of steps, employing a Directed Acyclic Graph (DAG) model to manage the dependencies between steps. A DAG is a graph data structure that includes vertices and directed edges, and there is no path that starts from a vertex, traverses several edges, and returns to that vertex.

[0060] In one embodiment of this application, a dependency resolver traverses all step objects, extracts the reference relationships in the `dependencies` field of each step, and determines the dependencies between multiple steps. Dependencies are implemented using step name references. In the `dependencies` field of a `Step` object, other steps are referenced using the `stepname` syntax. For example, if step B depends on step A, then B's `dependencies` list will contain "A". This referencing method is simple, intuitive, and easy to understand and maintain.

[0061] S1024, construct a directed acyclic graph corresponding to the data processing flow based on the dependency relationship, and determine the execution order of multiple steps based on the directed acyclic graph.

[0062] In one embodiment of this application, an adjacency list for multiple steps is constructed based on dependencies, a directed acyclic graph is constructed based on the adjacency list, a topological sorting algorithm is performed on the directed acyclic graph, and an execution order list for multiple steps is generated.

[0063] Specifically, a dependency graph represented by an adjacency list is constructed based on the dependencies between multiple steps. Adjacency lists are an efficient graph storage structure suitable for sparse graph scenarios. Cycle detection is performed to check for circular dependencies, throwing an exception when a cycle is detected. Circular dependencies can lead to deadlocks, so they are detected and eliminated before execution. A topological sorting algorithm (such as Kahn's algorithm or DFS algorithm) is performed on the DAG graph to generate an execution order list. The result of the topological sorting ensures that for any given step, all its dependent steps are executed before it.

[0064] S103 generates SQL fragments for each step based on the execution order.

[0065] In one embodiment of this application, based on the execution order of the steps, the template parameters of each step are filled into the corresponding SQL template to generate the corresponding SQL fragment, thereby obtaining multiple SQL fragments.

[0066] In one embodiment of this application, the SQL fragment generation includes a wrapping SQL generation method. The wrapping SQL generation method defines wrapping rules to embed an existing SQL fragment as a subquery or CTE (Common Table Expression) into another SQL fragment, thereby achieving hierarchical combination and solving the problems of syntax correctness and alias management in complex SQL nested combinations.

[0067] The wrapper-style SQL generation method defines wrapper rules, manages aliases, handles multi-level nesting, and ensures syntax correctness. Specifically, it predefines various wrapper types (SUBQUERY, CTE, AGGREGATE, FILTER, etc.), describing the wrapper logic through templates and parameters, making the inner SQL part of the outer SQL. It automatically generates unique aliases (e.g., _subquery_1, _cte_2) to avoid naming conflicts and records the mapping between aliases and SQL fragments. In cases of multi-level nesting, it recursively applies wrapper rules, supporting arbitrary depths of nesting while maintaining a clear SQL structure. It employs a structured generation method, using template placeholder replacement rather than string concatenation to ensure the correct syntax of the wrapped SQL.

[0068] The wrapping rules define how to embed inner SQL into the structure of outer SQL. Wrapping rule types include SUBQUERY, CTE, AGGREGATE, FILTER, and JOIN. The SUBQUERY type embeds the inner SQL as a subquery into the outer FROM clause, suitable for simple nested query scenarios. The CTE type defines the inner SQL as a common table expression, suitable for scenarios requiring multiple references to the same subquery. The AGGREGATE type adds aggregation calculations to the inner SQL, suitable for grouped statistical scenarios. The FILTER type adds filtering conditions to the inner SQL, suitable for conditional filtering scenarios. The JOIN type implements table joins, suitable for multi-table combined queries. Each rule type has a predefined template structure, applicable scenarios, and alias generation format to ensure that the generated SQL syntax is correct and easy to understand.

[0069] Each wrapping rule type is defined through a WrapRule object, which contains a template string, a parameter list, and validation rules. The template string uses the {{ placeholder}} syntax to define placeholders, which are replaced with actual values ​​at runtime. The wrapping rule types are shown in Table 3.

[0070] Table 3 In one embodiment of this application, alias management ensures that table aliases in nested SQL statements are unique, avoiding naming conflicts. The alias generation rules are shown in Table 4.

[0071] Table 4 The alias manager employs several tactical methods: sequential numbering generates aliases in sequence for simplicity and intuitiveness; step association generates aliases based on step names for easy tracking and debugging; timestamp rules generate aliases based on the current timestamp to ensure uniqueness; UUID short code rules generate 8-digit short codes based on UUIDs, suitable for high-concurrency scenarios; and hash value rules generate 6-digit short codes based on content hashes for easy identification when associated with content. The alias manager maintains an alias registry, recording used aliases and their corresponding fragment information, and checks the registry to ensure uniqueness when generating new aliases.

[0072] In one embodiment of this application, multi-level nesting is achieved by recursively applying wrapping rules. The wrapping generator uses a recursive algorithm, with the recursion terminating when the current layer is the innermost SQL, at which point it returns directly. Inner layer generation: Recursively call the wrapping generator to obtain the innermost SQL. Current layer wrapping: Apply the wrapping rules of the current layer to embed the innermost SQL into the template. Alias ​​assignment: Assign a unique alias to the current layer and record the mapping relationship. Result return: Return the wrapped SQL. The recursive process starts from the innermost layer and wraps outwards layer by layer, passing the alias reference of the upper layer at each level to ensure clear hierarchy. This recursive method supports nested combinations of arbitrary depth, meeting the needs of complex queries. For example, if the wrapping rules of any step contain multiple levels of nesting, obtain the innermost SQL fragment; apply the wrapping rules of the current layer to embed the innermost SQL fragment into the current layer's SQL template, and return the wrapped SQL fragment.

[0073] In one embodiment of this application, syntactic correctness is guaranteed through a structured generation method, avoiding the syntactic error risks of traditional string concatenation. Template validation ensures the correctness of the predefined template through syntactic verification. Placeholder checking checks whether all placeholders have been provided with values ​​before filling the template. Parentheses matching ensures that parentheses appear in pairs within the template, maintaining matching during recursion. Keyword checking avoids SQL keyword conflicts, such as aliases not using reserved words. Syntax tree validation is optionally performed after generation.

[0074] See Figure 5The diagram shown is a flowchart of a packaged SQL generation method provided in one embodiment of this application. In one embodiment of this application, if the step object of any step includes a package type field, the first SQL fragment corresponding to the step that precedes any step in the execution order is used as the inner input, an SQL template is loaded based on the package type, an alias is assigned to any step, and an SQL fragment corresponding to any step is generated based on the SQL template, the first SQL fragment, and the alias.

[0075] Starting with receiving the inner SQL and wrapper type parameters, the process loads the corresponding template based on the wrapper type (SUBQUERY, CTE, AGGREGATE, FILTER); generates a unique alias; populates template placeholders; determines if multi-level nesting is needed, and if so, recursively calls the wrapper generation function; performs syntax correctness checks; and finally outputs the generated result. This process implements hierarchical SQL composition, supports various wrapper types such as subqueries, CTEs, aggregations, and filters, ensures syntax correctness through template replacement, and avoids naming conflicts through alias management.

[0076] See Figure 6 The diagram illustrates the interaction between a wrapper generator and related components according to an embodiment of this application. The wrapper generator is invoked, passing in the inner SQL and wrapper type parameters; the wrapper generator requests the template manager to load a template of the corresponding type; the template manager returns a template object; the wrapper generator requests the alias manager to generate a unique alias; the alias manager returns the alias; the wrapper generator fills template placeholders; it requests a syntax check from the syntax checker; the syntax checker returns the check result; if multiple levels of nesting are required, the wrapper generator recursively calls itself; finally, the final generated SQL is returned. This sequence diagram illustrates the collaborative relationship between the components in the wrapper generation process, forming the interactive foundation for implementing hierarchical SQL generation.

[0077] This application combines a step stack generation method with a wrapper generation method. The step stack supports wrapper-type steps, enabling more flexible and powerful SQL generation capabilities and addressing the need for both step-based organization and hierarchical combination in complex data processing scenarios. The combination of the step stack generation method and the wrapper generation method achieves a unified framework. WRAP-type steps are introduced into the step stack, and the wrapper generator is called during step execution, wrapping the upstream SQL as the inner input. Seamless integration is achieved: the step stack handles overall process organization and dependency management, while the wrapper generation handles hierarchical combination. The two work together to retain the clear structure of step-based processes while possessing the ability to express complex nested structures. Configuration-driven implementation is also provided; the wrapper type and parameters can be specified in the step configuration, which is entirely defined by YAML and supports runtime hot updates.

[0078] See Figure 7The diagram illustrates a framework for step stack and wrapper generation provided in an embodiment of this application. The unified framework for step stack and wrapper generation includes a configuration-driven layer, a collaborative layer, a step stack management layer, and a wrapper generation layer. The configuration-driven layer contains YAML configuration, defining steps and wrapping rules; the collaborative layer contains a coordinator, coordinating the work of the step stack manager and the wrapper generator. The coordinator coordinates the work of the step stack manager and the wrapper generator. A WRAP type step (WrapStep) is introduced into the step stack, and its configuration includes the wrapping type and wrapping parameters. When the step stack executes to a WRAP type step, the coordinator calls the wrapper generator, wrapping the SQL generated by the upstream step as the inner input. The step stack management layer includes the step stack manager and three operation primitives: PUSH, POP, and WRAP; the wrapper generation layer includes the wrapper generator and four wrapping types: subquery, CTE, aggregation, and filtering. Input configuration enters the system through the configuration-driven layer, the coordinator coordinates the work of each component, and finally outputs the generated SQL fragment. This framework achieves seamless integration of the step stack and wrapper generation.

[0079] For example, the collaborative workflow of the step stack and wrapper generation includes: parsing the YAML configuration to identify regular steps and wrapper steps; adding all steps to the step stack and establishing dependencies; performing topological sorting on the step stack to determine the execution order; and executing each step sequentially. Specifically, for regular steps, SQL fragments are generated directly. For wrapper steps, the wrapper generator is invoked, using upstream SQL as inner input. Finally, all generated SQL fragments are combined into the final result.

[0080] The combined application of step stacks and wrapper-style generation is driven by YAML configuration, enabling complex SQL generation logic without code modification. For example, the YAML configuration code is as follows: Furthermore, the combined application of step stacks and wrapper generation supports runtime hot updates, eliminating the need to restart the system after configuration changes. Specifically, the system monitors for changes in configuration files. When a configuration change occurs, a difference detector is invoked to analyze the changes. Based on the changes, the affected steps and wrapper rules are updated. The state and cache of unchanged parts remain intact. Service remains uninterrupted during the new configuration's application process.

[0081] S104 combines multiple SQL fragments to generate the first SQL statement corresponding to the data processing flow.

[0082] In one embodiment of this application, multiple SQL fragments are combined using methods such as CTE chaining, temporary tables, and view layering to generate a first SQL statement corresponding to the data processing flow, and the first SQL statement is output.

[0083] S105, if the business configuration is updated, obtain the change set of the business configuration, and generate an update SQL fragment corresponding to at least one step based on the change set.

[0084] This application applies a configuration difference-driven SQL fragment reorganization method. By detecting configuration differences, it updates the affected SQL fragments and reuses the unchanged parts using a cache, achieving efficient incremental updates and solving the efficiency problem in SQL generation technology where a full regeneration of SQL is required when configuration changes.

[0085] In one embodiment of this application, a difference detection algorithm is performed on the updated business configuration and the business configuration before the update to identify at least one difference configuration item, which forms a change set. The SQL fragment corresponding to each difference configuration item is determined, and multiple second SQL fragments are determined based on the dependencies between the SQL fragment corresponding to each difference configuration item and other SQL fragments. These multiple second SQL fragments are the SQL fragments that need to be changed.

[0086] Specifically, a configuration difference comparison algorithm (such as JSON Diff) is used to generate change sets and accurately locate configuration change points. A mapping relationship (ConfigMapping) is established between configuration items and SQL fragments. The affected steps are located through the change sets, and dependency propagation analysis is used to determine the SQL fragments that need to be regenerated. SQL fragments are regenerated for affected steps, while other steps directly retrieve them from the cache (SQLFragmentCache), avoiding a full regeneration. SQL fragments are managed through caching; a configuration hash value is calculated for each SQL fragment as a version identifier, and the results are cached. When changes occur, related fragments and their dependencies are automatically invalidated to ensure data consistency.

[0087] In one embodiment of this application, a JSON Diff algorithm is used to compare the old and new configurations to generate a structured ChangeSet. The configuration difference types and impact analysis table are shown in Table 5.

[0088] Table 5 Adding configuration items is detected through key-value comparison, affecting only the addition step; the handling strategy is to generate a new fragment and insert it. Modifying configuration values ​​is detected through value comparison, affecting the current step and dependent steps; the handling strategy is to regenerate the affected fragment. Deleting configuration items is detected through missing keys, affecting the deletion step and dependent steps; the handling strategy is to remove the fragment and update dependencies. Structural adjustments are detected through structural comparison, affecting a wide range of areas; the handling strategy is to regenerate all elements. This table provides clear rules for configuration difference detection and impact analysis, forming the foundation for implementing the incremental update mechanism.

[0089] In one embodiment of this application, the difference detection algorithm performs a depth-first search comparison. It recursively traverses the two configuration objects. For basic types (e.g., strings, numbers, booleans), it directly compares whether the values ​​are equal. For object types, it recursively compares each attribute. For array types, it compares each element by index. All differences are recorded, generating a changeset.

[0090] See Figure 8 The diagram illustrates a configuration difference detection method provided in one embodiment of this application. Configuration difference detection is implemented by an input layer, a difference detection engine, and an output layer. The input layer receives two JSON objects: an old configuration and a new configuration. The difference detection engine includes a depth-first search comparator, a change type classifier, and an impact scope analyzer. The depth-first search comparator compares configuration differences, the change type classifier determines the change type, and the impact scope analyzer analyzes the scope of impact. The output layer generates a change set, a list of affected fragments, and cached items that need to be invalidated. This architecture enables automated detection and impact analysis of configuration differences.

[0091] Establishing a mapping relationship between configuration items and SQL fragments is crucial for achieving accurate impact analysis. This mapping relationship is maintained through a ConfigMapping object, containing information such as configuration path, fragment ID, fragment type, and scope of impact. Table 6 shows the mapping relationship between configuration items and SQL fragments.

[0092] Table 6 The configuration path indicates the position of the configuration item in the configuration tree; for example, steps[0].filter represents the filtering condition for step 0. The corresponding fragment ID is the unique identifier of the SQL fragment generated by this configuration item. The fragment type indicates the type of SQL clause for this fragment, such as WHERE clause, GROUP BY clause, etc. The scope of influence indicates the range of steps affected by the change of this configuration item. By establishing this mapping relationship, the system can quickly locate the affected SQL fragments when configuration changes occur, achieving accurate incremental updates.

[0093] The mapping relationship is established during system initialization. By traversing the configuration items of all steps, the correspondence between them and SQL fragments is analyzed. Establishing the mapping relationship is a key step in the configuration difference-driven refactoring method, directly affecting the accuracy and efficiency of incremental updates.

[0094] After identifying the differing configuration items, the impact analyzer iterates through the changeset, determining the change type based on the configuration path. It uses ConfigMapping to find the SQL fragment ID corresponding to each changed configuration item. For each affected fragment, it recursively searches for other SQL fragments that depend on it and adds them to the impact set. All affected SQL fragment IDs are merged, and duplicates are removed. A list of fragment tasks that need to be regenerated is created.

[0095] The impact analysis results in a set of SQL fragment IDs, representing all SQL fragments that need to be regenerated. Through dependency propagation analysis, the system can accurately determine the scope of impact of configuration changes, avoid unnecessary regeneration, and improve the efficiency of incremental updates.

[0096] See Figure 9 The diagram shown illustrates an embodiment of the SQL fragment cache management structure provided in this application. This embodiment employs a multi-level caching strategy. The cache management structure includes a cache management layer, a cache storage area, an index structure, and an invalidation policy. The cache management layer (CacheManager) is the core controller, coordinating the work of each component. The cache storage area stores multiple SQL fragment caches, each containing a hash value and SQL content. The index structure includes a configuration path index, a step name index, and a hash value index, supporting fast lookup. The invalidation policy includes dependency propagation invalidation, version number checking, and LRU (Least Recently Used) eviction, ensuring cache consistency and effectiveness. This structure achieves efficient cache management and invalidation control, serving as the core support for the configuration difference-driven reorganization method. The cache management strategies are shown in Table 7.

[0097] Table 7 Cache management employs multiple invalidation strategies: dependency propagation invalidation is triggered upon configuration changes, automatically invalidating affected fragments and their dependent fragments to ensure data consistency; version number checks are performed before each generation, determining cache validity by comparing configuration hash values, with low performance overhead; LRU (Least Recently Used) eviction is triggered when the cache is full, removing the least recently used cache item to ensure efficient use of cache space; full clearing is triggered when the configuration structure undergoes fundamental changes, clearing all caches and regenerating; and pre-loading is performed at system startup, pre-generating frequently used fragments to improve first-access speed. These strategies collectively ensure efficient cache management and data consistency.

[0098] S106, Reorganize multiple SQL fragments and at least one update SQL fragment in the first SQL statement to generate the second SQL statement.

[0099] In one embodiment of this application, the cache status of multiple second SQL fragments is changed, and the cache status of each SQL fragment in the first SQL statement is checked. If any SQL fragment is not cached, an updated SQL fragment corresponding to the uncached SQL fragment is generated. If the check determines that any SQL fragment is cached, the cached SQL fragment is read from the cache. The cached SQL fragment and the updated SQL fragment are combined to generate the second SQL statement.

[0100] See Figure 10 The diagram illustrates a configuration-difference-driven SQL fragment reorganization process according to an embodiment of this application. Upon receiving a new configuration, the old configuration is loaded, and a JSON Diff algorithm is executed to generate a changeset. Branching is performed based on the change type (e.g., add, modify, delete) to establish a mapping relationship between configuration and fragments. Dependency propagation analysis is performed to determine the set of fragments that need to be regenerated. The cache status of each fragment is checked; if the cache hits, it is read directly; otherwise, it is regenerated. All SQL fragments (newly generated SQL fragments and cached reused SQL fragments) are combined into a complete SQL statement. The cache status is updated, and finally, the generated result is output. This process enables incremental updates for configuration changes, requiring only updates to the affected fragments while reusing the unchanged parts of the cache, significantly reducing the response time for configuration changes.

[0101] See Figure 11 The diagram shown is a schematic of an edge computing data processing flow provided in an embodiment of this application. For example, in an edge computing scenario, the data source layer includes four types of data sources: sensor devices, industrial equipment, IoT gateways, and API interfaces; the edge node internally includes a data access system, an SQL processing engine, and an output adaptation system. The data access system is used for protocol adaptation and format conversion, the SQL processing engine includes a step stack manager, a configuration difference detector, an SQL generator, and an execution engine, and the output adaptation system supports local storage, cloud upload, and real-time alarms.

[0102] In edge computing scenarios, the components in this embodiment are loaded on demand, reducing startup overhead. Frequently created objects are reused, reducing GC pressure. Caching is prioritized to reduce redundant computations. When configuration changes occur, only the changed parts need to be processed. Table 8 shows a performance comparison between this embodiment and other frameworks.

[0103] Table 8 To address the resource limitations of edge devices, multiple performance optimization strategies are employed. The memory usage of this embodiment is less than 5MB, representing a 90-95% improvement compared to SQLAlchemy's 50-100MB and MyBatis's 20-50MB. Startup time is less than 10ms, an improvement of over 99% compared to SQLAlchemy's 1-3 seconds and MyBatis's 0.5-1 seconds. Code size is less than 100KB, an improvement of over 99% compared to SQLAlchemy's 10MB+ and MyBatis's 5MB+. Furthermore, this embodiment has zero external dependencies, while mainstream frameworks have multiple dependencies. This embodiment supports incremental updates with a time of less than 1ms, while mainstream frameworks do not support incremental updates.

[0104] This application's embodiments utilize a step-stack SQL generation method to achieve intuitive step-based organization. Through stack structure and dependency management, the multi-step data processing flow becomes clear and maintainable. This application's embodiments utilize a configuration difference-driven SQL fragment reorganization method to achieve efficient incremental updates. Through difference detection and cache reuse, the response time for configuration changes is reduced by 60-90%. This application's embodiments utilize a wrapper-style SQL generation method to achieve flexible hierarchical combination. Through predefined wrapper rules and automatic alias management, it supports SQL nesting of arbitrary depth. This application's embodiments combine the step-stack SQL generation method and the wrapper-style SQL generation method, coordinating step stack and wrapper-style generation within a unified framework. This retains the clear step-based structure while possessing the expressive power of complex nesting, fully adapting to the resource constraints and operational needs of edge computing scenarios.

[0105] Taking a real-world business scenario as an example, the database table `sql_step_stack` is designed as the metadata storage for the step stack operations. This includes `step_order` (step execution order), `operation_type` (operation type), `input_fields` (list of input fields), `output_field` (output field names), `parameters` (operation parameters), and `parent_step` (parent step number). This table structure reflects the SQL generation method for the step stack, using the `step_order` field for intuitive step organization and the `parent_step` field for dependency management, making the multi-step data processing flow clear and maintainable.

[0106] Connect to the data source that needs to be processed, such as the vehicle_orders table of a car manufacturer's sales data, which contains fields such as order ID, vehicle model, unit price, and quantity.

[0107] Configure the nodes and data processing types at the processing nodes, add them to the step stack, save the configuration, and store the operation parameters through the parameters field in JSON format. When the configuration changes, the system can automatically detect the differences and perform incremental updates, improving performance by 60-90%.

[0108] The system executes the data processing flow according to the configured step stack. Each step generates an independent SQL fragment, which is automatically combined hierarchically, supporting SQL nesting of any depth. For example, the SQL nesting level is: This unified framework integrates a step stack and wrapper-style generation, preserving the clear structure of step-by-step generation while also possessing the expressive power for complex nesting. The unified framework architecture includes a step stack configuration table, an SQL generation engine (combining step stack and wrapper-style generation), and output SQL statements. For example, a combined application example of step stack and wrapper-style generation is as follows: See Figure 12 The diagram shown is a structural schematic of a data processing apparatus provided in one embodiment of this application. In one embodiment of this application, the data processing apparatus 200 may include multiple functional modules composed of computer program segments. The computer program segments in the data processing apparatus 200 may be stored in the memory of an electronic device and executed by at least one processor to perform data processing functions.

[0109] In one embodiment of this application, the data processing device 200 can be divided into multiple functional modules according to the functions it performs. The functional modules of the data processing device 200 may include: a first generation module 201, a parsing module 202, a second generation module 203, a combination module 204, a third generation module 205, and a reassembly module 206. In this embodiment, a module refers to a series of computer program segments that can be executed by at least one processor and perform a fixed function, and which are stored in memory.

[0110] The first generation module 201 is used to load business configuration and generate a data processing flow based on the business configuration. The data processing flow includes multiple steps.

[0111] The parsing module 202 is used to parse multiple steps and determine the execution order of the multiple steps.

[0112] The second generation module 203 is used to generate SQL fragments corresponding to each step based on the execution order.

[0113] The combination module 204 is used to combine multiple SQL fragments to generate the first SQL statement corresponding to the data processing flow.

[0114] The third generation module 205 is used to obtain the change set of the business configuration if the business configuration is updated, and generate at least one update SQL fragment based on the change set.

[0115] The reorganization module 206 is used to reorganize multiple SQL fragments and at least one update SQL fragment in the first SQL statement to generate a second SQL statement.

[0116] This application also provides an electronic device 1, see embodiments thereof. Figure 13 The diagram shown is a hardware structure schematic of an electronic device according to an embodiment of this application. The data processing method provided in this embodiment is applied to electronic device 1. Electronic device 1 may be a personal computer, server, etc. Electronic device 1 includes, but is not limited to, a processor 110 and a memory 120 connected via a communication bus 130. Figure 13 This is merely an example of an electronic device and does not constitute a limitation thereof. In other embodiments, the electronic device may include more components than those shown in the figure.

[0117] The memory 120 may include one or more random access memory (RAM) and one or more non-volatile memory (NVM). The RAM can be directly read and written by the processor 110 and can be used to store executable programs (such as machine instructions) of the operating system or other running programs, as well as user and application data. The RAM may include static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), etc.

[0118] Non-volatile memory can also store executable programs and user and application data, and can be pre-loaded into random access memory for direct reading and writing by the processor 110. Non-volatile memory can include disk storage devices and flash memory.

[0119] The memory 120 is used to store one or more computer programs. The one or more computer programs are configured to be executed by the processor 110. The one or more computer programs include multiple instructions that, when executed by the processor 110, can implement a data processing method that can be executed on the electronic device 1.

[0120] In other embodiments, the electronic device 1 further includes an external memory interface for connecting to an external memory to expand the storage capacity of the electronic device 1.

[0121] Processor 110 may include one or more processing units, such as an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural network processing unit (NPU). These different processing units may be independent devices or integrated into one or more processors.

[0122] The processor 110 provides computing and control capabilities. For example, the processor 110 is used to execute computer programs stored in the memory 120 to implement the data processing method described above.

[0123] The communication bus 130 is used to provide a channel for communication between the memory 120 and the processor 110 in the electronic device 1.

[0124] It is understood that the structures illustrated in the embodiments of this application do not constitute a specific limitation on the electronic device 1. In other embodiments of this application, the electronic device 1 may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0125] This application also provides a computer storage medium storing computer instructions. When the computer instructions are executed on the electronic device 1, the electronic device 1 performs the aforementioned related method steps to implement the data processing method described in the above embodiments.

[0126] This application also provides a computer program product that, when run on a computer, causes the computer to perform the aforementioned steps to implement the data processing method described in the above embodiments.

[0127] In addition, embodiments of this application also provide an apparatus, which may specifically be a chip, component or module. The apparatus may include a connected processor and a memory; wherein the memory is used to store computer execution instructions, and when the apparatus is running, the processor may execute the computer execution instructions stored in the memory to cause the chip to execute the data processing methods in the above-described method embodiments.

[0128] In this application, the electronic devices, computer storage media, computer program products or chips provided in the embodiments are all used to execute the corresponding methods provided above. Therefore, the beneficial effects they can achieve can be referred to the beneficial effects in the corresponding methods provided above, and will not be repeated here.

[0129] Through the above description of the embodiments, those skilled in the art can clearly understand that, for the sake of convenience and brevity, only the division of the above functional modules is used as an example. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0130] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another device, or some features may be ignored or not executed. Furthermore, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0131] The units described as separate components may or may not be physically separate. A component shown as a unit can be one or more physical units; that is, it can be located in one place or distributed in multiple different locations. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0132] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0133] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solutions of the embodiments of this application, or the parts or all or part of the technical solutions that contribute to the prior art, can be embodied in the form of a software product. The software product is stored in a storage medium and includes several instructions to cause a device (which may be a microcontroller, chip, etc.) or processor to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0134] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application and are not intended to limit it. Although this application has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of this application without departing from the spirit and scope of the technical solutions of this application.

Claims

1. A data processing method, characterized in that, The method includes: Load the business configuration, and generate a data processing flow based on the business configuration. The data processing flow includes multiple steps. The multiple steps are analyzed to determine their execution order; Based on the execution order, generate a Structured Query Language (SQL) fragment corresponding to each step; The multiple SQL fragments are combined to generate the first SQL statement corresponding to the data processing flow; If the business configuration is updated, obtain the change set of the business configuration, and generate at least one update SQL fragment corresponding to the change set; The first SQL statement is reorganized by combining multiple SQL fragments and at least one of the update SQL fragments to generate a second SQL statement.

2. The data processing method as described in claim 1, characterized in that, The step of parsing the multiple steps and determining the execution order of the multiple steps includes: The multiple steps are analyzed to determine the type of each step; Create a step object for each step based on the type of each step; Determine the dependencies between the multiple steps based on the step object; Based on the dependencies, a directed acyclic graph corresponding to the data processing flow is constructed, and the execution order of the multiple steps is determined based on the directed acyclic graph.

3. The data processing method as described in claim 2, characterized in that, The process of constructing a directed acyclic graph (DAG) corresponding to the data processing flow based on the dependencies, and determining the execution order of the multiple steps based on the DAG, includes: Based on the aforementioned dependencies, construct the adjacency table for the multiple steps; The directed acyclic graph is constructed based on the adjacency list; Perform a topological sorting algorithm on the directed acyclic graph to generate an execution order list of the multiple steps.

4. The data processing method as described in claim 2, characterized in that, The generation of Structured Query Language (SQL) fragments for each step includes: If any step object includes a package type field, the first SQL fragment corresponding to the step whose execution order precedes any step shall be used as the inner input. Load the SQL template based on the package type and assign an alias to any of the steps; Based on the SQL template, the first SQL fragment, and the alias, generate the SQL fragment corresponding to any of the steps.

5. The data processing method as described in claim 4, characterized in that, The generation of Structured Query Language (SQL) fragments for each step also includes: If the wrapping rule in any of the steps contains multiple nested layers, obtain the inner SQL fragment; Apply the current layer's wrapping rules to embed the inner layer's SQL fragment into the current layer's SQL template, generating the wrapped SQL fragment.

6. The data processing method as described in claim 1, characterized in that, If the service configuration is updated, the change set of the service configuration is obtained, including: A difference detection algorithm is performed on the updated business configuration and the business configuration before the update to identify at least one difference configuration item, and the at least one difference configuration item forms the change set.

7. The data processing method as described in claim 6, characterized in that, The step of generating at least one update SQL fragment corresponding to the change set includes: Determine the SQL fragment corresponding to each difference configuration item, and determine multiple second SQL fragments based on the dependency relationship between the SQL fragment corresponding to each difference configuration item and other SQL fragments. The multiple second SQL fragments are the SQL fragments that need to be changed. Change the cache status of the multiple second SQL fragments; Check the cache status of each SQL fragment in the first SQL statement. If any SQL fragment is not cached, generate an update SQL fragment corresponding to the uncached SQL fragment.

8. The data processing method as described in claim 7, characterized in that, The step of reorganizing multiple SQL fragments and at least one update SQL fragment in the SQL statement to generate a second SQL statement includes: If the check determines that any SQL fragment is cached, read the cached SQL fragment from the cache; The cached SQL fragment and at least one of the updated SQL fragments are combined to generate the second SQL statement.

9. An electronic device, characterized in that, The electronic device includes a memory and a processor: The memory is used to store program instructions; The processor is configured to read and execute the program instructions stored in the memory, and when the program instructions are executed by the processor, cause the electronic device to perform the data processing method as described in any one of claims 1 to 8.

10. A computer storage medium, characterized in that, The computer storage medium stores program instructions that, when executed on the electronic device, cause the processor of the electronic device to perform the data processing method as described in any one of claims 1 to 8.

Images