Big data application scenarized development method and device

Abstract

The invention relates to the technical field of big data application development, and particularly discloses a big data application scenarized development method and device.The method comprises the steps that historical task execution data in a big data platform is analyzed so as to identify, verify and package high-frequency and stable-process scene workflow, and the historical task execution data is stored in the big data platform; forming a reusable scene workflow template library; providing a visual development interface containing a scene workflow template library for a user; responding to a selection operation of a user on a target template in the scene workflow template library, and loading a multi-layer editable structure corresponding to the target template in the visual development interface; receiving operation environment parameters input by a user to the multi-layer editable structure; and generating and storing an executable target big data application workflow based on the received operation environment parameters and the component dependency relationship, the data flow direction agreement and the parameter transfer rule in the target template. According to the embodiment, the development efficiency of the big data application is effectively improved, and multi-scene requirements can be met.

Description

Technical Field

[0001] This invention relates to the field of big data application development technology, and in particular to a method and apparatus for scenario-based development of big data applications. Background Technology

[0002] With the surge in enterprise data volume and the deepening of business analysis, big data applications have become core support for decision-making and innovation. Traditional big data application development is a highly specialized and complex process. Development teams not only need a deep understanding of business requirements but also must be proficient in diverse and rapidly evolving big data technology stacks, such as Apache Flink, Spark, Kafka, various data warehouses, and cloud-native services. From data collection, cleaning, and computation to service exposure, it involves a large amount of low-level code writing, component selection, parameter tuning, and operation and maintenance configuration, resulting in long development cycles, high technical barriers, and huge human resource costs. To reduce development difficulty, low-code / no-code big data development platforms characterized by drag-and-drop and visual configuration have emerged in the market. These platforms encapsulate commonly used functions such as data preprocessing, algorithms, and visualization into graphical components, allowing users to combine components through drag-and-drop to build data processing workflows. This, to some extent, liberates developers from tedious low-level coding, allowing them to focus more on business logic.

[0003] However, in practice, especially in industries such as finance and telecommunications, when facing the development needs of complex, standardized, and large-scale reusable data products, existing low-code solutions still have significant limitations. First, existing solutions suffer from bottlenecks in development efficiency and reusability. Most existing solutions provide basic, atomic functional components. When building complex end-to-end applications such as automatically generating branch regulatory reports or calculating real-time risk control indicators, developers still need to start from scratch, manually dragging and dropping dozens of components and configuring complex connections and parameter passing between them one by one. Even if the business processes are highly similar, differing only in parameters such as institution and date, this repetitive configuration work is unavoidable, leading to low development efficiency and difficulty in ensuring consistency of processing logic between different instances. Second, existing solutions involve technical complexity and configuration burden. Although visualization is achieved, platforms often expose a large number of technical parameters (such as the parallelism of the computing engine, memory configuration, checkpoint settings, and data serialization formats) to the configurer. This requires the configurer to still have considerable technical background to perform reasonable optimization; otherwise, it can easily lead to poor application performance or failure, failing to truly achieve a complete separation of business and technical concerns.

[0004] There is currently no effective solution to the above problems. Summary of the Invention

[0005] This specification provides a method and apparatus for developing big data application scenarios, in order to solve the problems of low development efficiency and high technical complexity in existing low-code development solutions.

[0006] This specification provides an embodiment of a big data application scenario-based development method, including:

[0007] Analyze historical task execution data within the big data platform to identify, verify, and encapsulate high-frequency and stable scenario workflows, forming a reusable scenario workflow template library; each scenario workflow template in the scenario workflow template library encapsulates multiple big data processing components as well as the dependencies, data flow conventions, and parameter passing rules between these multiple big data processing components;

[0008] Provide users with a visual development interface containing the scenario workflow template library; in response to the user's selection operation of the target template in the scenario workflow template library, load the multi-layer editable structure corresponding to the target template in the visual development interface;

[0009] The system receives runtime environment parameters input by the user for the multi-layer editable structure; the runtime environment parameters include at least one of the following: workflow identifier, access operation permission, data endpoint, execution scheduling policy, and resource declaration parameters;

[0010] Based on the received runtime environment parameters, component dependencies, data flow conventions, and parameter passing rules within the target template, an executable target big data application workflow is generated and saved.

[0011] In one embodiment, historical task execution data within a big data platform is analyzed to identify, verify, and encapsulate high-frequency and stable workflow scenarios, including:

[0012] Collect the execution directed acyclic graph, component configuration snapshots, and key performance indicators of historical task instances that have been successfully executed within the big data platform;

[0013] Each historical task instance is abstracted into a task graph with big data processing components as nodes and data streams as edges, resulting in a set of task graphs;

[0014] A frequent subgraph mining algorithm is used to identify common subgraph structures that appear more than a preset threshold from the task graph set as high-frequency workflows;

[0015] For each high-frequency workflow, based on the execution records of the high-frequency workflow in multiple historical task instances, calculate the stability coefficient related to the success rate and performance indicators of the high-frequency workflow, and determine the high-frequency workflows whose stability coefficients meet the preset conditions as candidate stable workflows.

[0016] The candidate stable workflow, along with the inter-component dependencies, data flow conventions, and parameter passing rules contained within it, are encapsulated into a structured scenario workflow template.

[0017] In one embodiment, the method further includes:

[0018] Automatically generate business scenario description text for the encapsulated scenario workflow template;

[0019] The scenario workflow templates and their corresponding business scenario descriptions are injected into the knowledge graph of the scenario workflow template library; the knowledge graph is used to record the functional similarity, composability, and upstream and downstream relationships between scenario workflow templates.

[0020] In one embodiment, the multi-layered editable structure includes:

[0021] A canvas layer, comprising a canvas, which serves as the carrier for the overall arrangement and display of the workflow;

[0022] The logical parent node layer includes at least one parent node. Each parent node represents a complete scene workflow template on the canvas, which encapsulates a defined set of input ports, a set of output ports, and internal sub-workflow logic.

[0023] The atomic child node layer consists of multiple atomic child nodes connected by internal data edges within each parent node. Each atomic child node corresponds to an execution operator that is parameterized and encapsulated from the underlying capability interface of the big data platform.

[0024] The scenario workflow template pre-sets the connection attributes and parameter passing logic of the internal data edges at the atomic child node layer, and hides these attributes from external users at the logical parent node layer, exposing only configurable items based on business semantics.

[0025] In one embodiment, receiving runtime environment parameters input by the user for the multi-layered editable structure includes:

[0026] Using the parent node as the basic configuration unit, a unified parameter configuration panel is provided to the user;

[0027] In the parameter configuration panel, only configuration items related to business logic are exposed, while the underlying technical connections and parameter transmission details of the atomic sub-node layer are hidden.

[0028] In one embodiment, the method further includes:

[0029] When a user connects the output port of the first parent node to the input port of the second parent node in the canvas, the output data pattern declared by the output port of the first parent node is parsed, and the input data pattern declared by the input port of the second parent node is parsed.

[0030] If the output data pattern is compared with the input data pattern, and there are incompatibilities in field names, data types, or constraints, at least one field mapping scheme is generated based on the semantic similarity of the field names and type conversion rules, and provided to the user for confirmation or automatically executed for data conversion.

[0031] In one embodiment, receiving runtime environment parameters input by the user for the multi-layered editable structure includes:

[0032] Based on the historical deployment records of the target template, the resource usage habits of the user's organization, and / or the real-time resource load of the target deployment environment, at least one set of technical resource parameters is determined as recommended values;

[0033] The recommended values ​​are displayed to the user in a visual development interface.

[0034] In one embodiment, based on the historical deployment records of the target template, the resource usage habits of the user's organization, and / or the real-time resource load of the target deployment environment, at least one set of technical resource parameters is determined as recommended values, including:

[0035] Based on the historical deployment records of the target template, the resource usage habits of the user's organization, and / or the real-time resource load of the target deployment environment, at least one set of technical resource parameters is predicted as recommended values ​​using a parameter recommendation model. The parameter recommendation model is a regression model trained based on machine learning or a rule-based empirical model. The input features of the parameter recommendation model include at least: task data volume estimation, template complexity index, historical resource consumption pattern, and current resource pool idle rate.

[0036] In one embodiment, based on the received runtime environment parameters and the component dependencies, data flow conventions, and parameter passing rules within the target template, an executable target big data application workflow is generated and saved, including:

[0037] Traverse the internal structure of the target template to parse out the dependency topology between multiple processing components;

[0038] Based on the default parameters preset in the target template, the runtime environment parameters input by the user, and the recommended computing resource parameters, the final runtime configuration of each processing component is generated.

[0039] Based on the dependency topology between the multiple processing components and the final runtime configuration of each processing component, the multiple processing components are merged into a complete workflow. The merged complete workflow definition is serialized into a deployment artifact, and a version snapshot containing a hash identifier, timestamp, and creation context is generated for the deployment artifact.

[0040] In one embodiment, the method further includes:

[0041] During the execution of the target big data application workflow, runtime telemetry data is collected during the execution process;

[0042] The runtime telemetry data is analyzed and compared with the preset operating parameters of the target template. When a continuous performance deviation or resource utilization imbalance is detected, parameter optimization suggestions are generated for the target template to trigger version iteration of the target template.

[0043] In one embodiment, after generating and saving the executable target big data application workflow, the method further includes:

[0044] The generated target big data application workflow is automatically submitted to the scheduling system of the big data platform;

[0045] In an isolated testing environment, the target big data application workflow is automated for testing using historical data samples or synthetic data;

[0046] If the automated test passes, the target big data application workflow is published to the production scheduling queue of the scheduling system; otherwise, an alarm is triggered and the system is rolled back to the configuration stage.

[0047] This specification also provides an embodiment of a big data application scenario development device, including:

[0048] The template generation module is used to analyze historical task execution data within the big data platform to identify, verify, and encapsulate high-frequency and stable scenario workflows, forming a reusable scenario workflow template library. Each scenario workflow template in the scenario workflow template library encapsulates multiple big data processing components as well as the dependencies, data flow conventions, and parameter passing rules between these components.

[0049] The template loading module is used to provide users with a visual development interface containing the scenario workflow template library; in response to the user's selection operation of the target template in the scenario workflow template library, the module loads the multi-layer editable structure corresponding to the target template in the visual development interface;

[0050] The parameter receiving module is used to receive runtime environment parameters input by the user for the multi-layer editable structure; the runtime environment parameters include at least one of the following: workflow identifier, access operation permission, data endpoint, execution scheduling strategy, and resource declaration parameters;

[0051] The workflow generation module is used to generate and save an executable target big data application workflow based on the received runtime environment parameters and the component dependencies, data flow conventions and parameter passing rules in the target template.

[0052] This specification also provides a computer device, including a processor and a memory for storing processor-executable instructions, wherein the processor executes the instructions to implement the steps of the big data application scenario development method described in any of the above embodiments.

[0053] This specification also provides a computer-readable storage medium storing computer instructions that, when executed by a processor, implement the steps of the big data application scenario-based development method described in any of the above embodiments.

[0054] This specification also provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements the steps of the big data application scenario-based development method described in any of the above embodiments.

[0055] This specification provides a method for scenario-based development of big data applications. By automatically mining and encapsulating high-frequency, stable scenario workflows from massive historical task data, a reusable template library is formed. This transforms big data application development from the traditional model of starting from scratch and dragging and connecting components one by one to a model of selecting scenario templates and filling in key parameters. Users do not need to worry about how multiple underlying processing components are connected; they only need to perform lightweight configurations oriented towards business needs to generate a complete application. This significantly shortens the development cycle from requirements to finished product, and is especially suitable for scenarios that require rapid construction or batch replication of similar data products. Through a multi-layered editable structure and highly structured runtime environment parameter configuration items, complex underlying technical parameters (such as component connection details, serialization formats, and performance tuning parameters) are completely decoupled from business logic configuration. This allows business analysts or product managers without a deep background in big data technology to independently complete the configuration and deployment of complex big data applications, significantly reducing the technical barriers and human resource costs of big data application development. Moreover, the scenario workflow templates provided in this solution are essentially carriers of validated standardized processes. All applications generated based on the same template maintain a high degree of consistency in their internal processing logic, data flow, and core rules. This fundamentally solves the problems of inconsistent data definitions and chaotic processing logic caused by differences in manual development in multi-person, multi-team collaborations, effectively promoting the standardization of enterprise data processing and improving governance. Since the templates are derived from the analysis and verification of historical successful tasks, their built-in component dependencies and parameter passing rules have been tested in practice, possessing high inherent stability and performance baselines. Compared to workflows built from scratch in existing technologies, template-based big data applications have a higher success rate and predictable performance on their first run. Furthermore, when a common process needs optimization or repair, only the corresponding scenario template needs to be updated, which updates all applications created based on that template, simplifying the complexity of later maintenance and batch upgrades. In summary, the above solution systematically solves the core pain points of low-code development models—inefficiency, high barriers to entry, and insufficient standardization—through intelligent template generation, structured business configuration, and automated application assembly, providing effective technical support for the large-scale development of big data applications. Attached Figure Description

[0056] The accompanying drawings, which are included to provide a further understanding of this specification and form part of it, do not constitute a limitation thereof. In the drawings:

[0057] Figure 1 A flowchart of a big data application scenario-based development method in one embodiment of this specification is shown;

[0058] Figure 2 A schematic diagram of a canvas in one embodiment of this specification is shown;

[0059] Figure 3 A schematic diagram of a big data application scenario development device according to one embodiment of this specification is shown;

[0060] Figure 4 A schematic diagram of a computer device according to one embodiment of this specification is shown. Detailed Implementation

[0061] The principles and spirit of this specification will now be described with reference to several exemplary embodiments. It should be understood that these embodiments are given merely to enable those skilled in the art to better understand and implement this specification, and are not intended to limit the scope of this specification in any way. Rather, these embodiments are provided to make this disclosure more thorough and complete, and to fully convey the scope of this disclosure to those skilled in the art.

[0062] Those skilled in the art will recognize that the embodiments described in this specification can be implemented as a system, apparatus, method, or computer program product. Therefore, the disclosure of this specification can be specifically implemented in the following forms: entirely hardware, entirely software (including firmware, resident software, microcode, etc.), or a combination of hardware and software.

[0063] It should be noted that the information and data related to users involved in the embodiments of this specification are all information and data authorized by the user or fully authorized by the relevant parties. Furthermore, the collection, storage, use, processing, transmission, provision, disclosure, and application of the relevant data all comply with relevant laws, regulations, and standards, and necessary confidentiality measures have been taken. They do not violate public order and good morals, and corresponding operation entry points are provided for users or relevant parties to choose to authorize or refuse.

[0064] It should also be noted that in the embodiments of this specification, certain software, components, models and other existing solutions in the industry may be mentioned. These should be regarded as exemplary and are only intended to illustrate the feasibility of implementing the technical solution of this application. However, they do not mean that the applicant has used or necessarily used the solution.

[0065] This specification provides an example of a big data application scenario-based development method. Figure 1A flowchart of a big data application scenario-based development method according to one embodiment of this specification is shown. Although this specification provides method operation steps or apparatus structures as shown in the following embodiments or figures, more or fewer operation steps or module units may be included in the method or apparatus based on conventional or non-inventive effort. In steps or structures where there is no logically necessary causal relationship, the execution order of these steps or the module structure of the apparatus is not limited to the execution order or module structure described in the embodiments and figures of this specification. When the big data application scenario-based development method or module structure is applied in actual devices or terminal products, it can be executed sequentially or in parallel according to the method or module structure shown in the embodiments or figures (e.g., in a parallel processor or multi-threaded processing environment, or even a distributed processing environment).

[0066] Specifically, such as Figure 1 As shown, one embodiment of this specification provides a big data application scenario-based development method that may include the following steps.

[0067] Step S101: Analyze the historical task execution data within the big data platform to identify, verify, and encapsulate high-frequency and stable scenario workflows to form a reusable scenario workflow template library; each scenario workflow template in the scenario workflow template library encapsulates multiple big data processing components as well as the dependencies, data flow conventions, and parameter passing rules between the multiple big data processing components.

[0068] The method in this embodiment can be applied to big data platforms or servers connected to big data platforms. The component log analysis engine of the big data platform can be used to collect data related to historically executed tasks, including the big data processing components used in the task, the order of calls between components, the execution flow chain, and successful execution records. Using graph algorithms and frequent subgraph mining techniques, the collected historical task data is analyzed to identify repeatedly used component chains (i.e., high-frequency workflows, such as those used more times than a preset number of times within a preset time period) from a large number of task flows. This further verifies the stability of these high-frequency workflows (e.g., stable and efficient execution flow, no frequent errors, and no redundancy), confirming their suitability for general data application scenarios. The verified high-frequency stable workflows are then encapsulated into scenario workflow templates. Each template must clearly include: multiple big data processing components (e.g., data acquisition components, data integration components, storage components, etc.), dependencies between components (e.g., acquisition before integration), data flow conventions (e.g., data flows from the acquisition component to the storage component), and parameter passing rules (e.g., mapping relationships between data fields between components). All encapsulated templates are then compiled to form a reusable scenario workflow template library.

[0069] Step S102: Provide the user with a visual development interface containing the scenario workflow template library; in response to the user's selection operation of the target template in the scenario workflow template library, load the multi-layer editable structure corresponding to the target template in the visual development interface.

[0070] Set up a dedicated scenario-based development entry point on the big data platform. This entry point should be prominently marked with a rapid development mode and guidance (e.g., scenario-based development, one step ahead). This entry point corresponds to a visual development interface containing a scenario workflow template library. The visual development interface displays all scenario workflow templates in the library, each template accompanied by basic information such as applicable scenario descriptions and minimum resource requirements, facilitating user selection.

[0071] When a user selects a target scenario workflow template (e.g., batch data collection and delivery MPP) in the visual interface, the selection action will load the corresponding multi-layered editable structure in the interface. This structure uses a canvas as its core carrier and includes parent nodes (the smallest unit for executing the scenario workflow, such as the parent node for batch data collection and delivery) and child nodes (the components that perform specific operations, such as the unified collection child node and the object storage child node). The parent and child nodes are displayed in the canvas according to preset dependencies, forming a visual hierarchical editable structure.

[0072] Step S103: Receive runtime environment parameters input by the user for the multi-layer editable structure; the runtime environment parameters include at least one of the following: workflow identifier, access operation permission, data endpoint, execution scheduling strategy, and resource declaration parameters.

[0073] The visual development interface uses parent nodes in a loaded multi-layered structure as the basic configuration unit, providing a unified parameter configuration panel. The panel only displays runtime environment parameters related to business logic, without exposing underlying technical parameters. Users input or select corresponding runtime environment parameters in the configuration panel according to their business needs, including but not limited to: workflow identifiers (such as workflow name and description), access operation permissions (such as data access authorization scope), data endpoints (such as data source address and data target address), execution scheduling strategies (such as execution frequency and execution time window), and resource declaration parameters (such as available resource groups). These user-inputted parameters can be received in real time, serving as the core basis for subsequent workflow generation.

[0074] Step S104: Based on the received runtime environment parameters and the component dependencies, data flow conventions, and parameter passing rules within the target template, generate and save an executable target big data application workflow.

[0075] It can read the preset component dependencies, data flow conventions, and parameter passing rules from the target template, and automatically complete the logical connection and parameter matching between components based on the user-input runtime environment parameters. Users do not need to manually configure the dependencies and data passing details between components; it integrates according to template rules and user parameters to form a complete, directly executable big data application workflow. The generated target big data application workflow can be saved. The saved content includes: workflow component connection information, user-configured runtime environment parameters, template association information, and workflow version information (such as creation timestamp), forming a standardized deployment artifact, providing a foundation for subsequent workflow operation, monitoring, and version management.

[0076] In the above embodiments, by automatically mining and encapsulating high-frequency and stable scenario workflows from massive historical task data, a reusable template library is formed. This transforms big data application development from the traditional model of starting from scratch and dragging and connecting components one by one to a model of selecting scenario templates and filling in key parameters. Users do not need to worry about how multiple underlying processing components are connected; they only need to perform lightweight configurations oriented towards their business needs to generate a complete application. This significantly shortens the development cycle from requirements to finished product, and is especially suitable for scenarios that require rapid construction or batch replication of similar data products. Through a multi-layered editable structure and highly structured runtime environment parameter configuration items, complex underlying technical parameters (such as component connection details, serialization formats, and performance tuning parameters) are completely decoupled from business logic configuration. This allows business analysts or product managers without a deep background in big data technology to independently complete the configuration and deployment of complex big data applications, significantly reducing the technical barriers and human resource costs of big data application development. Moreover, the scenario workflow templates provided in this solution are essentially carriers of validated standardized processes. All applications generated based on the same template maintain a high degree of consistency in their internal processing logic, data flow, and core rules. This fundamentally solves the problems of inconsistent data definitions and chaotic processing logic caused by differences in manual development in multi-person, multi-team collaborations, effectively promoting the standardization of enterprise data processing and improving governance. Because the templates are derived from the analysis and verification of historical successful tasks, their built-in component dependencies and parameter passing rules have been tested in practice, possessing high inherent stability and performance baselines. Compared to workflows built from scratch in existing technologies, template-based big data applications exhibit higher success rates and predictable performance on their first run.

[0077] In some embodiments of this specification, historical task execution data within a big data platform is analyzed to identify, verify, and encapsulate high-frequency and stable scenario workflows. This includes: collecting the execution directed acyclic graph, component configuration snapshots, and key performance indicators of successfully executed historical task instances within the big data platform; abstracting each historical task instance into a task graph set with big data processing components as nodes and data flows as edges; using a frequent subgraph mining algorithm to identify common subgraph structures in the task graph set whose frequency exceeds a preset threshold as high-frequency workflows; for each high-frequency workflow, calculating the success rate and stability coefficient related to performance indicators based on the execution records of the high-frequency workflow in multiple historical task instances, and determining high-frequency workflows whose stability coefficients meet preset conditions as candidate stable workflows; and encapsulating the candidate stable workflows and the inter-component dependencies, data flow conventions, and parameter passing rules contained within the candidate stable workflows into a structured scenario workflow template.

[0078] In this embodiment, the component log analysis engine built into the big data platform can be used to collect relevant data from historically executed task instances within the platform, ensuring the authenticity and validity of the data source. The historical task instance data can include the execution directed acyclic graph (DAG) of the historical task instance, component configuration snapshots, and key performance indicators. The execution DAG records the complete execution flow of the task instance, including the sequence and dependencies of each processing stage (e.g., the execution chain of data collection → data storage → data integration), clearly presenting the task's logical architecture. The component configuration snapshot captures detailed configuration information for each big data processing component in the task, including static configuration data such as component type (e.g., Kafka, Flink, MPP), business parameters (data table name, running frequency), and technical parameters (resource allocation, connection address). Key performance indicators collect dynamic runtime data during task execution, including execution time, success rate, resource utilization (CPU, memory usage), data throughput, and error frequency, reflecting the task's operational efficiency and stability.

[0079] Each historical task instance collected can be structurally abstracted to form a standardized task graph. Each big data processing component in the task (such as batch data acquisition components, data integration components, file service components, etc.) can be abstracted as a task graph node, and the node type (computation type, data type) and core function description can be labeled. The data flow relationship between components (such as the output data of the acquisition component flowing into the storage component, and the data of the storage component flowing into the computation component) can be abstracted as edges of the task graph, with the direction of the edges consistent with the direction of the data flow. All abstracted task graphs are summarized into a task graph set, covering various big data application scenarios that have been implemented within the platform (such as batch data processing, streaming data computation, data subscription, etc.).

[0080] Frequent subgraph mining algorithms (e.g., FP-Growth graph mining variants, Gspan algorithm, etc.) can be used for in-depth analysis of task graph sets. The task graph set can be traversed to identify the recurring component-dataflow structures (i.e., common subgraphs) across different task graphs. A preset frequency threshold is set (e.g., a common subgraph appears in more than 100 different task instances), and common subgraph structures with a frequency exceeding this threshold are identified as high-frequency workflows. For example, the subgraph structure of unified collection → object storage → data integration → MPP storage appears repeatedly in the regulatory report development tasks of multiple branches, thus being identified as a high-frequency workflow. Afterwards, stability verification can be performed on each identified high-frequency workflow to ensure its reusability. The execution records of this high-frequency workflow in all historical task instances can be statistically analyzed, and the ratio of successful executions to total executions can be calculated to reflect the execution success rate. A stability coefficient can be constructed based on the fluctuations in key performance indicators (such as the amplitude of execution time fluctuations and the range of resource utilization fluctuations). For example, the stability coefficient = 1 - (performance indicator standard deviation / performance indicator mean) . The closer the coefficient is to 1, the more stable the performance. In one embodiment, a combination of a success rate threshold and a stability coefficient threshold can be used to filter high-frequency workflows. Only high-quality patterns that simultaneously meet the criteria of high frequency, high success rate, and high stability will be identified as candidate stable workflows. This ensures that applications generated from the template library possess high reliability and predictable performance.

[0081] Candidate stable workflows are standardized and encapsulated to generate directly reusable scenario workflow templates. The internal component types, quantities, and precise connection relationships (dependencies and data flow) can be preserved. Core elements are encapsulated, including three key rules within the template: inter-component dependencies (e.g., data collection must be completed before data integration), data flow conventions (e.g., Kafka component output data is automatically used as Flink component input), and parameter passing rules (e.g., collection task time parameters are automatically synchronized to subsequent computation tasks). Configuration parameters of this pattern across all historical instances are analyzed. Fixed, common technical parameters (e.g., checkpoint interval, serializer type) are set as default values ​​for the template; variable, business-related parameters (e.g., source table name, filter date) are extracted and defined as configurable input interfaces for the template. In one embodiment, default technical parameters can be preset in the template (e.g., automatically setting Flink checkpoints, masking the physical location of COS storage) to reduce subsequent configuration workload for users. All of the above information is encapsulated into a complete scenario workflow template object that can be parsed and invoked by the system and stored in a centralized template library.

[0082] The above embodiments replace the traditional method of manually sorting out scenarios and manually packaging templates. Through log analysis and algorithmic mining, high-frequency and stable workflows are automatically identified and verified, reducing the workload and subjective errors of manual intervention. For example, instead of requiring technicians to analyze thousands of historical tasks one by one, processes with high repetition rates and strong stability can be automatically identified, significantly improving the efficiency of scenario template construction. Through dual verification of high-frequency screening and stability validation, the packaged templates are ensured to have high reusability and high stability.

[0083] In some embodiments of this specification, the method may further include: automatically generating business scenario description text for the encapsulated scenario workflow template; injecting the scenario workflow template and the corresponding business scenario description into the knowledge graph of the scenario workflow template library; the knowledge graph is used to record the functional similarity, composability and upstream and downstream relationships between scenario workflow templates.

[0084] In this embodiment, standardized and easy-to-understand business scenario description text can be automatically generated based on the core metadata of the scenario workflow template through structured rules and business semantic mapping, without the need for manual writing. In one embodiment, key information can be automatically extracted from the encapsulated scenario workflow template as the core basis for the description text. For example, the core basis for the description text may include the core functions of the template, such as batch data collection, storage in an MPP database, and integrated batch and stream data computation and service publishing. Another example is that the core basis for the description text may include the core component chain, such as unified collection → object storage → data integration → MPP table, or Kafka caching → Flink computation → file data service. Yet another example is that the core basis for the description text may include applicable scenario characteristics, such as branch-level data reporting, regulatory report development, and unstructured file subscription. Still another example is that the core basis for the description text may include key configuration parameters, such as the requirement to configure the institution number, collection table name, and support for custom running frequency. In one embodiment, a description text framework adapted to general big data scenarios is preset. The description text framework may include scenario name, applicable scenarios, core processes, and configuration points. The scenario name clarifies the core purpose of the template (e.g., batch data collection and delivery MPP template). Applicable scenarios can be described in conjunction with industry / business scenarios (e.g., applicable to the entire process of data collection, storage, and integration for regulatory reports of banking branches). The core process can be simplified by describing the component links and data flow (e.g., raw data is obtained through a unified collection component, temporarily stored in object storage, processed through data integration, and finally written to an MPP table for querying and analysis). Configuration points highlight key parameters that users need to focus on (e.g., data source address, branch number, and data write frequency need to be configured; technical details such as physical storage location and parameter passing between components are not required).

[0085] The extracted template metadata can be incorporated into a structured framework and semantically integrated and optimized using Natural Language Processing (NLP) technology to ensure concise and accurate descriptive text. For example, for the "Data Lake Entry and Archiving" template, the following description is automatically generated: "Data Lake Entry and Archiving Workflow Template, suitable for non-high-frequency data processing scenarios requiring long-term storage; the core process is data lake entry component collection and integration → archiving component storage according to strategy; key configuration points include filling in the data lake address, archiving storage path, and retention period, automatically completing the connection between data lake entry and archiving, shielding the underlying technical configuration."

[0086] The encapsulated scenario workflow templates and automatically generated business scenario description texts can be injected into the knowledge graph of the scenario workflow template library to build a network of relationships between templates. The core nodes of the knowledge graph are the scenario workflow templates, and auxiliary nodes include business scenario types (such as regulatory reports, data migration) and core components (such as Kafka, MPP, and data integration components). Scenario workflow template nodes can contain attributes such as template ID, business scenario description text, a list of core components, applicable industries, key parameters, and creation time. The relationship types in the knowledge graph can be automatically calculated based on template metadata and business logic. Relationship types can include functional similarity relationships, composability relationships, and upstream / downstream relationships. Functional similarity relationships are used to associate templates with a core functional overlap greater than or equal to a preset overlap (e.g., 60%) (e.g., batch data collection and delivery MPP and batch data collection and delivery ClickHouse). Composability relationships are used to associate templates whose upstream and downstream processes are complementary and can be combined to form a complete solution (e.g., batch data collection and delivery MPP and batch data subscription). Upstream and downstream relationships are used to link templates with data flow dependencies (e.g., data ingestion templates and data archiving templates). Each scenario workflow template can be treated as an independent node and entered into the knowledge graph, while simultaneously being linked to its automatically generated business scenario description text and core metadata (such as applicable industries and core components) to ensure the completeness of node information. When a new template is encapsulated, an old template is iterated, or template relationships change, the nodes and relationships in the knowledge graph are automatically updated to ensure the accuracy of the associated information.

[0087] In the above embodiments, the automatically generated business scenario description text transforms the technical template structure into business language, allowing users to quickly determine whether a template suits their needs without needing to deeply understand the internal component logic. The structured description framework ensures consistent explanatory logic across all templates, eliminating the need for users to adapt to different document styles and further lowering the barrier to template use. The knowledge graph records functional similarities, composability, and upstream / downstream relationships, breaking down template isolation and helping users quickly discover potential reuse and combination solutions. The knowledge graph provides a structured management method for the template library, allowing administrators to intuitively view template distribution, association popularity (e.g., which templates are frequently combined), and functional coverage gaps (e.g., a lack of corresponding templates for a certain type of business scenario), providing data support for template library expansion and iteration. The automatically generated description text avoids repetitive work and errors associated with manual document writing. When templates iterate (e.g., component updates, parameter adjustments), the description text is automatically updated synchronously, ensuring consistency between documents and templates and reducing management costs. The associated data in the knowledge graph supports the platform's intelligent recommendation function. Based on the user's currently selected template, it can automatically recommend composable downstream templates (such as recommending a data entry template after selecting a data entry template), alternative templates with similar functions, or recommend suitable templates based on the user's industry (such as banking), making the development process more guided and further reducing the operational difficulty of low-code development.

[0088] In some embodiments of this specification, the multi-layered editable structure may include: a canvas layer, which may include a canvas and serve as the carrier for the overall orchestration and display of the workflow; a logical parent node layer, which may include at least one parent node, each parent node representing a complete scene workflow template on the canvas, which internally encapsulates a defined set of input ports, a set of output ports, and internal sub-workflow logic; and an atomic child node layer, where each parent node is composed of multiple atomic child nodes connected by internal data edges, and each atomic child node corresponds to an execution operator that parameterizes and encapsulates the underlying capability interface of the big data platform; wherein, the scene workflow template pre-sets the connection attributes and parameter passing logic of the internal data edges in the atomic child node layer, and hides these attributes from external users in the logical parent node layer, exposing only configurable items based on business semantics.

[0089] In this embodiment, the multi-layered editable structure achieves separation of technical logic and business configuration through hierarchical division of canvas layer, logical parent node layer, and atomic child node layer. In the visual development interface, the canvas layer is presented as a visual drag-and-drop canvas, providing users with an intuitive graphical operation space. The canvas size adapts to the interface and supports basic operations such as zooming and panning, facilitating users' viewing and orchestration of complex workflows. The canvas layer clearly presents the loaded logical parent nodes and the connections between parent nodes (such as upstream and downstream data flow), forming a visual overall workflow architecture. Users can view the orchestration effect in real time. The canvas supports users to drag and drop logical parent nodes to add, adjust their positions, connect, or disconnect them. Multiple parent nodes can be combined and orchestrated without writing code or configuring complex dependency scripts (e.g., dragging and connecting batch data collection and delivery and batch data subscription parent nodes to the canvas to form a complete process). Different colors and icons are used in the canvas to indicate the configuration status of parent nodes (e.g., not configured, configured, connectable), helping users quickly locate operation nodes.

[0090] The logical parent node layer comprises encapsulated units of business scenarios and technical logic. Each logical parent node in the layer corresponds to a complete scenario workflow template, displayed as an independent functional module on the canvas (e.g., a rectangle labeling the node name for batch data collection and delivery MPP). Core business attributes (applicable scenarios, input / output types) are clearly labeled on the nodes for easy user identification and selection. The parent node encapsulates a set of input ports, specifying the source type of the data received (e.g., data source address, data table name, authorization information, etc.). It also encapsulates a set of output ports, specifying the target type of the output data (e.g., MPP table storage address, file service interface, etc.). Furthermore, the parent node encapsulates internal sub-workflow logic, encapsulating the combination logic of atomic child nodes (e.g., the execution order of collection → storage → integration), without requiring users to know the specific internal components and flow details. The parent node layer only exposes configurable items based on business semantics to the user, completely hiding the technical details of the atomic child node layer (e.g., the connection attributes of data edges between components, parameter passing logic). For example, the batch data collection and delivery MPP parent node only exposes the configuration entry points for business parameters such as the data source, target MPP table name, running frequency, and authorization scope, and does not display the parameter mapping relationship between the internal unified collection sub-node and the object storage sub-node.

[0091] The atomic child node layer comprises parameterized encapsulation units of underlying technical capabilities. Each atomic child node is a parameterized encapsulation of the underlying technical capability interface of the big data platform (such as data acquisition interface, Kafka caching interface, Flink computing interface, and MPP storage interface). Essentially, it is a directly executable technical operator, with each child node responsible for only a single technical function (such as data acquisition, caching, computing, or storage). Multiple atomic child nodes within the same parent node are connected by internal data edges according to a pre-defined logic, with the direction of the data edges consistent with the data flow direction (e.g., unified acquisition child node → object storage child node → data integration child node). Parameter passing rules between atomic child nodes are pre-configured in the data edges. For example, the acquisition time range parameter of the unified acquisition child node is automatically synchronized to the data integration child node, and the output data of the Kafka child node is automatically used as the input data of the Flink child node, without requiring manual configuration by the user. The atomic child node layer is the internal core of the parent node. All technical logic (component dependencies, parameter passing, data flow) is encapsulated by the parent node. Users cannot directly edit the connection relationship or technical parameters of the atomic child nodes. They can only indirectly control the execution logic of the atomic child nodes through the business configuration items of the parent node (such as modifying the running frequency of the parent node and automatically synchronizing it to all related atomic child nodes).

[0092] In the above embodiments, a layered design that encapsulates technical logic at the atomic child node layer and exposes business configuration at the logical parent node layer achieves separation of technical details and business configuration. Business users do not need to understand the underlying interfaces, component dependencies, parameter passing rules, and other technical knowledge of the big data platform. They only need to configure the business parameters of the parent node (such as data table name and running frequency) based on their business needs to complete the development of big data applications. This solves the pain point of business users understanding the requirements but not the technology, and meets the requirements of low-code and easy-to-operate scenario-based development. The logical parent node layer encapsulates the complete scenario workflow into ready-to-use functional modules. Users do not need to build a complete process of component-connection-configuration from scratch. They only need to drag and drop to combine parent nodes in the canvas layer to quickly build complex workflows (such as splicing data collection and data subscription parent nodes to form a complete process), which greatly reduces the workload of repetitive development and orchestration. The atomic child node layer pre-configures data edge connection attributes and parameter passing logic, avoiding the tedious operation of manually configuring parameter mapping and dependency relationships between components. This reduces the technical parameter configuration work by about 70% and significantly improves development efficiency. The canvas layer supports free dragging, combining, and connecting of parent nodes, allowing users to flexibly orchestrate workflows according to their own business scenarios (such as data collection-storage-computation-service), thus broadening the adaptability of scenario-based development. The technical logic (component interfaces, data flow rules) of the atomic child node layer is uniformly pre-configured and maintained by the platform, avoiding workflow errors caused by users accidentally modifying technical parameters and ensuring the operational stability of big data applications. Simultaneously, the standardized encapsulation of parent nodes ensures consistency in the logic of similar applications developed by different users, improving the standardization of data applications. These embodiments significantly lower the development threshold, adapt to business usage scenarios, simplify workflow orchestration, improve development efficiency, balance flexibility and stability, and can adapt to multiple scenario requirements.

[0093] In some embodiments of this specification, receiving runtime environment parameters input by the user for the multi-layer editable structure may include: providing the user with a unified parameter configuration panel using the parent node as the basic configuration unit; in the parameter configuration panel, only configuration items related to business logic are exposed, while the underlying technical connections and parameter transmission details of the atomic child node layer are hidden.

[0094] In this embodiment, the logical parent node is used as the smallest unit for parameter configuration. Users do not need to configure atomic child nodes separately; they only need to input business-level parameters centrally around the scenario workflow template corresponding to the parent node. For example, after selecting the batch data collection and delivery MPP workflow template, all configuration operations revolve around the parent node of this workflow template, without needing to pay attention to the technical parameters of its internal atomic child nodes such as unified collection and object storage. In the visual development interface, users can trigger the pop-up of the corresponding unified parameter configuration panel by clicking on the target parent node on the canvas (the panel corresponds one-to-one with the parent node, supporting separate configuration of multiple parent nodes without interference). If the user selects multiple parent nodes at the same time, a hybrid mode of batch configuration of common parameters (such as running frequency and resource group) and individual configuration of private parameters (such as data source address and target table name) can be provided to improve the efficiency of multi-node configuration. In the configuration panel, only parameters directly related to business logic are filtered and exposed, completely hiding the underlying technical details of the atomic sub-node layer, including: hiding underlying technical connection details (such as the physical server address of COS storage, the Broker node configuration of Kafka, and the internal data edge connection attributes between atomic sub-nodes); hiding parameter passing logic (such as field mapping rules between atomic sub-nodes, data format conversion parameters, CheckPoint configuration, and cache expiration time); and hiding platform-level default parameters (such as data transmission protocol, component communication port, and default encoding format).

[0095] In the above embodiments, by hiding the underlying technical details and exposing only the business configuration items, the core pain point of business users understanding the requirements but lacking technical knowledge and being unable to complete complex parameter configurations is completely solved. Users do not need to understand the connection logic of atomic sub-nodes, parameter passing rules, or other technical knowledge. They only need to fill in simple parameters based on their own business scenarios (such as which table to collect data from, how often to run the configuration, and who to authorize to access it) to complete the configuration. This truly achieves a low-code, zero-technical-barrier configuration experience, which aligns with the core goal of scenario-based development.

[0096] In some embodiments of this specification, the method may further include: when a user connects the output port of a first parent node to the input port of a second parent node in the canvas, parsing the output data pattern declared by the output port of the first parent node and parsing the input data pattern declared by the input port of the second parent node; comparing the output data pattern with the input data pattern, and if there is incompatibility in field names, data types, or constraints, generating at least one field mapping scheme based on the semantic similarity of field names and type conversion rules, and providing it to the user for confirmation or automatically performing data conversion.

[0097] In this embodiment, when a user establishes a connection between the output port of the first parent node (e.g., batch data collection and delivery MPP) and the input port of the second parent node (e.g., batch data subscription) through drag-and-drop operations in the visualization canvas, the data pattern parsing process can be automatically triggered without manual initiation by the user. The output data pattern of the first parent node can be parsed: core information declared by the output port is extracted from the parent node's pre-defined metadata, including field names (e.g., customer ID, transaction amount, transaction date), data types (e.g., int, decimal, datetime), constraints (e.g., not null, field length ≤ 50), and data format (e.g., date format yyyy-MM-dd). The input data pattern of the second parent node is parsed similarly: the field names, data types, constraints, and data formats declared by the input port are extracted from the second parent node's pre-defined metadata. All data pattern information comes from the standardized declarations made during parent node encapsulation (consistent with the workflow template encapsulation logic described earlier), ensuring the accuracy and consistency of the parsed data. The two parsed data patterns can be compared dimension-by-dimensionally according to the priority of field name → data type → constraints → data format to generate a compatibility assessment result. Fields with identical names, matching data types, no conflicting constraints, and consistent data formats (e.g., the first parent node outputs customer ID (int, non-empty), and the second parent node inputs customer ID (int, non-empty)) are directly considered to be able to flow normally. Fields with different names but related semantics (e.g., the first parent node outputs user ID, and the second parent node inputs customer ID; outputs transaction amount, and inputs transaction amount (yuan)) are considered incompatible. Data types that do not match (e.g., outputs transaction amount (decimal(10,2)), and inputs transaction amount (int); outputs transaction date (string), and inputs transaction date (datetime)) are considered incompatible. Conflicting constraints (e.g., output fields can be empty, but input fields must be non-empty; output field length ≤ 100, input field length ≤ 50) are considered incompatible. Inconsistent data formats (e.g., output date format yyyyMMdd, input date format yyyy-MM-dd) are considered incompatible. For some incompatible cases, at least one feasible field mapping scheme is automatically generated based on semantic similarity matching and standardized type conversion rules. In one embodiment, semantic matching of field names can be performed. It can associate semantically related fields (e.g., mapping user ID to customer ID) based on a built-in business semantic dictionary (e.g., banking user ID = customer ID transaction number = order number) or a semantic similarity algorithm (e.g., cosine similarity). For missing fields without semantic association, it generates default filling suggestions (e.g., if the second parent node inputs a field requiring a comment but the first parent node has no output, it suggests filling it with no comment by default). In one embodiment, data type or format conversion can be performed.Based on preset standardized conversion rules, type conversion logic can be generated (e.g., converting decimal to int while retaining the integer part, converting string type dates to datetime type, and automatically truncating excessively long fields to the allowed length of the input field). The conversion rules follow industry-standard practices (e.g., no loss of core data, unambiguous format conversion) to avoid data anomalies caused by custom conversions. Each generated scheme clearly labels the field mapping relationship, conversion logic, and impact description. For example, Scheme 1: User ID → Customer ID (semantic matching), Transaction Amount (decimal) → Transaction Amount (int) (retaining the integer part), Transaction Date (string'yyyyMMdd') → Transaction Date (datetime'yyyy-MM-dd') (format conversion). Impact description: The decimal part of the transaction amount is lost; confirmation is recommended. All generated mapping schemes can be displayed to the user via a pop-up window. After the user confirms or selects automatic execution, the conversion logic corresponding to the mapping scheme is automatically embedded into the connection link between the two parent nodes (hidden at the underlying level, invisible to the user), forming a complete flow link: first parent node output → data conversion → second parent node input. The conversion status can be fed back through connection icons on the canvas (e.g., a green icon indicates that data conversion has been configured), and a summary of the conversion logic can be displayed in the auxiliary information panel (e.g., User ID → Customer ID, date format yyyyMMdd → yyyy-MM-dd), making it easy for users to trace. This approach lowers the barrier to entry for cross-node combinations, improves workflow orchestration flexibility, reduces manual configuration errors, enhances data flow accuracy, increases development efficiency, and shortens the workflow build cycle.

[0098] In some embodiments of this specification, receiving runtime environment parameters input by the user for the multi-layer editable structure may include: determining at least one set of technical resource parameters as recommended values ​​based on the historical deployment records of the target template, the resource usage habits of the user's organization, and / or the real-time resource load of the target deployment environment; and displaying the recommended values ​​to the user in a visual development interface.

[0099] In this embodiment, recommended values ​​for technical resource parameters are calculated by integrating multi-dimensional data, reducing the difficulty of user parameter configuration and improving configuration rationality. Multi-dimensional data may include: historical deployment records of the target template, resource usage habits of the user's organization, and / or the target deployment environment. Historical deployment records of the target template may include all successful deployment cases of the target scenario workflow template within a preset time period (filtering out failed cases to avoid the influence of abnormal data). Specific data content collected may include technical resource parameter configurations (such as the number of DCUs, memory allocation size, maximum concurrency, and data processing queue priority), core indicators of the corresponding business scenario (data processing volume, execution frequency, task complexity), and post-deployment operational performance data (resource utilization, execution time, success rate, and mean time between failures). The collected data can be preprocessed. For example, extreme values ​​(such as cases where abnormally high / low configurations result in resource utilization of <30% or >95%) can be removed, retaining valid samples with reasonable configurations and stable operation. The resource usage habits of a user's organization can refer to the deployment configuration data of similar scenario templates (such as batch data processing templates with ≥70% functional similarity to the target template) within a recently preset time period (within the last 3 months) of the user's organization (e.g., a branch of a bank, a data department of an enterprise). Specific data collection can include commonly used technical resource parameter ranges within the organization (e.g., when deploying batch data templates, the branch typically uses 4-6 DCUs and 8-16GB of memory), and organizational-level resource configuration specifications (e.g., maximum concurrency should not exceed 10, and the default queue priority is business priority 2). The collected data can be preprocessed. For example, the frequency distribution of parameter configurations can be statistically analyzed, and frequently used parameter ranges that conform to organizational specifications can be extracted as a reference benchmark for recommended values. The target deployment environment can include the real-time operating status of the user-selected target deployment environment (e.g., big data platform resource pool A, dedicated resource group B). Specifically, data such as the overall load indicators of the resource pool (CPU idle rate, memory idle rate, remaining storage space, network bandwidth utilization), the resource usage of currently running tasks, and the congestion level of the resource scheduling queue (e.g., the number of tasks waiting to be executed) can be collected. The collected data can be preprocessed. For example, the data can be updated in real time every 5 minutes to calculate the upper limit of available resources in the resource pool (e.g., CPU idle rate 60% → available CPU resources are 60% of the total resources), ensuring that the recommended values ​​do not exceed the carrying capacity of the current environment. Based on the collected data, at least one set of optimal technical resource parameter recommendation values ​​can be generated through preset calculation logic or a pre-trained parameter recommendation model. In one embodiment, the preset calculation logic may include at least one of the following: weight allocation, interval filtering, and rationality verification.In one embodiment, the parameter recommendation model can be a regression model trained based on machine learning or a rule-based empirical model. Its input features include at least: estimated task data volume, template complexity index, historical resource consumption patterns, and current resource pool idle rate. Recommended values ​​can be displayed in a user-friendly and interactive manner within the parameter configuration panel of the visual development interface, reducing user decision-making costs.

[0100] In the above embodiments, considering that configuring technical resource parameters (such as the number of DCUs and memory allocation) requires professional knowledge (such as understanding the matching relationship between data processing volume and resources), business users without technical backgrounds often find it difficult to configure them accurately. Therefore, by automatically recommending the optimal value, users do not need to understand the meaning of the parameters and can directly use the recommended value or make a simple selection. This solves the problem that business users understand the requirements but do not understand the configuration of technical parameters, and further improves development efficiency and resource utilization.

[0101] In some embodiments of this specification, at least one set of technical resource parameters is determined as recommended values ​​based on the historical deployment records of the target template, the resource usage habits of the user's organization, and / or the real-time resource load of the target deployment environment. This may include: predicting at least one set of technical resource parameters as recommended values ​​using a parameter recommendation model based on the historical deployment records of the target template, the resource usage habits of the user's organization, and / or the real-time resource load of the target deployment environment. The parameter recommendation model is a regression model trained based on machine learning or a rule-based empirical model. The input features of the parameter recommendation model may include at least: estimated task data volume, template complexity index, historical resource consumption pattern, and current resource pool idle rate. Compared to statistical recommendations based on single data, the model in this embodiment integrates multiple core features and combines the pattern recognition capabilities of machine learning or the precise matching of a rule engine, effectively reducing the error rate between the recommended value and the actual optimal configuration. Users do not need to repeatedly debug parameters, significantly reducing problems such as task timeouts and resource waste caused by improper configuration.

[0102] In some embodiments of this specification, generating and saving an executable target big data application workflow based on the received runtime environment parameters and the component dependencies, data flow conventions, and parameter passing rules within the target template may include: traversing the internal structure of the target template to parse the dependency topology between multiple processing components; generating the final runtime configuration of each processing component based on the default parameters preset in the target template, the runtime environment parameters input by the user, and the recommended computing resource parameters; merging the multiple processing components into a complete workflow based on the dependency topology between the multiple processing components and the final runtime configuration of each processing component; serializing the merged complete workflow definition into a deployment artifact; and generating a version snapshot for the deployment artifact containing a hash identifier, timestamp, and creation context.

[0103] In this embodiment, the dependency topology of the target template can be parsed. The complete internal structure of the target scenario workflow template is automatically traversed. Based on the component association metadata (such as dependency markers and execution order identifiers) pre-set during the template encapsulation stage, the relationships between big data processing components are identified, ultimately constructing a dependency topology graph with processing components as nodes and dependencies between components as edges. This topology graph fully presents the execution logic of the components, including serial dependencies (e.g., the data integration component can only be triggered after the data acquisition component completes its execution), parallel dependencies (e.g., multiple acquisition components from different data sources can be executed in parallel), and upstream and downstream data flow constraints. This provides a structured logical benchmark for the complete construction of the subsequent workflow. The entire parsing process requires no user intervention and is completed entirely in the background. The final runtime configuration for each processing component can be generated. Configuration generation can be completed by combining various core parameters. One is the default parameters pre-set in the template, such as component communication protocols, basic timeout times, and default data formats, which serve as the technical foundation for the configuration. Secondly, user-input runtime environment parameters, such as workflow identifiers, data endpoint addresses, and execution scheduling strategies, will be precisely mapped to the configuration items of the corresponding processing components according to the parameter passing rules agreed upon in the template (e.g., passing the target MPP table name to the output configuration of the storage component). Thirdly, recommended computing resource parameters, such as the number of DCUs and memory allocation size, will be added to the component's resource configuration module. Simultaneously, all parameters can be validated for legality and compatibility. If parameter conflicts occur (e.g., resource parameters exceeding platform quotas) or format errors occur (e.g., invalid data endpoint addresses), the system will automatically correct or prompt the user for confirmation via a pop-up window, ensuring that the configuration of each component aligns with both business requirements and technical operating specifications. Afterward, the complete workflow is generated, serialized, and a version snapshot is created. Based on the logical order of the component dependency topology, all configured processing components can be connected in series / parallel into a complete executable big data application workflow, ensuring that the data flow and parameter passing between components fully match the rules agreed upon in the template. Subsequently, the definition of this complete workflow (including component configuration, dependency logic, execution rules, etc.) is serialized into a standardized deployment artifact. Typically using the JSON / YAML format common to big data platforms, it can be directly submitted to the platform's scheduling engine for execution. Simultaneously, a version snapshot is automatically generated for the deployment artifact. This snapshot includes a unique hash identifier (used to verify artifact integrity and prevent tampering), a timestamp (recording the generation time), and a creation context (such as the owner, target template version, and deployment environment). The deployment artifact and version snapshot are then associated and stored on the platform, forming a traceable version record.

[0104] In the above embodiments, by parsing the dependency topology, the execution order and dependency logic between components are solidified in a structured manner, avoiding logical confusion that may occur when manually orchestrating workflows (such as triggering subsequent components before the preceding components are executed). This ensures that the workflow strictly follows the stable process preset by the template, significantly reducing the probability of runtime anomalies and improving the success rate of workflow execution. By integrating three types of configurations—template default parameters, user business parameters, and recommended resource parameters—and by using automated verification to avoid issues such as parameter exceeding limits and format errors, the technical stability of the template is preserved while meeting the personalized business needs of users. This ensures that the generated workflow can be deployed directly without additional debugging, reducing configuration trial and error costs. Standardized deployment artifacts are compatible with the general scheduling engine of big data platforms, allowing for cross-environment deployment without additional adaptation. The hash identifier of the version snapshot can verify the integrity of the artifact, while the timestamp and creation context enable full lifecycle traceability of the workflow version, facilitating subsequent version rollback and problem investigation (such as locating the creator of a certain version of the workflow and the template version used), especially suitable for cross-team collaboration and multi-version iteration scenarios.

[0105] In some embodiments of this specification, the method may further include: collecting runtime telemetry data during the execution of the target big data application workflow; analyzing the runtime telemetry data and comparing the runtime telemetry data with the preset operating parameters of the target template; and generating parameter optimization suggestions for the target template when a continuous performance deviation or resource utilization imbalance is detected, so as to trigger the version iteration of the target template.

[0106] In this embodiment, after the target big data application workflow starts execution, a runtime telemetry data acquisition mechanism is activated, seamlessly embedded into the workflow execution chain without requiring additional user configuration. The collected telemetry data covers two types of core metrics. One type is performance data, including the overall workflow execution time, the response time of each processing component, data throughput (e.g., data processed per second), task retries, and reasons for failure. The other type is resource data, covering real-time resource consumption such as CPU utilization, memory usage, storage I / O rate, and network bandwidth utilization. The collection frequency is dynamically adjusted according to the metric type; for example, key performance metrics (e.g., response time, throughput) are collected at the second level, while resource consumption metrics are collected at the minute level. All data is associated with identification information such as workflow ID, target template version, and deployment environment to ensure traceability. After data collection is completed, the telemetry data can be analyzed and compared with benchmarks in real time. First, preset running parameters are extracted from the target template's metadata as comparison benchmarks, including performance benchmarks (e.g., expected execution time, maximum response time threshold of components) and resource benchmarks (e.g., reasonable range of CPU utilization, maximum memory usage limit). Subsequently, a sliding window algorithm is used to analyze the changing trends of telemetry data, focusing on identifying persistent anomalies rather than single, accidental fluctuations. For example, this includes workflows whose execution time exceeds a preset threshold by more than 20% for three consecutive times, or CPU utilization that remains below 30% (resource idleness) or above 90% (resource overload) for more than 10 minutes. The analysis also correlates with historical execution data of similar workflows to eliminate the influence of temporary environmental fluctuations and ensure the accuracy of anomaly detection.

[0107] When a persistent performance deviation or resource utilization imbalance is detected that meets the criteria, targeted parameter optimization suggestions are generated. These suggestions focus on core adjustable parameters of the template. For example, for execution timeouts, suggestions include optimizing the component dependency topology order or increasing memory allocation for critical components. For persistently low CPU utilization, suggestions include reducing the number of DCUs or lowering resource queue priority. For insufficient data throughput, suggestions include adjusting component parallelism or optimizing data sharding strategies. Optimization suggestions are accompanied by supporting data (e.g., if the current average CPU utilization is 25%, lower than the preset reasonable range of 40%-70%, it is recommended to reduce the number of DCUs from 6 to 4) and submitted to the template management backend. After administrator approval, a version iteration of the target template can be triggered, updating the optimized parameters to the template's new default configuration, while retaining the old version for rollback, forming a closed loop of operation monitoring, analysis optimization, and version iteration.

[0108] In the above embodiments, runtime data feedback allows the template to continuously adjust parameters based on actual performance, thereby continuously improving template performance and resource adaptability and maintaining optimal performance over the long term. This reduces resource waste and performance degradation, improving the overall platform efficiency. By monitoring resource utilization imbalances in real time and optimizing parameters accordingly, it effectively avoids resource idleness caused by over-configuration (e.g., excessive DCU configuration with insufficient utilization) or performance bottlenecks caused by under-configuration (e.g., insufficient memory leading to execution lag). This ensures that the workflow corresponding to the template always runs within a balanced range of resources and performance, improving resource utilization. It also reduces manual monitoring and optimization costs, improving template management efficiency.

[0109] In some embodiments of this specification, after generating and saving the executable target big data application workflow, the process may further include: automatically submitting the generated target big data application workflow to the scheduling system of the big data platform; performing automated testing on the target big data application workflow using historical data samples or synthetic data in an isolated test environment; if the automated test passes, publishing the target big data application workflow to the production scheduling queue of the scheduling system; otherwise, triggering an alarm and rolling back to the configuration stage.

[0110] In this embodiment, after the target big data application workflow is generated and saved as a standardized deployment artifact, a closed-loop process of submission-testing-release can be initiated without manual user intervention. First, the serialized workflow deployment artifact is automatically submitted to the big data platform's scheduling system via a pre-defined interface. Upon receiving the artifact, the scheduling system temporarily stores it in a test queue, recording metadata such as submission time, workflow ID, and associated template version for future traceability. During submission, the system automatically verifies the workflow's format validity and the availability of dependent resources. If format errors or unavailable dependent resources are found, a lightweight alert is immediately triggered, and submission is paused. The submission process can be restarted after the user resolves the issue. After submission, the scheduling system distributes the workflow to an isolated test environment. This environment is completely physically isolated from the production environment, possessing independent resource pools, data source copies, and network links, preventing any impact on production operations during the testing process. The testing phase is executed automatically, automatically loading historical data samples matching the workflow scenario (such as raw data from branch supervision reports over the past 3 months) or synthetic data generated according to business rules (simulating boundary scenarios such as extreme data volumes and abnormal fields). Tests are triggered according to the workflow's preset scheduling strategy (such as scheduled execution or one-time execution). Test content covers functional verification (such as data collection integrity, correct data flow between components, and consistency between output results and expectations), performance verification (such as execution time and whether resource utilization meets preset benchmarks), and stability verification (such as no errors after 3 consecutive executions). Logs and execution results are recorded in real time and associated with the workflow ID. After automated testing, branch logic can be executed based on the test results. If all test dimensions meet the preset pass criteria (such as 100% functional verification pass rate, performance indicators not exceeding thresholds, and no stability anomalies), the test is considered passed. The scheduling system automatically migrates the workflow from the test queue to the production scheduling queue and updates the workflow status to pending execution, waiting to start production operation according to the preset scheduling strategy. If the test fails (e.g., incorrect output data, execution timeout, excessive resource usage), an alarm will be triggered immediately, notifying the user via platform notifications, emails, etc. Simultaneously, the workflow will be automatically rolled back to the parameter configuration stage, retaining the previously entered runtime environment parameters. The user can view the test logs to locate the problem (e.g., data conversion failure of a component, insufficient resource configuration), modify the parameters, regenerate the workflow, and trigger a new round of submit-test-release process. This approach significantly reduces the risk of production environment failures, ensures business stability, simplifies the release process, and improves development and delivery efficiency.

[0111] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to interchangeably. Each embodiment focuses on describing the differences from other embodiments. For details, please refer to the foregoing descriptions of the relevant processing embodiments; they will not be repeated here.

[0112] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0113] The above method will be described below with reference to a specific embodiment. However, it is worth noting that this specific embodiment is only for better illustration of this specification and does not constitute an improper limitation of this specification.

[0114] This specific embodiment provides a method for scenario-based development of big data applications. Specifically, based on fundamental visual data application development capabilities, it further designs scenario-based big data solutions and configuration templates, supporting application users to configure and develop low-code applications according to end-to-end application scenarios. The big data platform helps applications generate relevant code, tasks, and artifacts, replacing the original method of repetitive development and extensive configuration for big data applications. The scenario workflow nodes of the big data scenario template support editing operations such as splicing and cropping, and the canvas supports version management, as well as copying and artifact generation operations. This further broadens the applicability of scenarios and improves the development efficiency of end-to-end financial big data applications. For example, based on the basic drag-and-drop batch data collection, data integration, and data subscription components provided by the big data platform, batch data collection and subscription scenario capabilities can be assembled by branch and shared with different branches for direct configuration and use; or based on the already developed data entry (also based on fundamental data development capabilities) and data archiving capabilities, it can be further encapsulated into end-to-end scenario capabilities of entry into the lake and archiving immediately. It can also identify and recommend high-frequency scenario workflows that can be templated based on historical task execution logs. The component log analysis engine monitors and analyzes successful tasks created by users on the platform through drag-and-drop basic components. Utilizing graph algorithms and frequent subgraph mining techniques, it automatically identifies frequently used, stable, and efficient component chains. When the frequency and stability of a particular pattern exceed a threshold, it automatically prompts the administrator to encapsulate it into a new candidate scenario template, generates a scenario description, and recommends it to other users.

[0115] The biggest difference between scenario-based development of big data applications and conventional user-selected big data technology tools for development lies in the fact that scenario-based development involves the abstraction and automated connection of the operational objects and steps of one or more big data application scenarios. For example, a conventional data application development might include four stages: batch data collection, storage, processing, and service. The development process requires four components: data collection, data storage (MPP, etc.), data integration (offline computing), and data service. Each component has corresponding job development and job parameter configuration. After each job is developed, they are connected one by one through a development canvas, specifying their dependencies, thus forming a complete workflow. Scenario-based development, on the other hand, pre-encapsulates these components according to their dependencies in nodes and canvases. Users only need to configure key parameters, such as the name of the data table to be collected, the frequency of data writing, the name of the data service to be published, authorization information, etc., and perform low-code definition of the table structure, while ignoring technical or process parameters, such as data storage location and parameter passing between components.

[0116] Organizing scenario-based development according to scenario workflow mainly includes the following:

[0117] Conduct big data scenario analysis and workflow template creation. This may include the following:

[0118] The key technical capabilities of the big data platform are extracted and encapsulated independently. The encapsulation forms include page call interfaces and OpenAPI call interfaces, such as job configuration interfaces, batch / streaming data collection interfaces, task scheduling submission interfaces, and external data subscription interfaces. The input and output parameters of each interface are clearly defined to form an assemblable list of functions for configuration when called in a specific scenario.

[0119] The workflow of common big data scenarios is sorted out, which may include batch data collection scenarios, batch subscription scenarios, data migration scenarios, and integrated stream and batch data computing and service scenarios. It is also sorted out whether there are upstream and downstream calling relationships between various scenarios. For scenarios with calling relationships, scenario-based connection templates are further constructed.

[0120] The scenario workflow consists of a canvas, node groups, parent nodes, and child nodes, along with corresponding technical solutions, development processes, dependent environments, and usage instructions. Child nodes are the nodes that perform specific operations, including computational nodes and data nodes, such as batch data acquisition nodes and streaming data caching Kafka nodes, selected from the interface or function list in step 1). Parent nodes are the smallest unit for executing the scenario workflow, composed of several child nodes and their connections, such as real-time data integration parent nodes and batch data integration parent nodes. They can independently complete real-time data acquisition and integration, and batch data acquisition and integration functions. Scenario workflow parameter configuration is based on parent nodes. Node groups consist of several parent nodes; for example, a real-time integration node group may contain several real-time integration parent nodes, each acquiring and integrating data from different data sources in real time. The canvas is a container that holds parent and child nodes, as well as the scenario workflow and external relationships. Multiple parent node groups can be dragged into the canvas to connect the complete workflow.

[0121] In the parent-child node configuration, some interface configuration parameters that are typically found in conventional data application development are hidden. For example, in the unified collection child node, the physical location of COS file storage is hidden, and checkpoints are automatically set in the stream computing child node. Furthermore, some parameters are chained together, such as automatically using Kafka output as input data for stream computing. The system automatically detects data schema compatibility between upstream and downstream nodes, providing prompts or generating suggested data transformation logic. Through these methods, the configuration of technical parameters and the configuration of input / output parameters between components are reduced, resulting in approximately 70% reduction in intermediate parameter configuration for application development.

[0122] Conduct big data scenario-based development. Big data scenario-based development is based on the use of big data scenario workflow templates. Before using big data scenario-based development capabilities, applications or users should apply for and activate relevant big data services and resource quotas to support subsequent application development, which is consistent with regular big data application deployment. Regular big data application development involves application users combining actual business scenarios, selecting the required technical components, and performing scripted or drag-and-drop development, configuring the dependencies between tasks. Scenario-based development includes the following steps.

[0123] On the big data platform, users can access scenario-based development through a dedicated entry point. The menu entry point prominently displays the rapid development mode and provides explanations such as "scenario-based development, one step ahead," guiding users to the big data scenario center through this entry point.

[0124] Entering the scenario center displays the supported scenario workflow templates, such as batch data collection and delivery MPP, file subscription MPP, integrated stream and batch solution, data access acceleration, etc., as well as the applicable instructions and minimum resource requirements for each scenario workflow, prompting or helping users check whether the current application environment can carry out scenario-based development.

[0125] Users select the desired scenario workflow template and create the corresponding workflow. Working in units of parent nodes (workflows), users set the necessary runtime environment parameters all at once. Depending on the scenario template, these parameters may include workflow name, description, permissions, data source, data target, table structure definition, execution frequency, and available resource groups. Simultaneously, based on the resource availability of the target deployment environment, the system can automatically recommend or set optimal DCU and other key technical parameters for the tasks within the template.

[0126] After setting the runtime environment parameters, a function to save and generate artifacts is provided to save the current version of the scene workflow settings, specifically including information on the connected nodes and workflow version information. Simultaneously, the scene workflow containing the user-set parameters is displayed on the canvas. An example of a scene-connected canvas is shown below, where batch data collection and delivery and batch data subscription are parent nodes, and the rectangles inside are child nodes. The right-angled rectangles are calculation-type child nodes, the rounded rectangles are data-type child nodes, and the dashed box represents the workflow canvas.

[0127] The platform automatically generates corresponding artifact (workflow) versions based on the workflow settings of the scenario, and subsequent version releases and workflow monitoring are completed based on these artifacts.

[0128] The parent node of the scenario workflow supports splicing and trimming operations. For example... Figure 2 As shown in the canvas, two parent nodes can be combined to form a large scene workflow, jointly completing batch data collection and delivery, and batch data subscription functions. Simultaneously, the workflow canvas supports copying and version recording, facilitating the sharing of creation methods for similar or related scene workflows among different applications or users, as well as switching between different versions.

[0129] Tasks generated through scenario workflows operate and are monitored in the same way as tasks developed based on basic development capabilities.

[0130] The optimization and automated mining of big data scenario-based workflows can include the following:

[0131] After the scenario workflow is deployed and running, the monitoring page can provide optimization prompts based on the runtime resource usage of different organizations and deployment environments, such as suggestions for automated resource expansion.

[0132] Based on the platform's historical task execution logs, high-frequency workflow scenarios that can be templated are identified and recommended. Through a component log analysis engine, successful tasks created by users on the platform via drag-and-drop basic components are monitored and analyzed. Graph analysis algorithms and frequent subgraph mining techniques are used to discover repeatedly used, stable, and efficient component chains. When the frequency and stability of a workflow exceed a threshold, the administrator is automatically prompted to encapsulate it as a new candidate workflow template, generate a scenario description, and recommend it to other users.

[0133] In the above embodiments, the use of big data scenario-based development firstly reduces the difficulty for applications in selecting big data technology components in terms of platform launch. Applications only need to adapt their business processes to the big data workflow template according to the scenario template instructions and apply for services and quotas according to the template recommendations. Secondly, during the development process, the scenario-based workflow template hides the pure technical parameter configuration and realizes the automated transmission of parameters between upstream and downstream sub-nodes. Through low-code development and automated connection techniques, it reduces the workload of process parameter configuration and development by about 70%. The workflow canvas also supports the serial and parallel structures of multiple types of parent nodes, further enriching the applicable scenarios of scenario-based development. Through scenario-based development, the difficulty of selecting technical tools and configuring development for general big data application scenarios is reduced overall, and the compatibility between technical tools and scenarios is improved, thereby increasing resource utilization and service efficiency. It is especially suitable for application users who are not very familiar with big data technical tools but want to quickly carry out the development of universal big data applications. The developed scenario-based workflows can be easily shared with peer organizations through authorized sharing, reducing the R&D workload of these organizations while improving the standardization of overall data application and processing and the consistency of data definitions. The automated mining of scenario workflows further optimizes manual information collection and workflow usage analysis into automated identification and encapsulation suggestions, creating a positive synergy between technology and application with real business needs. This enhances the effectiveness of big data scenario workflow templates and improves the accuracy of template encapsulation.

[0134] Based on the same inventive concept, this specification also provides a big data application scenario-based development device, as described in the following embodiments. Since the principle of the big data application scenario-based development device in solving problems is similar to that of the big data application scenario-based development method, the implementation of the big data application scenario-based development device can refer to the implementation of the big data application scenario-based development method, and repeated details will not be elaborated further. As used below, the terms "unit" or "module" can refer to a combination of software and / or hardware that implements a predetermined function. Although the device described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated. Figure 3This is a structural block diagram of a big data application scenario development device according to an embodiment of this specification, such as... Figure 3 As shown, it includes: template forming module 301, template loading module 302, parameter receiving module 303 and workflow generation module 304. The structure is described below.

[0135] The template forming module 301 is used to analyze historical task execution data within the big data platform to identify, verify, and encapsulate high-frequency and stable scenario workflows, forming a reusable scenario workflow template library. Each scenario workflow template in the scenario workflow template library encapsulates multiple big data processing components as well as the dependencies, data flow conventions, and parameter passing rules between the multiple big data processing components.

[0136] The template loading module 302 is used to provide users with a visual development interface containing the scenario workflow template library; in response to the user's selection operation of the target template in the scenario workflow template library, the module loads the multi-layer editable structure corresponding to the target template in the visual development interface.

[0137] The parameter receiving module 303 is used to receive the operating environment parameters input by the user for the multi-layer editable structure; the operating environment parameters include at least one of the following: workflow identifier, access operation permission, data endpoint, execution scheduling strategy and resource declaration parameters.

[0138] The workflow generation module 304 is used to generate and save an executable target big data application workflow based on the received runtime environment parameters and the component dependencies, data flow conventions and parameter passing rules in the target template.

[0139] This specification also provides a computer device, which can be found in the following description. Figure 4 The diagram shown illustrates the computer device structure based on the big data application scenario-based development method provided in the embodiments of this specification. Specifically, the computer device may include an input device 41, a processor 42, and a memory 43. The memory 43 stores processor-executable instructions. When the processor 42 executes the instructions, it implements the steps of the big data application scenario-based development method described in any of the above embodiments.

[0140] In this embodiment, the input device can specifically be one of the main devices for information exchange between the user and the computer system. The input device may include a keyboard, mouse, camera, scanner, light pen, handwriting input tablet, voice input device, etc.; the input device is used to input raw data and programs for processing this data into the computer. The input device can also receive data transmitted from other modules, units, and devices. The processor can be implemented in any suitable manner. For example, the processor can take the form of a microprocessor or central processing unit, as well as a computer-readable medium storing computer-readable program code (e.g., software or firmware) executable by the (micro)processor, logic gates, switches, application-specific integrated circuits (ASICs), programmable logic controllers, and embedded microcontrollers, etc. The memory can specifically be a memory device used to store information in modern information technology. The memory can include multiple layers; in digital systems, anything that can store binary data can be considered memory; in integrated circuits, a circuit without physical form but with storage function is also called memory, such as RAM, FIFO, etc.; in a system, a storage device with physical form is also called memory, such as a memory stick, TF card, etc.

[0141] In this embodiment, the specific functions and effects implemented by the computer device can be explained in comparison with other embodiments, and will not be repeated here.

[0142] This specification also provides a computer storage medium based on a big data application scenario-based development method. The computer storage medium stores computer program instructions, which, when executed by a processor, implement the steps of the big data application scenario-based development method described in any of the above embodiments.

[0143] In this embodiment, the storage medium includes, but is not limited to, random access memory (RAM), read-only memory (ROM), cache, hard disk drive (HDD), or memory card. The memory can be used to store computer program instructions. The network communication unit can be an interface configured according to standards specified in the communication protocol for network connection communication.

[0144] In this embodiment, the specific functions and effects implemented by the program instructions stored in the computer storage medium can be explained by comparison with other embodiments, and will not be repeated here.

[0145] This specification also provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements the steps of the big data application scenario-based development method described in any of the above embodiments.

[0146] Obviously, those skilled in the art will understand that the modules or steps of the embodiments described above can be implemented using general-purpose computing devices. They can be centralized on a single computing device or distributed across a network of multiple computing devices. Optionally, they can be implemented using computer-executable program code, thereby storing them in a storage device for execution by a computing device. In some cases, the steps shown or described can be performed in a different order than those presented herein, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. Thus, the embodiments of this specification are not limited to any particular combination of hardware and software.

[0147] It should be understood that the above description is for illustrative purposes and not for limitation. Many embodiments and applications beyond the provided examples will be apparent to those skilled in the art upon reading the above description. Therefore, the scope of this specification should not be determined by reference to the above description, but rather by reference to the foregoing claims and the full scope of their equivalents.

[0148] The above description is merely a preferred embodiment of this specification and is not intended to limit this specification. Various modifications and variations can be made to the embodiments described herein by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this specification should be included within the scope of protection of this specification.

Claims

1. A method for developing big data application scenarios, characterized in that, include: Analyze historical task execution data within the big data platform to identify, verify, and encapsulate high-frequency and stable scenario workflows, forming a reusable scenario workflow template library; Each scenario workflow template in the scenario workflow template library encapsulates multiple big data processing components, as well as the dependencies, data flow conventions, and parameter passing rules among these components. Provide users with a visual development interface containing the scenario workflow template library; in response to the user's selection operation of the target template in the scenario workflow template library, load the multi-layer editable structure corresponding to the target template in the visual development interface; The system receives runtime environment parameters input by the user for the multi-layer editable structure; the runtime environment parameters include at least one of the following: workflow identifier, access operation permission, data endpoint, execution scheduling policy, and resource declaration parameters; Based on the received runtime environment parameters, component dependencies, data flow conventions, and parameter passing rules within the target template, an executable target big data application workflow is generated and saved.

2. The method according to claim 1, characterized in that, Analyze historical task execution data within the big data platform to identify, verify, and encapsulate high-frequency and stable workflow scenarios, including: Collect the execution directed acyclic graph, component configuration snapshots, and key performance indicators of historical task instances that have been successfully executed within the big data platform; Each historical task instance is abstracted into a task graph with big data processing components as nodes and data streams as edges, resulting in a set of task graphs; A frequent subgraph mining algorithm is used to identify common subgraph structures that appear more than a preset threshold from the task graph set as high-frequency workflows; For each high-frequency workflow, based on the execution records of the high-frequency workflow in multiple historical task instances, calculate the stability coefficient related to the success rate and performance indicators of the high-frequency workflow, and determine the high-frequency workflows whose stability coefficients meet the preset conditions as candidate stable workflows. The candidate stable workflow, along with the inter-component dependencies, data flow conventions, and parameter passing rules contained within it, are encapsulated into a structured scenario workflow template.

3. The method according to claim 2, characterized in that, Also includes: Automatically generate business scenario description text for the encapsulated scenario workflow template; The scenario workflow templates and their corresponding business scenario descriptions are injected into the knowledge graph of the scenario workflow template library; the knowledge graph is used to record the functional similarity, composability, and upstream and downstream relationships between scenario workflow templates.

4. The method according to claim 1, characterized in that, The multi-layer editable structure includes: A canvas layer, comprising a canvas, which serves as the carrier for the overall arrangement and display of the workflow; The logical parent node layer includes at least one parent node. Each parent node represents a complete scene workflow template on the canvas, which encapsulates a defined set of input ports, a set of output ports, and internal sub-workflow logic. The atomic child node layer consists of multiple atomic child nodes connected by internal data edges within each parent node. Each atomic child node corresponds to an execution operator that is parameterized and encapsulated from the underlying capability interface of the big data platform. The scenario workflow template pre-sets the connection attributes and parameter passing logic of the internal data edges at the atomic child node layer, and hides these attributes from external users at the logical parent node layer, exposing only configurable items based on business semantics.

5. The method according to claim 4, characterized in that, The system receives runtime environment parameters input by the user for the multi-layered editable structure, including: Using the parent node as the basic configuration unit, a unified parameter configuration panel is provided to the user; In the parameter configuration panel, only configuration items related to business logic are exposed, while the underlying technical connections and parameter transmission details of the atomic sub-node layer are hidden.

6. The method according to claim 4, characterized in that, Also includes: When a user connects the output port of the first parent node to the input port of the second parent node in the canvas, the output data pattern declared by the output port of the first parent node is parsed, and the input data pattern declared by the input port of the second parent node is parsed. If the output data pattern is compared with the input data pattern, and there are incompatibilities in field names, data types, or constraints, at least one field mapping scheme is generated based on the semantic similarity of the field names and type conversion rules, and provided to the user for confirmation or automatically executed for data conversion.

7. The method according to claim 1, characterized in that, The system receives runtime environment parameters input by the user for the multi-layered editable structure, including: Based on the historical deployment records of the target template, the resource usage habits of the user's organization, and / or the real-time resource load of the target deployment environment, at least one set of technical resource parameters is determined as recommended values; The recommended values ​​are displayed to the user in a visual development interface.

8. The method according to claim 7, characterized in that, Based on the historical deployment records of the target template, the resource usage habits of the user's organization, and / or the real-time resource load of the target deployment environment, at least one set of technical resource parameters is determined as recommended values, including: Based on the historical deployment records of the target template, the resource usage habits of the user's organization, and / or the real-time resource load of the target deployment environment, at least one set of technical resource parameters is predicted as recommended values ​​using a parameter recommendation model. The parameter recommendation model is a regression model trained based on machine learning or a rule-based empirical model. The input features of the parameter recommendation model include at least: task data volume estimation, template complexity index, historical resource consumption pattern, and current resource pool idle rate.

9. The method according to claim 1, characterized in that, Based on the received runtime environment parameters and the component dependencies, data flow conventions, and parameter passing rules within the target template, an executable target big data application workflow is generated and saved, including: Traverse the internal structure of the target template to parse out the dependency topology between multiple processing components; Based on the default parameters preset in the target template, the runtime environment parameters input by the user, and the recommended computing resource parameters, the final runtime configuration of each processing component is generated. Based on the dependency topology between the multiple processing components and the final runtime configuration of each processing component, the multiple processing components are merged into a complete workflow. The merged complete workflow definition is serialized into a deployment artifact, and a version snapshot containing a hash identifier, timestamp, and creation context is generated for the deployment artifact.

10. The method according to claim 1, characterized in that, Also includes: During the execution of the target big data application workflow, runtime telemetry data is collected during the execution process; The runtime telemetry data is analyzed and compared with the preset operating parameters of the target template. When a continuous performance deviation or resource utilization imbalance is detected, parameter optimization suggestions are generated for the target template to trigger version iteration of the target template.

11. The method according to claim 1, characterized in that, After generating and saving the executable workflow for the target big data application, the following steps are also included: The generated target big data application workflow is automatically submitted to the scheduling system of the big data platform; In an isolated testing environment, the target big data application workflow is automated for testing using historical data samples or synthetic data; If the automated test passes, the target big data application workflow is published to the production scheduling queue of the scheduling system; otherwise, an alarm is triggered and the system is rolled back to the configuration stage.

12. A big data application scenario-based development device, characterized in that, include: The template generation module is used to analyze historical task execution data within the big data platform to identify, verify, and encapsulate high-frequency and stable scenario workflows, forming a reusable scenario workflow template library. Each scenario workflow template in the scenario workflow template library encapsulates multiple big data processing components, as well as the dependencies, data flow conventions, and parameter passing rules among these components. The template loading module is used to provide users with a visual development interface containing the scenario workflow template library; in response to the user's selection operation of the target template in the scenario workflow template library, the module loads the multi-layer editable structure corresponding to the target template in the visual development interface; The parameter receiving module is used to receive runtime environment parameters input by the user for the multi-layer editable structure; the runtime environment parameters include at least one of the following: workflow identifier, access operation permission, data endpoint, execution scheduling strategy, and resource declaration parameters; The workflow generation module is used to generate and save an executable target big data application workflow based on the received runtime environment parameters and the component dependencies, data flow conventions and parameter passing rules in the target template.

13. A computer device, characterized in that, It includes a processor and a memory for storing processor-executable instructions, wherein the processor, when executing the instructions, implements the steps of the method according to any one of claims 1 to 11.

14. A computer-readable storage medium storing computer instructions thereon, characterized in that, When the instructions are executed by the processor, they implement the steps of the method according to any one of claims 1 to 11.

15. A computer program product comprising a computer program / instructions, characterized in that, When the computer program / instructions are executed by the processor, they implement the steps of the method according to any one of claims 1 to 11.

Images