Business process automatic expansion method and system based on data consanguinity

Abstract

The invention provides an automatic business process expansion method and system based on data consanguinity. The method comprises the following steps: constructing a consanguinity graph; setting an automatic expansion rule in the blood relationship graph; dynamically expanding the scheduling according to the automatic expansion rule to obtain an automatic expansion scheme; integrating an automatic expansion scheme to obtain an expansion task, and automatically implementing and feeding back optimization; the key technology supports an expansion task; and optimizing resource allocation, monitoring and adjustment of the extended task. According to the invention, through an automatic expansion scheme based on the blood relationship, the expansion accuracy, the flexibility of the system and the response speed are improved; through key technical support of real-time data monitoring and analysis, containerization, micro-service architecture and the like, the utilization efficiency of resources is improved; and through automatic implementation and feedback optimization, the real-time response capability of the system is improved.

Description

Technical Field

[0001] The present invention belongs to the field of database management, and in particular, relates to a method and system for automatically expanding a business process based on data lineage. Background Art

[0002] In the modern information technology environment, the automation and intelligence of business processes have become crucial means for enterprises to improve efficiency and competitiveness. In data-driven business environments, the automatic scalability of business processes has become a critical requirement. Traditional business process scalability methods, primarily based on predefined rules and static resource allocation, lack flexibility and are unable to adapt to dynamically changing data volumes and business needs.

[0003] Existing technical solutions mainly include rule-based process expansion and machine learning-based process optimization. Rule-based process expansion triggers and allocates resources through predefined rules. This approach lacks flexibility and cannot adapt to dynamically changing data volumes and business needs. It lacks real-time response capabilities and requires manual intervention for adjustments, which is inefficient. Machine learning-based process optimization, on the other hand, uses machine learning models to predict data volumes and business needs, thereby allocating resources in advance. However, this method requires a large amount of data and computing resources, and the accuracy of the model is also affected by the quality and quantity of the data, which can easily lead to a waste of computing resources. At the same time, existing technologies are unable to accurately identify the task nodes or data flow paths that need to be expanded, which can easily lead to resource waste and performance degradation. It has major limitations and cannot meet the needs of the modern information technology environment.

[0004] The patent document "A Method, System, Device, and Medium for Optimizing Task Scheduling Based on Lineage" (CN115827226A) focuses on automated configuration of task scheduling through lineage analysis. This approach automatically generates a task dependency graph, reducing the workload and error rate of manually configuring task dependencies. The automatically generated directed acyclic graph (DAG) also increases the number of tasks included. However, because the lineage relationship model it constructs primarily focuses on the direct dependencies between tasks and data, it lacks real-time dynamic adjustment and a robust automatic expansion mechanism.

[0005] Therefore, there is a need for an automatic expansion method for business processes that can flexibly adapt to dynamic changes in data volume and business needs and has low load, and can be widely used in application fields such as data analysis, data warehousing, and real-time data processing. Summary of the Invention

[0006] In view of the defects in the prior art, the purpose of the present invention is to provide a method and system for automatically expanding business processes based on data lineage.

[0007] The method for automatically expanding a business process based on data lineage provided by the present invention includes:

[0008] Step S1: construct a blood relationship model;

[0009] Step S2: setting automatic expansion rules in the blood relationship model;

[0010] Step S3: Dynamically expand the schedule according to the automatic expansion rules to obtain an automatic expansion plan;

[0011] Step S4: Integrate the automatic expansion plan to obtain the expansion task, automatically implement it and provide feedback for optimization;

[0012] Step S5: Expand the task through key technical support;

[0013] Step S6: Optimize resource allocation for the extended task, monitor and adjust.

[0014] Preferably, the step S1 includes:

[0015] Step S1.1: Collect metadata of business data, build a blood relationship diagram between task nodes, and clarify dependencies and data flows;

[0016] Step S1.2: Create a DAG graph to define dependencies, data flows, and task dependencies to form a blood relationship;

[0017] Step S1.3: Based on the blood relationship, use a relational database or modeling tool to establish a blood relationship model.

[0018] The step S2 comprises:

[0019] Step S2.1: Analyze the dependency relationship and the blood relationship model to analyze the DAG graph and select the task node to be expanded;

[0020] Step S2.2: Set trigger conditions for expansion based on data source access and data volume growth;

[0021] Step S2.3: Expand horizontally or vertically the node or sub-process.

[0022] Preferably, step S3 includes:

[0023] Identification step: Decompose dependencies and data flows, and use the blood relationship model to identify key paths and bottlenecks;

[0024] Allocation step: allocate computing resources and dynamically adjust CPU and memory;

[0025] Scheduling steps: Monitor the status of task nodes in real time, perform dynamic scheduling and parallel processing, automatically expand discovered bottlenecks, and record automatic expansion plans.

[0026] The step S4 comprises:

[0027] Step S4.1: Integrate the automatic scaling solution with DevOps tools to form and execute scaling tasks.

[0028] Step S4.2: Record the effects and problems of the expansion and optimize the automatic expansion plan through the feedback mechanism.

[0029] Preferably, the trigger condition is that the data volume exceeds 500MB or the task execution time exceeds 30 minutes.

[0030] In step S2.3, the task node type is determined based on the task node execution time, resource utilization rate, and position in the blood relationship diagram;

[0031] If it is a parallelizable task, horizontal expansion is adopted; if it is a resource bottleneck task, vertical expansion is adopted.

[0032] The feedback mechanism is to analyze the expansion effect based on the post-expansion task log and performance indicators, and adjust the automatic expansion rules.

[0033] Preferably, the key technologies in step S5 include using big data analysis tools and real-time monitoring platforms to monitor expansion tasks, supporting the expansion and migration of expansion tasks through containerization technology, and using machine learning models to predict traffic trends and expansion needs.

[0034] In step S6, computing resources are automatically allocated according to the predicted expansion demand, and the usage of computing resources is monitored in real time through the task scheduler;

[0035] Use monitoring tools to monitor the execution status of task nodes in real time, automatically expand discovered bottlenecks based on automatic expansion rules, or dynamically allocate computing resources based on priority.

[0036] The computing resources include CPU, memory and / or bandwidth.

[0037] The priority is obtained by ranking based on a comprehensive evaluation of the importance of the task node in the business process, the scope of data processing impact, and the real-time requirements.

[0038] According to the present invention, a business process automatic expansion system based on data lineage is provided, comprising:

[0039] Module M1: Constructing a kinship model;

[0040] Module M2: Setting automatic expansion rules in the blood relationship model;

[0041] Module M3: Dynamically expands the schedule according to the automatic expansion rules to obtain the automatic expansion plan;

[0042] Module M4: Integrate automatic expansion solutions to obtain expansion tasks, automate implementation, and provide feedback for optimization;

[0043] Module M5: Extending mission support through key technologies;

[0044] Module M6: Optimize resource allocation for expansion tasks, monitor and adjust.

[0045] Preferably, the module M1 includes:

[0046] Module M1.1: Collect metadata of business data, build a blood relationship diagram between task nodes, and clarify dependencies and data flows;

[0047] Module M1.2: Create a DAG graph to define dependencies, data flows, and task dependencies, forming a blood relationship;

[0048] Module M1.3: Use relational databases or modeling tools to establish a blood relationship model based on blood relationships.

[0049] The module M2 includes:

[0050] Module M2.1: Parse dependencies, analyze the DAG graph, and select and determine the task nodes to be expanded;

[0051] Module M2.2: Set expansion trigger conditions based on data source access and data volume growth;

[0052] Module M2.3: Horizontal or vertical expansion of nodes or sub-processes.

[0053] Preferably, the module M3 includes:

[0054] Identification module: decompose dependencies and data flows, identify critical paths and bottlenecks;

[0055] Allocation module: allocates computing resources and dynamically adjusts CPU and memory;

[0056] Scheduling module: monitors the status of task nodes in real time, performs dynamic scheduling and parallel processing, automatically expands discovered bottlenecks, and records automatic expansion plans.

[0057] The module M4 includes:

[0058] Module M4.1: Integrate the automatic scaling solution with DevOps tools to form and execute scaling tasks;

[0059] Module M4.2: Record the effects and problems of expansion, and optimize the automatic expansion plan through feedback mechanism.

[0060] Preferably, the trigger condition is that the data volume exceeds 500MB or the task execution time exceeds 30 minutes.

[0061] In the module M2.3, the task node type is determined based on the task node execution time, resource utilization rate, and position in the blood relationship diagram;

[0062] If it is a parallelizable task, horizontal expansion is adopted; if it is a resource bottleneck task, vertical expansion is adopted.

[0063] The feedback mechanism is to analyze the expansion effect based on the post-expansion task log and performance indicators, and adjust the automatic expansion rules.

[0064] Preferably, the key technologies in the module M5 include using big data analysis tools and real-time monitoring platforms to monitor expansion tasks, supporting the expansion and migration of expansion tasks through containerization technology, and using machine learning models to predict traffic trends and expansion needs.

[0065] The module M6 automatically allocates computing resources according to the predicted expansion demand and monitors the computing resource usage in real time through the task scheduler;

[0066] Use monitoring tools to monitor the execution status of task nodes in real time, automatically expand discovered bottlenecks based on automatic expansion rules, or dynamically allocate computing resources based on priority.

[0067] The computing resources include CPU, memory and / or bandwidth.

[0068] The priority is obtained by ranking based on a comprehensive evaluation of the importance of the task node in the business process, the scope of data processing impact, and the real-time requirements.

[0069] Compared with the prior art, the present invention has the following beneficial effects:

[0070] 1. The present invention realizes the automatic expansion of business processes and the accurate identification and expansion of task nodes through an automatic expansion solution based on blood relationship, thereby improving the accuracy of expansion, system flexibility and response speed.

[0071] 2. The present invention realizes dynamic allocation and optimization of resources and improves resource utilization efficiency through key technical support such as real-time data monitoring and analysis, containerization and microservice architecture.

[0072] 3. The present invention realizes real-time response and automatic adjustment of business processes through automated implementation and feedback optimization, thereby improving the real-time response capability of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0073] Other features, objects and advantages of the present invention will become more apparent upon reading the detailed description of non-limiting embodiments with reference to the following drawings:

[0074] Figure 1The flowchart of the automatic expansion method of business processes based on data lineage is shown. DETAILED DESCRIPTION

[0075] The present invention will be described in detail below with reference to specific embodiments. The following examples will help those skilled in the art to further understand the present invention, but are not intended to limit the present invention in any form. It should be noted that, for those skilled in the art, several changes and improvements can be made without departing from the scope of the present invention. These all fall within the scope of protection of the present invention.

[0076] According to the present invention, a method for automatically expanding business processes based on data lineage is provided, which focuses on the automatic expansion of business processes. In addition to task scheduling, it also emphasizes the real-time triggering, feedback optimization, resource allocation and dynamic adjustment of expansion rules, and pays more attention to task expansion optimization in real-time dynamic business environments. Figure 1 For example, including:

[0077] Step S1: Constructing a blood relationship model;

[0078] Collect metadata for business data and use data flow diagrams or workflow diagrams to construct a kinship diagram between tasks. This clearly defines the dependencies between tasks and the data transfer paths (i.e., data flows) to provide a basis for subsequent automatic scaling. This kinship diagram not only identifies dependencies between tasks but also emphasizes the dynamic formulation and adjustment of real-time scaling strategies to meet varying loads and business needs, ensuring precise and flexible resource utilization. Create a DAG (directed acyclic graph) to define data dependencies, data flows, and task dependencies. Store kinship relationships and establish a kinship model using a relational database or modeling tools.

[0079] In more preferred examples, there is a data analysis task, which includes four subtasks: data collection, data preprocessing, data analysis, and data storage. There are certain dependencies and data transmission paths between these subtasks.

[0080] Step S2: Automatic expansion rule setting, setting trigger conditions;

[0081] According to business characteristics and workload conditions, set corresponding trigger conditions, such as the data volume exceeds a certain threshold, the task execution time exceeds the predetermined range, etc., so as to start the automatic expansion mechanism when necessary.

[0082] Specifically including: Step S2.1: parsing dependencies, analyzing the DAG graph of the blood relationship model to determine the entry point for expansion;

[0083] Step S2.2: Set expansion trigger conditions based on data source access and data volume growth;

[0084] In more preferred embodiments, the trigger conditions include data volume exceeding 500MB or task execution time exceeding 30 minutes, reflecting the I / O-intensive or compute-intensive nature of the task, respectively, and applicable to different expansion strategies. The expansion entry point is determined by the key task nodes identified by the lineage graph, and the trigger condition is used to determine whether these nodes have reached the expansion threshold. These two factors work together to determine the timing and target range of the expansion action.

[0085] Step S2.3: Horizontally expand or vertically expand certain nodes or sub-processes.

[0086] Whether a node or sub-process is expanded is determined by the system's real-time monitoring based on the node execution time, resource usage (CPU, memory) and position in the lineage relationship graph.

[0087] For parallelizable tasks, such as data cleaning and preprocessing, horizontal scaling is used; for resource-bottleneck tasks, such as complex aggregation calculations, vertical scaling is used to increase resource limits or container specifications.

[0088] Step S3: Dynamically extend scheduling;

[0089] A comprehensive automatic expansion mechanism, including horizontal and vertical expansion and automatic feedback optimization processes, enables the system to automatically expand and adjust in a timely and accurate manner when resources are insufficient or the load increases. Based on the analysis of the blood relationship graph, the blood relationship model identifies critical paths and bottlenecks, dynamically scheduling and expanding tasks. For example, a dynamic parallel processing mechanism is set up for data-intensive tasks to optimize resource allocation and improve system performance.

[0090] Specifically include:

[0091] Identification step: decompose process dependencies and identify critical paths and bottlenecks;

[0092] Allocation step: Automatically allocate computing resources and dynamically adjust resources such as CPU and memory;

[0093] Scheduling steps: Monitor task status in real time, perform dynamic scheduling and parallel processing, automatically expand discovered bottlenecks, and record automatic expansion plans.

[0094] In more preferred examples, data preprocessing and data analysis are data-intensive tasks, so these two tasks are set up as dynamic parallel processing mechanisms to improve system performance. This can avoid resource waste, improve resource utilization, and meet the demand for efficient resource utilization in big data analysis. Some optimization actions, such as dynamic resource allocation, can be triggered concurrently after critical path identification is completed, but they all rely on the results of lineage analysis.

[0095] The automatic expansion solution based on lineage relationships enables real-time adjustments based on business process changes without manual intervention, improving system flexibility. The lineage relationship graph accurately identifies task nodes or data flow paths that require expansion, avoiding resource waste and performance degradation, and improving expansion accuracy.

[0096] Step S4: automated implementation and feedback optimization;

[0097] Integrate the automatic expansion plan with DevOps tools to form an automated expansion implementation process, and record the effects and problems after each expansion. The blood relationship model continuously optimizes the expansion plan through a feedback mechanism to improve the expansion effect.

[0098] The feedback mechanism automatically analyzes the expansion effect based on the expanded task log and performance indicators (such as resource utilization and execution time) and adjusts the original rule base. Compared with the general DevOps feedback process, it has the ability to learn and self-tune rules oriented to the lineage graph structure, which enhances the adaptability of the system. Specifically, step S4 includes:

[0099] Step S4.1: Automated implementation: Integrate with DevOps tools to automatically generate expansion tasks, that is, automate the expansion implementation process and execute it;

[0100] Step S4.2: Feedback loop, recording expansion effects and problems, and continuously optimizing the automatic expansion plan is based on the feedback optimization of the blood relationship graph.

[0101] Through automated implementation and feedback optimization, it is possible to respond to changes in business processes in real time, improve the system's response speed, and solve the problem of insufficient real-time response capabilities.

[0102] In more preferred examples, Jenkins is used as a DevOps tool, and the automatic expansion solution is integrated with Jenkins to form an automated expansion implementation process.

[0103] Step S5: key technical support;

[0104] Utilize big data analysis tools and real-time monitoring platforms to analyze data flow changes and business process status, use containerization and microservice architecture to achieve rapid task expansion and migration, and use machine learning models to predict data node traffic trends and anticipate expansion needs. Specifically, this includes:

[0105] Real-time data monitoring and analysis, using big data analysis tools (Spark, Flink) and real-time monitoring platforms (Prometheus, Grafana) to monitor expansion tasks.

[0106] Containerization and microservices architecture support rapid expansion and migration of tasks through containerization technologies such as Docker and Kubernetes.

[0107] Machine learning helps predict traffic trends and expansion requirements. With machine learning, traffic trends at key data nodes can be predicted to anticipate expansion requirements, reducing unnecessary expansion and lowering costs.

[0108] In more preferred examples, Apache Spark is used as a big data analysis tool, Prometheus and Grafana are used as real-time monitoring platforms, Docker and Kubernetes are used as containerization and microservice architectures, and TensorFlow and PyTorch are used as machine learning models.

[0109] Step S6: resource allocation optimization, monitoring and adjustment;

[0110] Automatically allocate computing resources such as CPU, memory, bandwidth, etc. according to expansion requirements, monitor resource usage in real time through the task scheduler, optimize resource allocation, and improve resource utilization efficiency.

[0111] In the data warehouse sector, this technology enables automatic expansion and optimization of data warehouses, dynamically allocating computing resources based on changes in data volume and business needs, improving data warehouse performance and responsiveness. Furthermore, through automated implementation and feedback optimization mechanisms, expansion plans can be continuously optimized, raising the intelligence level of data warehouses.

[0112] In more preferred examples, Apache Mesos is used as a task scheduler to automatically allocate computing resources such as CPU, memory, and bandwidth according to expansion requirements. Apache Mesos also monitors resource usage in real time, optimizes resource allocation, and improves resource utilization efficiency.

[0113] Monitor the execution status of each task node in real time. If a bottleneck is found, it will automatically expand or dynamically allocate resources through priority adjustment to ensure stable system operation and accurate expansion of nodes or subtasks.

[0114] Automatic scaling during this phase involves real-time adjustments and optimizations to actual operational bottlenecks during runtime. This process also uses the lineage graph and real-time monitoring to identify and scale bottlenecks. A comprehensive score is assigned based on factors such as the task's importance within the business process, the scope of data processing impact, and real-time requirements. This score is used to prioritize dynamic resource scheduling, ensuring that critical tasks receive priority resources.

[0115] Through technologies such as real-time data monitoring and analysis, containerization, and microservices architecture, resources can be dynamically allocated, ensuring performance while avoiding idle resources, optimizing resource utilization, and addressing the problem of wasted computing resources in existing technologies. This, coupled with machine learning-assisted traffic trend prediction, enables automatic expansion and optimization of real-time data processing, precise control and dynamic adjustment of data processing, and dynamic allocation of computing resources based on real-time data changes and business needs, thereby improving the efficiency and accuracy of data analysis, and enhancing the performance and responsiveness of real-time data processing.

[0116] In more preferred examples, Nagios is used as a monitoring tool to monitor the execution status of each task node in real time. If a bottleneck is found, it will automatically expand or dynamically allocate resources through priority adjustment to ensure stable operation of the system.

[0117] By combining big data analysis, real-time monitoring, containerization technology, machine learning and other advanced technologies, the automatic expansion and optimization of business processes are achieved, the automation and intelligence level, flexibility and resource utilization of the system are improved, the accuracy of expansion and real-time response capabilities are improved, and costs are reduced. It has significant advantages and has broad market demand and application prospects.

[0118] The present invention also provides a business process automatic expansion system based on data lineage. The business process automatic expansion system based on data lineage can be implemented by executing the process steps of the business process automatic expansion method based on data lineage. That is, those skilled in the art can understand the business process automatic expansion method based on data lineage as an optimal implementation method of the business process automatic expansion system based on data lineage.

[0119] According to the present invention, a business process automatic expansion system based on data lineage is provided, comprising:

[0120] Module M1: Constructing a kinship model;

[0121] Module M2: Setting automatic expansion rules in the blood relationship model;

[0122] Module M3: Dynamically expands the schedule according to the automatic expansion rules to obtain the automatic expansion plan;

[0123] Module M4: Integrate automatic expansion solutions to obtain expansion tasks, automate implementation, and provide feedback for optimization;

[0124] Module M5: Extending mission support through key technologies;

[0125] Module M6: Optimize resource allocation for expansion tasks, monitor and adjust.

[0126] In more preferred embodiments, the module M1 includes:

[0127] Module M1.1: Collect metadata of business data, build a blood relationship diagram between task nodes, and clarify dependencies and data flows;

[0128] Module M1.2: Create a DAG graph to define dependencies, data flows, and task dependencies, forming a blood relationship;

[0129] Module M1.3: Use relational databases or modeling tools to establish a blood relationship model based on blood relationships.

[0130] The module M2 includes:

[0131] Module M2.1: Parse dependencies, analyze the DAG graph, and select and determine the task nodes to be expanded;

[0132] Module M2.2: Set expansion trigger conditions based on data source access and data volume growth;

[0133] Module M2.3: Horizontal or vertical expansion of nodes or sub-processes.

[0134] In more preferred embodiments, the module M3 includes:

[0135] Identification module: decompose dependencies and data flows, identify critical paths and bottlenecks;

[0136] Allocation module: allocates computing resources and dynamically adjusts CPU and memory;

[0137] Scheduling module: monitors the status of task nodes in real time, performs dynamic scheduling and parallel processing, automatically expands discovered bottlenecks, and records automatic expansion plans.

[0138] The module M4 includes:

[0139] Module M4.1: Integrate the automatic scaling solution with DevOps tools to form and execute scaling tasks;

[0140] Module M4.2: Record the effects and problems of expansion, and optimize the automatic expansion plan through feedback mechanism.

[0141] In more preferred examples, the trigger condition is that the data volume exceeds 500MB or the task execution time exceeds 30 minutes.

[0142] In the module M2.3, the task node type is determined based on the task node execution time, resource utilization rate, and position in the blood relationship diagram;

[0143] If it is a parallelizable task, horizontal expansion is adopted; if it is a resource bottleneck task, vertical expansion is adopted.

[0144] The feedback mechanism is to analyze the expansion effect based on the post-expansion task log and performance indicators, and adjust the automatic expansion rules.

[0145] In more preferred examples, the key technologies in the module M5 include using big data analysis tools and real-time monitoring platforms to monitor expansion tasks, supporting the expansion and migration of expansion tasks through containerization technology, and using machine learning models to predict traffic trends and expansion needs.

[0146] The module M6 automatically allocates computing resources according to the predicted expansion demand and monitors the computing resource usage in real time through the task scheduler;

[0147] Use monitoring tools to monitor the execution status of task nodes in real time, automatically expand discovered bottlenecks based on automatic expansion rules, or dynamically allocate computing resources based on priority.

[0148] The computing resources include CPU, memory and / or bandwidth.

[0149] The priority is obtained by ranking based on a comprehensive evaluation of the importance of the task node in the business process, the scope of data processing impact, and real-time requirements.

[0150] Those skilled in the art will appreciate that, in addition to implementing the system and its various devices, modules, and units provided by the present invention in purely computer-readable program code, it is entirely possible to implement the same functions of the system and its various devices, modules, and units provided by the present invention in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers by logically programming the method steps. Therefore, the system and its various devices, modules, and units provided by the present invention can be considered a hardware component, and the devices, modules, and units included therein for implementing various functions can also be considered as structures within the hardware component; the devices, modules, and units for implementing various functions can also be considered as both software modules implementing the method and structures within the hardware component.

[0151] The above describes specific embodiments of the present invention. It should be understood that the present invention is not limited to the specific embodiments described above, and those skilled in the art may make various changes or modifications within the scope of the claims, which do not affect the essence of the present invention. The embodiments of this application and the features in the embodiments may be combined with each other in any manner unless there is a conflict.

Claims

1. A method for automatically expanding a business process based on data lineage, characterized in that: include: Step S1: construct a blood relationship model; Step S2: setting automatic expansion rules in the blood relationship model; Step S3: Dynamically expand the schedule according to the automatic expansion rules to obtain an automatic expansion plan; Step S4: Integrate the automatic expansion plan to obtain the expansion task, automatically implement it and provide feedback for optimization; Step S5: Expand the task through key technical support; Step S6: Optimize resource allocation for the extended task, monitor and adjust.

2. The method for automatically expanding business processes based on data lineage according to claim 1, characterized in that: The step S1 comprises: Step S1.1: Collect metadata of business data, build a blood relationship diagram between task nodes, and clarify dependencies and data flows; Step S1.2: Create a DAG graph to define dependencies, data flows, and task dependencies to form a blood relationship; Step S1.3: Based on the blood relationship, use a relational database or modeling tool to establish a blood relationship model; The step S2 comprises: Step S2.1: Analyze the dependency relationship and the blood relationship model to analyze the DAG graph and select the task node to be expanded; Step S2.2: Set trigger conditions for expansion based on data source access and data volume growth; Step S2.3: Expand horizontally or vertically the node or sub-process.

3. The method for automatically expanding business processes based on data lineage according to claim 2, characterized in that: The step S3 comprises: Identification step: Decompose dependencies and data flows, and use the blood relationship model to identify key paths and bottlenecks; Allocation step: allocate computing resources and dynamically adjust CPU and memory; Scheduling step: monitor the status of task nodes in real time, perform dynamic scheduling and parallel processing, automatically expand discovered bottlenecks, and record automatic expansion plans; The step S4 comprises: Step S4.1: Integrate the automatic scaling solution with DevOps tools to form and execute scaling tasks. Step S4.2: Record the effects and problems of the expansion and optimize the automatic expansion plan through the feedback mechanism.

4. The method for automatically expanding business processes based on data lineage according to claim 3, characterized in that: The trigger condition is that the data volume exceeds 500MB or the task execution time exceeds 30 minutes; In step S2.3, the task node type is determined based on the task node execution time, resource utilization rate, and position in the blood relationship diagram; If it is a parallelizable task, horizontal scaling is used; if it is a resource-bottleneck task, vertical scaling is used; The feedback mechanism is to analyze the expansion effect based on the post-expansion task log and performance indicators, and adjust the automatic expansion rules.

5. The method for automatically expanding business processes based on data lineage according to claim 1, characterized in that: The key technologies in step S5 include using big data analysis tools and a real-time monitoring platform to monitor expansion tasks, supporting the expansion and migration of expansion tasks through containerization technology, and using machine learning models to predict traffic trends and expansion needs; In step S6, computing resources are automatically allocated according to the predicted expansion demand, and the usage of computing resources is monitored in real time through the task scheduler; Use monitoring tools to monitor the execution status of task nodes in real time, automatically expand discovered bottlenecks based on automatic expansion rules, or dynamically allocate computing resources based on priority. The computing resources include CPU, memory and / or bandwidth; The priority is obtained by ranking based on a comprehensive evaluation of the importance of the task node in the business process, the scope of data processing impact, and the real-time requirements.

6. A business process automatic expansion system based on data lineage, characterized by: include: Module M1: Constructing a kinship model; Module M2: Setting automatic expansion rules in the blood relationship model; Module M3: Dynamically expands the schedule according to the automatic expansion rules to obtain the automatic expansion plan; Module M4: Integrate automatic expansion solutions to obtain expansion tasks, automate implementation, and provide feedback for optimization; Module M5: Extending mission support through key technologies; Module M6: Optimize resource allocation for expansion tasks, monitor and adjust.

7. The data lineage-based business process automatic expansion system according to claim 6 is characterized in that: The module M1 includes: Module M1.1: Collect metadata of business data, build a blood relationship diagram between task nodes, and clarify dependencies and data flows; Module M1.2: Create a DAG graph to define dependencies, data flows, and task dependencies, forming a blood relationship; Module M1.3: Use relational databases or modeling tools to build a kinship model based on kinship relationships; The module M2 includes: Module M2.1: Parse dependencies, analyze the DAG graph, and select and determine the task nodes to be expanded; Module M2.2: Set expansion trigger conditions based on data source access and data volume growth; Module M2.3: Horizontal or vertical expansion of nodes or sub-processes.

8. The data lineage-based business process automatic expansion system according to claim 7 is characterized in that: The module M3 includes: Identification module: decompose dependencies and data flows, identify critical paths and bottlenecks; Allocation module: allocates computing resources and dynamically adjusts CPU and memory; Scheduling module: monitors the status of task nodes in real time, performs dynamic scheduling and parallel processing, automatically expands discovered bottlenecks, and records automatic expansion plans; The module M4 includes: Module M4.1: Integrate the automatic scaling solution with DevOps tools to form and execute scaling tasks; Module M4.2: Record the effects and problems of expansion, and optimize the automatic expansion plan through feedback mechanism.

9. The data lineage-based business process automatic expansion system according to claim 8, characterized in that: The trigger condition is that the data volume exceeds 500MB or the task execution time exceeds 30 minutes; In the module M2.3, the task node type is determined based on the task node execution time, resource utilization rate, and position in the blood relationship diagram; If it is a parallelizable task, horizontal scaling is used; if it is a resource-bottleneck task, vertical scaling is used; The feedback mechanism is to analyze the expansion effect based on the post-expansion task log and performance indicators, and adjust the automatic expansion rules.

10. The automatic business process expansion system based on data lineage according to claim 6, characterized in that: Key technologies in module M5 include using big data analysis tools and a real-time monitoring platform to monitor expansion tasks, supporting the expansion and migration of expansion tasks through containerization technology, and using machine learning models to predict traffic trends and expansion needs; The module M6 automatically allocates computing resources according to the predicted expansion demand and monitors the computing resource usage in real time through the task scheduler; Use monitoring tools to monitor the execution status of task nodes in real time, automatically expand discovered bottlenecks based on automatic expansion rules, or dynamically allocate computing resources based on priority. The computing resources include CPU, memory and / or bandwidth; The priority is obtained by ranking based on a comprehensive evaluation of the importance of the task node in the business process, the scope of data processing impact, and the real-time requirements.

Images