Data pipeline integration method and system

Abstract

The invention discloses a data pipeline integration method and system, relates to the technical field of data processing, and solves the problems of poor flexibility and high delay of a traditional data pipeline. According to the method, a customized complexity index CCI is calculated based on parameters such as a structural deviation degree Sd through a meta-analysis task request, and after the customized complexity index CCI is compared with a threshold value alpha, a standard or customized dynamic adaptation module is decided and dynamically assembled to efficiently cope with a non-standard data source and complex logic; meanwhile, according to the method, a batch processing mode or a stream processing mode is selected according to delay requirements, key indexes including delay Ld are monitored in real time, features such as a delay jitter coefficient Jd and a cost-to-benefit ratio Cb are fitted into an adaptive efficiency index AEI, then the AEI is compared with an efficiency threshold value beta, pipeline closed-loop self-optimization is driven, such as automatic switching of the processing mode, and the delay jitter coefficient Jd, the cost-to-benefit ratio Cb and the like are matched. And low delay and high adaptability under the scenes of SaaS operation and maintenance and the like are ensured.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a data pipeline integration method and system. Background Art

[0002] This data integration approach, designed specifically for business scenarios involving system development and SaaS operations and maintenance, focuses on building a standardized, reusable data pipeline system to efficiently and stably connect and integrate data from heterogeneous sources, including diverse business systems, user behavior logs, and third-party application APIs. By extracting, cleaning, transforming, and loading data, this approach transforms raw data into valuable information and delivers it uniformly to a data warehouse or analytics platform. Ultimately, it aims to automate and standardize data flows, providing reliable data support for rapid iteration in system development, intelligent SaaS operations and maintenance, and business decision-making.

[0003] Based on the characteristics of data pipeline integration methods, the following technical shortcomings often exist in practical applications:

[0004] Lack of flexibility and difficulty in customization:

[0005] The emphasis on building a standardized, reusable pipeline system can become rigid when addressing diverse and rapidly changing business needs. For example, when a SaaS customer needs to access a private data source with a non-standard, specialized format, or requires unconventional, complex, and personalized data cleansing logic, the standardized pipeline may not be able to directly support it. Forced modifications to the generic templates undermine their reusability, increase maintenance costs and risks, and make customized development longer and more difficult.

[0006] Data processing latency is high, making it difficult to support real-time scenarios:

[0007] To achieve standardization and stability, these data pipelines typically employ a batch processing architecture, executing ETL (Extract, Transform, Load) tasks on a scheduled basis (e.g., hourly or daily). This architecture directly leads to the first drawback—lack of flexibility. Its most direct consequence is data latency. For example, in SaaS system operations, if real-time monitoring of anomalies in key user operations is required and immediate alerts are issued, a data update frequency of only once an hour is unacceptable. Problem detection and response will be significantly delayed, making it impossible to meet the requirements of operations monitoring or business risk control scenarios that require high real-time performance. Summary of the Invention

[0008] In view of the deficiencies in the prior art, the present invention provides a data pipeline integration method and system, which solves the technical shortcomings mentioned in the background technology.

[0009] To achieve the above objectives, the present invention is implemented through the following technical solutions: A data pipeline integration method includes the following steps:

[0010] S1. Receive and parse a data integration task request, which includes data source characteristics, data targets, and business scenario requirements. Perform meta-analysis on the task request to generate an initial pipeline configuration strategy.

[0011] S2. Based on the initial pipeline configuration strategy, evaluate the non-scaling of the data source and the real-time requirements of the business scenario, calculate and obtain a customized complexity index (CCI), and compare the customized complexity index (CCI) with a preset complexity threshold α to determine a pipeline construction mode.

[0012] S3. Dynamically assemble a data pipeline based on the determined pipeline construction mode. The pipeline consists of one or more pluggable, dynamically adaptable modules and selects batch or stream processing mode to perform data integration tasks. During pipeline operation, monitor its key performance indicators in real time, including data processing latency Ld, throughput Tp, and resource consumption Rc, to construct a pipeline operation status set.

[0013] S4. Performing a time series feature analysis on the pipeline operation status set obtained in step S3 to obtain the delay jitter coefficient Jd, effective throughput Te, and cost-effectiveness ratio Cb in different monitoring periods. Using a pre-trained pipeline performance prediction model, the adaptive performance index AEI is obtained through fitting calculation.

[0014] S5. Pre-set an efficiency threshold β, and compare and analyze the efficiency threshold β with the adaptive efficiency index AEI to comprehensively evaluate whether the current data pipeline configuration is optimal, and generate dynamic tuning instructions based on the evaluation results to achieve closed-loop self-optimization of the pipeline.

[0015] Preferably, S11, data source characteristics include data format, interface type and data change frequency;

[0016] Business scenario requirements include latency requirements, data consistency level, and target applications;

[0017] S111. Meta-analysis includes analysis of the difference between the data source format and the standard library, and analysis of the matching degree between the business delay requirements and the system's default processing capabilities, in order to generate an initial pipeline configuration strategy including recommended adaptation modules, processing modes, and resource estimates.

[0018] Preferably, in S21, the structural deviation degree Sd, the conversion logic complexity Tc, and the interface call frequency Fc of the collected data sources are used as calculation parameters, and the customized complexity index CCI is calculated using the following formula:

[0019]

[0020] What needs to be explained in this formula is that w1, w2, and w3 are weight coefficients of different dimensions, and they satisfy w1+w2+w3=1.

[0021] Preferably, S22, the customized complexity index CCI is compared with the complexity threshold α, and the pipeline construction mode is determined in combination with the delay requirements of the business scenario. The specific contents are as follows:

[0022] If the CCI exceeds the complexity threshold α, or the latency requirement is in seconds, it is considered a highly customized or real-time scenario. The pipeline construction mode will select a customized dynamic adaptation module and prioritize the stream processing mode.

[0023] If the CCI does not exceed the complexity threshold α and the latency requirement is at the minute level or lower, it is determined to be a standard scenario. The pipeline construction mode will use a standardized dynamic adaptation module and prioritize batch processing mode to save costs.

[0024] Preferably, S31, after receiving the task request, dynamically assemble the data pipeline, the specific content of the dynamic assembly of the data pipeline is as follows:

[0025] S311. The dynamic adaptation module is a containerized service encapsulated for a specific data processing task and can be independently deployed and called;

[0026] S312. According to the pipeline construction mode, the corresponding standardized or customized DAM is automatically selected from the module library and linked into a complete data processing link in a logical order.

[0027] Preferably, S32, during pipeline operation, dividing the monitoring cycle into a plurality of discrete monitoring time periods to monitor and record key performance indicators in real time;

[0028] S321, the data processing delay Ld refers to the end-to-end time from data generation to final loading into the target system;

[0029] S322. The resource consumption Rc includes the CPU, memory, and network bandwidth occupied by executing the pipeline task, and the cost-benefit ratio Cb is calculated based on the throughput Tp and the resource consumption Rc.

[0030] Preferably, S41, use a long short-term memory network to build an original model, use the historical pipeline operation status set as training data to train the original model, and use the trained model as a pipeline performance prediction model to predict the performance trend in the future under the current configuration.

[0031] S42. Calculate and obtain a delay jitter coefficient Jd based on the data processing delay Ld sequence within the continuous monitoring period. The delay jitter coefficient Jd is obtained by the following formula:

[0032]

[0033] It should be noted that in this formula: σ(Ld) represents the standard deviation of the data processing delay Ld during the monitoring period, and μ(Ld) represents the average value of the data processing delay Ld.

[0034] Preferably, in S43, a cost-benefit ratio Cb is calculated based on the throughput Tp and the resource consumption Rc. The cost-benefit ratio Cb is obtained by the following formula:

[0035]

[0036] Where k1 and k2 are the cost weight coefficients of different resources, CPU and Memory are the normalized consumption values ​​of CPU and memory, respectively.

[0037] The trained pipeline performance prediction model is used to normalize the delay jitter coefficient Jd, effective throughput Te, and cost-effectiveness ratio Cb, and then fits and calculates the adaptive performance index AEI. The adaptive performance index AEI is obtained using the following formula:

[0038]

[0039] Where Wt, Wc, and Wj are weight coefficients, representing the importance attached to throughput, cost-effectiveness, and delay jitter, respectively; Te_norm, Cb_norm, and Jd_norm are the normalized values ​​of each indicator.

[0040] Preferably, S5.1 generates a dynamic tuning instruction based on the evaluation result, the specific content of which is as follows:

[0041] If the adaptive efficiency index (AEI) falls below the efficiency threshold β, indicating that the current pipeline configuration is inefficient, the system will automatically execute tuning instructions, such as switching from batch processing to stream processing mode, increasing resource allocation, or issuing warnings to R&D personnel, suggesting optimization of related dynamic adaptation modules.

[0042] If the adaptive efficiency index AEI is not lower than the efficiency threshold β, it means that the current pipeline configuration meets the business needs and the current configuration will be maintained and continuously monitored.

[0043] A data pipeline integration system includes the following modules:

[0044] The task parsing and strategy generation module receives and parses data integration task requests, which include data source characteristics, data targets, and business scenario requirements. It then performs meta-analysis on the task requests to generate an initial pipeline configuration strategy.

[0045] The complexity assessment and mode decision module evaluates the non-scaling of the data source and the real-time requirements of the business scenario based on the initial pipeline configuration strategy, calculates and obtains a customized complexity index CCI, and compares the customized complexity index CCI with a preset complexity threshold α to determine the pipeline construction mode;

[0046] The pipeline dynamic assembly and runtime monitoring module dynamically assembles a data pipeline based on a defined pipeline construction mode. The pipeline consists of one or more pluggable dynamic adapter modules and selects batch or stream processing mode to perform data integration tasks. During pipeline operation, key performance indicators (KPIs) are monitored in real time, including data processing latency Ld, throughput Tp, and resource consumption Rc, to construct a pipeline operation status set.

[0047] The pipeline efficiency analysis and prediction module performs time series feature analysis on the pipeline operation status set obtained in step S3 to obtain the delay jitter coefficient Jd, effective throughput Te, and cost-effectiveness ratio Cb in different monitoring periods. It also obtains the adaptive efficiency index AEI by fitting and calculating the pre-trained pipeline efficiency prediction model.

[0048] The closed-loop adaptive tuning module pre-sets an efficiency threshold β and compares and analyzes the efficiency threshold β with the adaptive efficiency index AEI to comprehensively evaluate whether the current data pipeline configuration is optimal, and generates dynamic tuning instructions based on the evaluation results to achieve closed-loop self-optimization of the pipeline.

[0049] The present invention provides a data pipeline integration method and system. It has the following beneficial effects:

[0050] (1) This data pipeline integration method and system effectively solves the problems of insufficient flexibility and customization difficulty of traditional data pipelines by introducing meta-analysis of task requests and customized complexity evaluation. Specifically, this method first analyzes the task request containing data source characteristics and business scenario requirements through the task parsing and strategy generation module; then, through the complexity evaluation and pattern decision module, the customized complexity index CCI is calculated using the structural deviation Sd of the data source, the conversion logic complexity Tc and the interface call frequency Fc, and compared with the preset complexity threshold α to scientifically decide the pipeline construction pattern; finally, through the pipeline dynamic assembly and runtime monitoring module, according to the pattern decision result, standardized or customized dynamic adaptation modules are automatically selected from the module library for link assembly, so that non-standard data sources and complex business logic can be efficiently processed, avoiding the high maintenance cost and risk brought by modifying the general template, and significantly improving the adaptability and scalability of the data pipeline.

[0051] (2) This data pipeline integration method and system successfully overcomes the defects of high data latency and difficulty in supporting real-time scenarios in traditional batch processing architectures by establishing a hybrid processing mode and a closed-loop self-optimization mechanism. Specifically, when determining the pipeline construction mode, this method will make a choice between batch processing and stream processing modes based on the latency requirements of the business scenario. During the pipeline operation, key performance indicators including data processing delay Ld, throughput Tp and resource consumption Rc will be continuously monitored, and indicators such as the delay jitter coefficient Jd reflecting stability and the cost-effectiveness ratio Cb reflecting efficiency will be calculated through the pipeline performance analysis and prediction module. Then, through the pipeline performance prediction model trained by the long short-term memory network, each indicator is fitted into a comprehensive adaptive performance index AEI. Finally, the closed-loop adaptive tuning module compares the AEI with the preset performance threshold β. If the performance does not meet the standard, a dynamic tuning instruction is generated, such as automatically switching from batch processing to stream processing mode, thereby ensuring that the data pipeline can dynamically meet the second-level real-time requirements of scenarios such as SaaS operation and maintenance, thereby ensuring the timeliness of business response. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 Schematic diagram of the method steps of the present invention;

[0053] Figure 2 It is a schematic diagram of the system framework structure of the present invention. DETAILED DESCRIPTION

[0054] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0055] Example 1

[0056] See also Figure 1 The present invention provides a data pipeline integration method, comprising the following steps:

[0057] S1. Receive and parse a data integration task request, which includes data source characteristics, data targets, and business scenario requirements. Perform meta-analysis on the task request to generate an initial pipeline configuration strategy.

[0058] S2. Based on the initial pipeline configuration strategy, evaluate the non-scaling of the data source and the real-time requirements of the business scenario, calculate and obtain a customized complexity index (CCI), and compare the customized complexity index (CCI) with a preset complexity threshold α to determine a pipeline construction mode.

[0059] S3. Dynamically assemble a data pipeline based on the determined pipeline construction mode. The pipeline consists of one or more pluggable, dynamically adaptable modules and selects batch or stream processing mode to perform data integration tasks. During pipeline operation, monitor its key performance indicators in real time, including data processing latency Ld, throughput Tp, and resource consumption Rc, to construct a pipeline operation status set.

[0060] S4. Performing a time series feature analysis on the pipeline operation status set obtained in step S3 to obtain the delay jitter coefficient Jd, effective throughput Te, and cost-effectiveness ratio Cb in different monitoring periods. Using a pre-trained pipeline performance prediction model, the adaptive performance index AEI is obtained through fitting calculation.

[0061] S5. Pre-set an efficiency threshold β, and compare and analyze the efficiency threshold β with the adaptive efficiency index AEI to comprehensively evaluate whether the current data pipeline configuration is optimal, and generate dynamic tuning instructions based on the evaluation results to achieve closed-loop self-optimization of the pipeline.

[0062] Furthermore, the S1 step enables accurate interpretation of data integration task requests, laying an intelligent foundation for subsequent automated processes;

[0063] Then, through step S2, the customized complexity index CCI is calculated and compared with the preset threshold α, achieving scientific quantification of task complexity and accurate decision-making on construction mode;

[0064] In step S3, dynamic assembly and real-time monitoring of data processing latency Ld, throughput Tp, and resource consumption Rc enable agile pipeline construction and transparent collection of operational status.

[0065] In step S4, the original metrics are refined through timing analysis into the delay jitter coefficient Jd, effective throughput Te, and cost-effectiveness ratio Cb, ultimately fitting them into a comprehensive adaptive efficiency index (AEI), providing in-depth insight and quantitative assessment of pipeline performance.

[0066] Finally, step S5 compares the AEI with the efficiency threshold β to drive the generation of dynamic tuning instructions, building a complete closed-loop self-optimization mechanism to ensure that the data pipeline can continue to operate in an optimal state, significantly improving the system's automation, intelligence, and resource utilization efficiency.

[0067] S11. Data source characteristics include data format, interface type, and data change frequency;

[0068] Business scenario requirements include latency requirements, data consistency level, and target applications;

[0069] S111. Meta-analysis includes analysis of the difference between the data source format and the standard library, and analysis of the matching degree between the business delay requirements and the system's default processing capabilities, in order to generate an initial pipeline configuration strategy including recommended adaptation modules, processing modes, and resource estimates.

[0070] S21. Collect the data source's structural deviation Sd, conversion logic complexity Tc, and interface call frequency Fc as calculation parameters, and calculate the customized complexity index CCI using the following formula:

[0071]

[0072] It should be noted that this formula is as follows: w1, w2, and w3 are weight coefficients of different dimensions, and they satisfy w1+w2+w3=1; the structural deviation Sd quantifies the difference between the data source structure and the standard template; the conversion logic complexity Tc quantifies the difficulty of data cleaning and conversion by the number of lines of code or logical branches; and the interface call frequency Fc reflects the frequency of data interaction.

[0073] S22. Compare the customized complexity index CCI with the complexity threshold α, and determine the pipeline construction mode based on the latency requirements of the business scenario. The specific contents are as follows:

[0074] If the CCI exceeds the complexity threshold α, or the latency requirement is in seconds, it is considered a highly customized or real-time scenario. The pipeline construction mode will select a customized dynamic adaptation module and prioritize the stream processing mode.

[0075] If the CCI does not exceed the complexity threshold α and the latency requirement is at the minute level or lower, it is determined to be a standard scenario. The pipeline construction mode will use a standardized dynamic adaptation module and prioritize batch processing mode to save costs.

[0076] Furthermore, steps S1 to S2 form the core of intelligent decision-making and planning for the data pipeline, transforming ambiguous business requests into precise, executable architecture blueprints. Specifically, the system first analyzes data source characteristics, including data format, interface type, and data change frequency, as well as business scenario requirements such as latency requirements and data consistency level. Meta-analysis then assesses the differences between data sources and standard libraries, and the compatibility between business latency and system capabilities, ultimately forming a preliminary configuration strategy.

[0077] Subsequently, in order to quantitatively evaluate the task, the system further collects the structural deviation Sd to measure the non-standard degree of the data structure, collects the conversion logic complexity Tc to quantify the difficulty of data cleaning, and records the interface call frequency Fc to reflect the interaction intensity. These parameters are then weighted using the weight coefficients w1, w2, and w3 to calculate a unified customized complexity index CCI.

[0078] Finally, the system compares this CCI with the preset complexity threshold α, and combines it with the business's second-level or minute-level latency requirements to ultimately decide on the pipeline construction mode. That is, when the CCI exceeds α or there is a real-time demand, a customized dynamic adaptation module and stream processing mode are used to ensure performance and flexibility. Otherwise, a standard module and batch processing mode are selected to achieve cost optimization, thereby ensuring the scientific and efficient subsequent pipeline construction.

[0079] S31. After receiving the task request, dynamically assemble the data pipeline. The specific content of the dynamic assembly of the data pipeline is as follows:

[0080] S311. The dynamic adaptation module is a containerized service encapsulated for a specific data processing task and can be independently deployed and called;

[0081] S312. According to the pipeline construction mode, the corresponding standardized or customized DAM is automatically selected from the module library and linked into a complete data processing link in a logical order.

[0082] S32. During pipeline operation, the monitoring cycle is divided into a number of discrete monitoring periods to monitor and record key performance indicators in real time;

[0083] S321, the data processing delay Ld refers to the end-to-end time from data generation to final loading into the target system;

[0084] S322. The resource consumption Rc includes the CPU, memory, and network bandwidth occupied by executing the pipeline task, and the cost-benefit ratio Cb is calculated based on the throughput Tp and the resource consumption Rc.

[0085] S41. Use the long short-term memory network to build the original model, use the historical pipeline operation status set as training data to train the original model, and use the trained model as the pipeline efficiency prediction model to predict the performance trend in the future under the current configuration.

[0086] S42. Calculate and obtain a delay jitter coefficient Jd based on the data processing delay Ld sequence within the continuous monitoring period. The delay jitter coefficient Jd is obtained by the following formula:

[0087]

[0088] It should be noted that this formula is as follows: σ(Ld) represents the standard deviation of the data processing delay Ld during the monitoring period, and μ(Ld) represents the average value of the data processing delay Ld. This coefficient reflects the stability of the data processing delay; the smaller the value, the more stable it is.

[0089] S43. Calculate and obtain a cost-effectiveness ratio Cb based on the throughput Tp and the resource consumption Rc. The cost-effectiveness ratio Cb is obtained by the following formula:

[0090]

[0091] Where k1 and k2 are cost weight coefficients for different resources, CPU and Memory are the normalized CPU and memory consumption values, respectively; these coefficients measure the processing efficiency per unit resource input.

[0092] The trained pipeline performance prediction model is used to normalize the delay jitter coefficient Jd, effective throughput Te, and cost-effectiveness ratio Cb, and then fits and calculates the adaptive performance index AEI. The adaptive performance index AEI is obtained using the following formula:

[0093]

[0094] Where Wt, Wc, and Wj are weight coefficients, representing the importance attached to throughput, cost-effectiveness, and delay jitter, respectively; Te_norm, Cb_norm, and Jd_norm are the normalized values ​​of each indicator.

[0095] S5.1 generates dynamic tuning instructions based on the evaluation results. The specific contents are as follows:

[0096] If the adaptive efficiency index (AEI) falls below the efficiency threshold β, indicating that the current pipeline configuration is inefficient, the system will automatically execute tuning instructions, such as switching from batch processing to stream processing mode, increasing resource allocation, or issuing warnings to R&D personnel, suggesting optimization of related dynamic adaptation modules.

[0097] If the adaptive efficiency index AEI is not lower than the efficiency threshold β, it means that the current pipeline configuration meets the business needs and the current configuration will be maintained and continuously monitored.

[0098] Furthermore, this series of steps forms the core of the data pipeline's execution, monitoring, and closed-loop self-optimization. Its systemic significance lies in endowing the pipeline with self-awareness and autonomous evolution capabilities throughout its lifecycle. Specifically, the system first dynamically selects and links standard or customized containerized dynamic adaptation modules from a module library based on the build model to assemble the data processing chain.

[0099] During pipeline operation, the system monitors key performance indicators in real time, including data processing latency Ld, which measures end-to-end timeliness, throughput Tp, which reflects processing capacity, and resource consumption Rc, which includes CPU and memory consumption.

[0100] The system then uses a long short-term memory (LSTM) prediction model to transform these raw metrics into more insightful features. For example, the system calculates the latency jitter coefficient (Jd), which reflects stability, by calculating the ratio of the standard deviation to the mean of latency (Ld). It also calculates the cost-benefit ratio (Cb), which measures efficiency, by comparing throughput (Tp) to weighted resource costs (k1CPU + k2Memory).

[0101] Ultimately, the model fits the normalized effective throughput Te_norm, cost-effectiveness ratio Cb_norm, and delay jitter coefficient Jd_norm according to their respective weights Wt, Wc, and Wj to generate a comprehensive adaptive efficiency index (AEI). This AEI is then compared with the preset efficiency threshold β. If the AEI is lower than β, dynamic tuning instructions such as switching processing modes or adjusting resources are automatically triggered, thereby achieving continuous, intelligent, and automated assurance of pipeline performance.

[0102] Example 2

[0103] See also Figure 2 , a data pipeline integration system, including the following modules:

[0104] The task parsing and strategy generation module receives and parses data integration task requests, which include data source characteristics, data targets, and business scenario requirements. It then performs meta-analysis on the task requests to generate an initial pipeline configuration strategy.

[0105] The complexity assessment and mode decision module evaluates the non-scaling of the data source and the real-time requirements of the business scenario based on the initial pipeline configuration strategy, calculates and obtains a customized complexity index CCI, and compares the customized complexity index CCI with a preset complexity threshold α to determine the pipeline construction mode;

[0106] The pipeline dynamic assembly and runtime monitoring module dynamically assembles a data pipeline based on a defined pipeline construction mode. The pipeline consists of one or more pluggable dynamic adapter modules and selects batch or stream processing mode to perform data integration tasks. During pipeline operation, key performance indicators (KPIs) are monitored in real time, including data processing latency Ld, throughput Tp, and resource consumption Rc, to construct a pipeline operation status set.

[0107] The pipeline efficiency analysis and prediction module performs time series feature analysis on the pipeline operation status set obtained in step S3 to obtain the delay jitter coefficient Jd, effective throughput Te, and cost-effectiveness ratio Cb in different monitoring periods. It also obtains the adaptive efficiency index AEI by fitting and calculating the pre-trained pipeline efficiency prediction model.

[0108] The closed-loop adaptive tuning module pre-sets an efficiency threshold β and compares and analyzes the efficiency threshold β with the adaptive efficiency index AEI to comprehensively evaluate whether the current data pipeline configuration is optimal, and generates dynamic tuning instructions based on the evaluation results to achieve closed-loop self-optimization of the pipeline.

[0109] While embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions, and variations may be made to these embodiments without departing from the principles and spirit of the invention, and that the scope of the invention is defined by the appended claims and their equivalents.

Claims

1. A data pipeline integration method, characterized in that: The following steps are involved: S1. Receive and parse a data integration task request, which includes data source characteristics, data targets, and business scenario requirements. Perform meta-analysis on the task request to generate an initial pipeline configuration strategy. S2. Based on the initial pipeline configuration strategy, evaluate the non-scaling of the data source and the real-time requirements of the business scenario, calculate and obtain a customized complexity index (CCI), and compare the customized complexity index (CCI) with a preset complexity threshold α to determine a pipeline construction mode. S3. Dynamically assemble a data pipeline based on the determined pipeline construction mode. The pipeline consists of one or more pluggable, dynamically adaptable modules and selects batch or stream processing mode to perform data integration tasks. During pipeline operation, monitor its key performance indicators in real time, including data processing latency Ld, throughput Tp, and resource consumption Rc, to construct a pipeline operation status set. S4. Performing a time series feature analysis on the pipeline operation status set obtained in step S3 to obtain the delay jitter coefficient Jd, effective throughput Te, and cost-effectiveness ratio Cb in different monitoring periods. Using a pre-trained pipeline performance prediction model, the adaptive performance index AEI is obtained through fitting calculation. S5. Pre-set an efficiency threshold β, and compare and analyze the efficiency threshold β with the adaptive efficiency index AEI to comprehensively evaluate whether the current data pipeline configuration is optimal, and generate dynamic tuning instructions based on the evaluation results to achieve closed-loop self-optimization of the pipeline.

2. The data pipeline integration method according to claim 1, characterized in that: S11. Data source characteristics include data format, interface type, and data change frequency; Business scenario requirements include latency requirements, data consistency level, and target applications; S111. Meta-analysis includes analysis of the difference between the data source format and the standard library, and analysis of the matching degree between the business delay requirements and the system's default processing capabilities, in order to generate an initial pipeline configuration strategy including recommended adaptation modules, processing modes, and resource estimates.

3. The data pipeline integration method according to claim 1, characterized in that: S21. Collect the data source's structural deviation Sd, conversion logic complexity Tc, and interface call frequency Fc as calculation parameters, and calculate the customized complexity index CCI using the following formula: What needs to be explained in this formula is that w1, w2, and w3 are weight coefficients of different dimensions, and they satisfy w1+w2+w3=1.

4. The data pipeline integration method according to claim 1, characterized in that: S22. Compare the customized complexity index CCI with the complexity threshold α, and determine the pipeline construction mode based on the latency requirements of the business scenario. The specific contents are as follows: If the CCI exceeds the complexity threshold α, or the latency requirement is in seconds, it is considered a highly customized or real-time scenario. The pipeline construction mode will select a customized dynamic adaptation module and prioritize the stream processing mode. If the CCI does not exceed the complexity threshold α and the latency requirement is at the minute level or lower, it is determined to be a standard scenario. The pipeline construction mode will use a standardized dynamic adaptation module and prioritize batch processing mode to save costs.

5. The data pipeline integration method according to claim 1, characterized in that: S31. After receiving the task request, dynamically assemble the data pipeline. The specific content of the dynamic assembly of the data pipeline is as follows: S311. The dynamic adaptation module is a containerized service encapsulated for a specific data processing task and can be independently deployed and called; S312. According to the pipeline construction mode, the corresponding standardized or customized DAM is automatically selected from the module library and linked into a complete data processing link in a logical order.

6. The data pipeline integration method according to claim 1, characterized in that: S32. During pipeline operation, the monitoring cycle is divided into a number of discrete monitoring periods to monitor and record key performance indicators in real time; S321, the data processing delay Ld refers to the end-to-end time from data generation to final loading into the target system; S322. The resource consumption Rc includes the CPU, memory, and network bandwidth occupied by executing the pipeline task, and the cost-benefit ratio Cb is calculated based on the throughput Tp and the resource consumption Rc.

7. The data pipeline integration method according to claim 1, characterized in that: S41. Use the long short-term memory network to build the original model, use the historical pipeline operation status set as training data to train the original model, and use the trained model as the pipeline efficiency prediction model to predict the performance trend in the future under the current configuration. S42. Calculate and obtain a delay jitter coefficient Jd based on the data processing delay Ld sequence within the continuous monitoring period. The delay jitter coefficient Jd is obtained by the following formula: It should be noted that in this formula: σ(Ld) represents the standard deviation of the data processing delay Ld during the monitoring period, and μ(Ld) represents the average value of the data processing delay Ld.

8. The data pipeline integration method according to claim 1, characterized in that: S43. Calculate and obtain a cost-effectiveness ratio Cb based on the throughput Tp and the resource consumption Rc. The cost-effectiveness ratio Cb is obtained by the following formula: Where k1 and k2 are the cost weight coefficients of different resources, CPU and Memory are the normalized consumption values ​​of CPU and memory, respectively. The trained pipeline performance prediction model is used to normalize the delay jitter coefficient Jd, effective throughput Te, and cost-effectiveness ratio Cb, and then fits and calculates the adaptive performance index AEI. The adaptive performance index AEI is obtained using the following formula: Where Wt, Wc, and Wj are weight coefficients, representing the importance attached to throughput, cost-effectiveness, and delay jitter, respectively; Te_norm, Cb_norm, and Jd_norm are the normalized values ​​of each indicator.

9. The data pipeline integration method according to claim 1, characterized in that: S5.1 generates dynamic tuning instructions based on the evaluation results. The specific contents are as follows: If the adaptive efficiency index (AEI) falls below the efficiency threshold β, indicating that the current pipeline configuration is inefficient, the system will automatically execute tuning instructions, such as switching from batch processing to stream processing mode, increasing resource allocation, or issuing warnings to R&D personnel, suggesting optimization of related dynamic adaptation modules. If the adaptive efficiency index AEI is not lower than the efficiency threshold β, it means that the current pipeline configuration meets the business needs and the current configuration will be maintained and continuously monitored.

10. A data pipeline integration system, according to a data pipeline integration method according to any one of claims 1 to 9, characterized in that: The task parsing and strategy generation module receives and parses data integration task requests, which include data source characteristics, data targets, and business scenario requirements. It then performs meta-analysis on the task requests to generate an initial pipeline configuration strategy. The complexity assessment and mode decision module evaluates the non-scaling of the data source and the real-time requirements of the business scenario based on the initial pipeline configuration strategy, calculates and obtains a customized complexity index CCI, and compares the customized complexity index CCI with a preset complexity threshold α to determine the pipeline construction mode; The pipeline dynamic assembly and runtime monitoring module dynamically assembles a data pipeline based on a defined pipeline construction mode. The pipeline consists of one or more pluggable dynamic adapter modules and selects batch or stream processing mode to perform data integration tasks. During pipeline operation, key performance indicators (KPIs) are monitored in real time, including data processing latency Ld, throughput Tp, and resource consumption Rc, to construct a pipeline operation status set. The pipeline efficiency analysis and prediction module performs time series feature analysis on the pipeline operation status set obtained in step S3 to obtain the delay jitter coefficient Jd, effective throughput Te, and cost-effectiveness ratio Cb in different monitoring periods. It also obtains the adaptive efficiency index AEI by fitting and calculating the pre-trained pipeline efficiency prediction model. The closed-loop adaptive tuning module pre-sets an efficiency threshold β and compares and analyzes the efficiency threshold β with the adaptive efficiency index AEI to comprehensively evaluate whether the current data pipeline configuration is optimal, and generates dynamic tuning instructions based on the evaluation results to achieve closed-loop self-optimization of the pipeline.

Images