A data pipeline integration method and system

Abstract

The application discloses a data pipeline integration method and system, relates to the technical field of data processing, and solves the problems of poor flexibility and high delay of a traditional data pipeline. The application calculates a customized complexity index CCI based on parameters such as a structural deviation degree Sd through a meta-analysis task request, compares the CCI with a threshold value alpha, and then decides and dynamically assembles a standard or customized dynamic adaptation module to efficiently cope with non-standard data sources and complex logic. Meanwhile, the method selects a batch processing or stream processing mode according to delay requirements, monitors key indicators such as a delay Ld in real time, fits characteristics such as a delay jitter coefficient Jd and a cost-benefit ratio Cb into an adaptive performance index AEI, compares the adaptive performance index AEI with a performance threshold value beta, and drives pipeline closed-loop self-optimization, such as automatic switching of processing modes, so that low delay and high adaptability in SaaS operation and maintenance and other scenarios are ensured.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, specifically to a data pipeline integration method and system. Background Technology

[0002] This data integration method, specifically designed for business scenarios involving system development and SaaS operation and maintenance services, centers on building a standardized and reusable data pipeline system. This system efficiently and stably connects and integrates data from heterogeneous data sources such as different business systems, user behavior logs, and third-party application APIs. Through data extraction, cleaning, transformation, and loading, this method converts raw data into valuable information and delivers it uniformly to a data warehouse or analytics platform. Ultimately, it aims to automate and standardize data flow, providing reliable data support for rapid iteration in system development, intelligent operation and maintenance of SaaS services, and business decision-making.

[0003] Based on the characteristics of data pipeline integration methods, the following technical drawbacks often exist in practical applications:

[0004] Insufficient flexibility and difficulty in customization:

[0005] Because of the emphasis on building a "standardized and reusable" pipeline system, this approach can become rigid when dealing with diverse and rapidly changing business needs. For example, when a SaaS customer needs to access a non-standard, specially formatted private data source, or requires unconventional, complex, and personalized data cleansing logic, the standardized pipeline may not be able to directly support it. Forcibly modifying a generic template will destroy its reusability, increase maintenance costs and risks, and lead to long development cycles and high difficulty in customized development.

[0006] The data processing latency is too high, making it difficult to support real-time scenarios.

[0007] To achieve standardization and stability, these data pipelines typically employ a batch processing architecture, such as executing ETL (extract, transform, load) tasks on a scheduled basis (e.g., hourly or daily). This architecture directly leads to the first drawback—poor flexibility. Its most direct consequence is data latency. For example, in SaaS system operations and maintenance, if real-time monitoring of anomalies in critical user operations and immediate alerts are required, an hourly data update frequency is unacceptable. Problem detection and response will be severely delayed, failing to meet the demands of real-time operational monitoring or business risk control scenarios. Summary of the Invention

[0008] To address the shortcomings of existing technologies, this invention provides a data pipeline integration method and system, which solves the technical deficiencies mentioned in the background section.

[0009] To achieve the above objectives, the present invention provides the following technical solution: a data pipeline integration method, comprising the following steps:

[0010] S1. Receive and parse the data integration task request. The task request includes data source characteristics, data objectives, and business scenario requirements. Perform meta-analysis on the task request to generate an initial pipeline configuration strategy.

[0011] S2. Based on the initial pipeline configuration strategy, evaluate the non-standardization of the data source and the real-time requirements of the business scenario, calculate and obtain the customized complexity index CCI, and compare the customized complexity index CCI with the preset complexity threshold α to determine the pipeline construction mode.

[0012] S3. According to the determined pipeline construction mode, dynamically assemble the data pipeline, which consists of one or more pluggable dynamic adaptation modules, and select batch processing or stream processing mode to execute the data integration task; during pipeline operation, monitor its key performance indicators in real time, including data processing latency Ld, throughput Tp and resource consumption Rc, in order to construct a pipeline operation status set.

[0013] S4. By performing time-series feature analysis on the pipeline operating status set obtained in step S3, the delay jitter coefficient Jd, effective throughput Te, and cost-benefit ratio Cb in different monitoring periods are obtained respectively. The adaptive performance index AEI is obtained by fitting and calculating the pre-trained pipeline performance prediction model.

[0014] S5. Pre-set the performance threshold β, and compare and analyze the performance threshold β with the adaptive performance index AEI to comprehensively evaluate whether the current data pipeline configuration is optimal, and generate dynamic tuning instructions based on the evaluation results to achieve closed-loop self-optimization of the pipeline.

[0015] Preferably, S11, the data source characteristics include data format, interface type, and data change frequency;

[0016] Business scenario requirements include latency requirements, data consistency levels, and target applications;

[0017] S111. Meta-analysis includes analysis of the differences between data source formats and standard libraries, and analysis of the matching degree between business latency requirements and system default processing capabilities, in order to generate an initial pipeline configuration strategy that includes recommended adaptation modules, processing modes, and resource estimates.

[0018] Preferably, S21, the structural deviation degree Sd of the collected data source, the transformation logic complexity Tc, and the interface call frequency Fc are used as calculation parameters, and the customized complexity index CCI is calculated using the following formula:

[0019]

[0020] It should be noted that w1, w2, and w3 are weight coefficients for different dimensions, and w1 + w2 + w3 = 1.

[0021] Preferably, in step S22, the customized complexity index (CCI) is compared with the complexity threshold α, and the pipeline construction mode is determined in conjunction with the latency requirements of the business scenario. The specific details are as follows:

[0022] If the CCI exceeds the complexity threshold α, or the latency requirement is in the second range, it is determined to be a highly customized or real-time scenario. The pipeline construction mode will select a customized dynamic adaptation module and prioritize the stream processing mode.

[0023] If the CCI does not exceed the complexity threshold α and the latency requirement is in the minute range or lower, it is judged as a standard scenario. The pipeline construction mode will select a standardized dynamic adaptation module and prioritize the batch processing mode to save costs.

[0024] Preferably, in step S31, after receiving the task request, the data pipeline is dynamically assembled, and the specific content of the dynamic assembly of the data pipeline is as follows:

[0025] S311. The dynamic adaptation module is a containerized service that is encapsulated for specific data processing tasks and can be deployed and invoked independently.

[0026] S312. Based on the pipeline construction mode, automatically select the corresponding standardized or customized DAM from the module library and link them into a complete data processing link in logical order.

[0027] Preferably, S32, during pipeline operation, the monitoring cycle is divided into several discrete monitoring periods to monitor and record key performance indicators in real time;

[0028] S321, the data processing delay Ld refers to the end-to-end time from the generation of data at the source to its final loading into the target system;

[0029] S322, The resource consumption Rc includes the CPU, memory and network bandwidth occupied by the pipeline task, and the cost-benefit ratio Cb is calculated based on the throughput Tp and the resource consumption Rc.

[0030] Preferably, in step S41, a long short-term memory network is used to construct the original model, the historical pipeline running state set is used as training data to train the original model, and the trained model is used as a pipeline performance prediction model to predict the performance trend in the future under the current configuration.

[0031] S42. Based on the data processing delay Ld sequence within the continuous monitoring period, calculate and obtain the delay jitter coefficient Jd, which is obtained by the following formula:

[0032]

[0033] It should be noted that in this formula: σ(Ld) represents the standard deviation of the data processing delay Ld within the monitoring period, and μ(Ld) represents the average value of the data processing delay Ld.

[0034] Preferably, in step S43, the cost-benefit ratio Cb is calculated based on the throughput Tp and resource consumption Rc, and the cost-benefit ratio Cb is obtained by the following formula:

[0035]

[0036] In the formula, k1 and k2 are cost weighting coefficients for different resources, and CPU and Memory are the normalized consumption values ​​of CPU and memory, respectively.

[0037] After normalizing the delay jitter coefficient Jd, effective throughput Te, and cost-effectiveness ratio Cb using a trained pipeline performance prediction model, the adaptive performance index AEI is calculated. The adaptive performance index AEI is obtained using the following formula:

[0038]

[0039] In the formula, Wt, Wc, and Wj are all weighting coefficients, representing the degree of importance attached to throughput, cost-effectiveness, and latency jitter, respectively. , as well as These are the normalized values ​​for each indicator.

[0040] Preferably, S5.1 generates dynamic optimization instructions based on the evaluation results, the specific content of which is as follows:

[0041] If the Adaptive Performance Index (AEI) is lower than the performance threshold β, it indicates that the current pipeline configuration is not performing well. The system will automatically execute tuning instructions, including switching from batch processing to stream processing mode, increasing resource allocation, or issuing warnings to R&D personnel, suggesting optimization of related dynamic adaptation modules.

[0042] If the Adaptive Performance Index (AEI) is not lower than the performance threshold β, it means that the current pipeline configuration meets the business requirements, and the current configuration will be maintained and continuously monitored.

[0043] A data pipeline integration system includes the following modules:

[0044] The task parsing and strategy generation module receives and parses data integration task requests, which include data source characteristics, data objectives, and business scenario requirements. It generates an initial pipeline configuration strategy by performing meta-analysis on the task requests.

[0045] The complexity assessment and pattern decision module, based on the initial pipeline configuration strategy, assesses the non-standard nature of the data source and the real-time requirements of the business scenario, calculates and obtains a customized complexity index (CCI), and compares the customized complexity index CCI with a preset complexity threshold α to determine the pipeline construction pattern.

[0046] The pipeline dynamic assembly and runtime monitoring module dynamically assembles the data pipeline according to the determined pipeline construction mode. The pipeline consists of one or more pluggable dynamic adaptation modules, and selects batch processing or stream processing mode to execute data integration tasks. During pipeline operation, its key performance indicators (KPIs) are monitored in real time, including data processing latency Ld, throughput Tp and resource consumption Rc, in order to construct a pipeline operation status set.

[0047] The pipeline performance analysis and prediction module performs time-series feature analysis on the pipeline operating status set obtained in step S3 to obtain the delay jitter coefficient Jd, effective throughput Te, and cost-benefit ratio Cb in different monitoring periods. It then uses a pre-trained pipeline performance prediction model to fit and calculate the adaptive performance index AEI.

[0048] The closed-loop adaptive tuning module pre-sets a performance threshold β and compares and analyzes the performance threshold β with the adaptive performance index AEI to comprehensively evaluate whether the current data pipeline configuration is optimal. Based on the evaluation results, it generates dynamic tuning instructions to achieve closed-loop self-optimization of the pipeline.

[0049] This invention provides a data pipeline integration method and system. It has the following beneficial effects:

[0050] (1) This data pipeline integration method and system effectively solves the problems of insufficient flexibility and difficulty in customization of traditional data pipelines by introducing meta-analysis of task requests and customized complexity assessment. Specifically, this method first analyzes task requests containing data source characteristics and business scenario requirements through the task parsing and strategy generation module; then, through the complexity assessment and pattern decision module, it calculates the customized complexity index CCI using the structural deviation degree Sd of the data source, the transformation logic complexity Tc, and the interface call frequency Fc, and compares it with the preset complexity threshold α to scientifically decide the pipeline construction mode; finally, through the pipeline dynamic assembly and runtime monitoring module, it automatically selects standardized or customized dynamic adaptation modules from the module library for linking and assembly based on the pattern decision results, thereby efficiently handling non-standard data sources and complex business logic, avoiding the high maintenance costs and risks brought about by modifying general templates, and significantly improving the adaptability and scalability of data pipelines.

[0051] (2) This data pipeline integration method and system successfully overcomes the shortcomings of traditional batch processing architecture, such as high data latency and difficulty in supporting real-time scenarios, by establishing a hybrid processing mode and a closed-loop self-optimization mechanism. Specifically, when determining the pipeline construction mode, this method will choose between batch processing and stream processing modes based on the latency requirements of the business scenario. During pipeline operation, key performance indicators, including data processing latency Ld, throughput Tp, and resource consumption Rc, will be continuously monitored. Through the pipeline performance analysis and prediction module, indicators such as latency jitter coefficient Jd reflecting stability and cost-benefit ratio Cb reflecting efficiency will be calculated. Then, through the pipeline performance prediction model trained by the long short-term memory network, each indicator will be fitted into a comprehensive adaptive performance index AEI. Finally, the closed-loop adaptive optimization module will compare the AEI with the preset performance threshold β. If the performance does not meet the standard, dynamic optimization instructions will be generated, such as automatically switching from batch processing to stream processing mode, thereby ensuring that the data pipeline can dynamically meet the second-level real-time requirements of SaaS operation and maintenance scenarios and ensuring the timeliness of business response. Attached Figure Description

[0052] Figure 1 This is a schematic diagram of the method steps of the present invention;

[0053] Figure 2 This is a schematic diagram of the system framework structure of the present invention. Detailed Implementation

[0054] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0055] Example 1

[0056] Please see Figure 1 This invention provides a data pipeline integration method, comprising the following steps:

[0057] S1. Receive and parse the data integration task request. The task request includes data source characteristics, data objectives, and business scenario requirements. Perform meta-analysis on the task request to generate an initial pipeline configuration strategy.

[0058] S2. Based on the initial pipeline configuration strategy, evaluate the non-standardization of the data source and the real-time requirements of the business scenario, calculate and obtain the customized complexity index CCI, and compare the customized complexity index CCI with the preset complexity threshold α to determine the pipeline construction mode.

[0059] S3. According to the determined pipeline construction mode, dynamically assemble the data pipeline, which consists of one or more pluggable dynamic adaptation modules, and select batch processing or stream processing mode to execute the data integration task; during pipeline operation, monitor its key performance indicators in real time, including data processing latency Ld, throughput Tp and resource consumption Rc, in order to construct a pipeline operation status set.

[0060] S4. By performing time-series feature analysis on the pipeline operating status set obtained in step S3, the delay jitter coefficient Jd, effective throughput Te, and cost-benefit ratio Cb in different monitoring periods are obtained respectively. The adaptive performance index AEI is obtained by fitting and calculating the pre-trained pipeline performance prediction model.

[0061] S5. Pre-set the performance threshold β, and compare and analyze the performance threshold β with the adaptive performance index AEI to comprehensively evaluate whether the current data pipeline configuration is optimal, and generate dynamic tuning instructions based on the evaluation results to achieve closed-loop self-optimization of the pipeline.

[0062] Furthermore, the S1 step enables precise interpretation of data integration task requests, laying an intelligent foundation for subsequent automated processes;

[0063] Then, the customized complexity index CCI is calculated through step S2 and compared with the preset threshold α, thus achieving a scientific quantification of task complexity and a precise decision on the construction mode.

[0064] Step S3 achieves agile pipeline construction and transparent acquisition of operational status by dynamically assembling and real-time monitoring of data processing latency Ld, throughput Tp, and resource consumption Rc.

[0065] Step S4 then uses time-series analysis to refine the original indicators into delay jitter coefficient Jd, effective throughput Te, and cost-effectiveness ratio Cb, and finally fits them into a comprehensive adaptive performance index AEI, completing a deep insight and quantitative assessment of pipeline performance.

[0066] Finally, step S5 compares the AEI with the performance threshold β to drive the generation of dynamic tuning instructions, thus constructing a complete closed-loop self-optimization mechanism. This ensures that the data pipeline can continuously operate in the optimal state, significantly improving the system's automation, intelligence, and resource utilization efficiency.

[0067] S11. Data source characteristics include data format, interface type, and data change frequency;

[0068] Business scenario requirements include latency requirements, data consistency levels, and target applications;

[0069] S111. Meta-analysis includes analysis of the differences between data source formats and standard libraries, and analysis of the matching degree between business latency requirements and system default processing capabilities, in order to generate an initial pipeline configuration strategy that includes recommended adaptation modules, processing modes, and resource estimates.

[0070] S21. Collect the structural deviation Sd of the data source, the transformation logic complexity Tc, and the interface call frequency Fc as calculation parameters, and calculate the customized complexity index CCI using the following formula:

[0071]

[0072] It should be noted that: w1, w2, and w3 are weight coefficients for different dimensions, and w1+w2+w3=1; the structural deviation Sd quantifies the difference between the data source structure and the standard template; the transformation logic complexity Tc quantifies the difficulty of data cleaning and transformation through the number of lines of code or logical branches; and the interface call frequency Fc reflects the frequency of data interaction.

[0073] S22. Compare the customized complexity index CCI with the complexity threshold α, and determine the pipeline construction mode based on the latency requirements of the business scenario. The specific details are as follows:

[0074] If the CCI exceeds the complexity threshold α, or the latency requirement is in the second range, it is determined to be a highly customized or real-time scenario. The pipeline construction mode will select a customized dynamic adaptation module and prioritize the stream processing mode.

[0075] If the CCI does not exceed the complexity threshold α and the latency requirement is in the minute range or lower, it is judged as a standard scenario. The pipeline construction mode will select a standardized dynamic adaptation module and prioritize the batch processing mode to save costs.

[0076] Furthermore, steps S1 to S2 constitute the core of intelligent decision-making and planning for the data pipeline. Their significance lies in transforming ambiguous business requests into precise and executable architectural blueprints. Specifically, the system first analyzes data source characteristics, including data format, interface type, data change frequency, and business scenario requirements such as latency requirements and data consistency levels. Through meta-analysis, it evaluates the differences between the data source and the standard library, as well as the matching degree between business latency and system capabilities, to form a preliminary configuration strategy.

[0077] Subsequently, in order to quantitatively evaluate the task, the system further collected the structural deviation degree Sd to measure the non-standardity of the data structure, collected the transformation logic complexity Tc to quantify the difficulty of data cleaning, and recorded the interface call frequency Fc to reflect the interaction intensity. Then, the weight coefficients w1, w2, and w3 were used to weight these parameters to calculate a unified customized complexity index CCI.

[0078] Finally, the system compares this CCI with the preset complexity threshold α, and combines it with the business's second-level or minute-level latency requirements to ultimately decide on the pipeline construction mode. That is, when the CCI exceeds α or there is a real-time requirement, a customized dynamic adaptation module and stream processing mode are used to ensure performance and flexibility; otherwise, a standard module and batch processing mode are selected to achieve cost optimization, thereby ensuring the scientific and efficient construction of the subsequent pipeline.

[0079] S31. Upon receiving a task request, dynamically assemble the data pipeline. The specific details of the dynamic data pipeline assembly are as follows:

[0080] S311. The dynamic adaptation module is a containerized service that is encapsulated for specific data processing tasks and can be deployed and invoked independently.

[0081] S312. Based on the pipeline construction mode, automatically select the corresponding standardized or customized DAM from the module library and link them into a complete data processing link in logical order.

[0082] S32. During pipeline operation, the monitoring cycle is divided into several discrete monitoring periods to monitor and record key performance indicators in real time.

[0083] S321, the data processing delay Ld refers to the end-to-end time from the generation of data at the source to its final loading into the target system;

[0084] S322. The resource consumption Rc includes the CPU, memory and network bandwidth occupied by the pipeline task, and the cost-benefit ratio Cb is calculated based on the throughput Tp and the resource consumption Rc.

[0085] S41. Use a long short-term memory network to build the original model, use the set of historical pipeline running states as training data to train the original model, and use the trained model as a pipeline performance prediction model to predict the performance trend in the future under the current configuration.

[0086] S42. Based on the data processing delay Ld sequence within the continuous monitoring period, calculate and obtain the delay jitter coefficient Jd, which is obtained by the following formula:

[0087]

[0088] It should be noted that in this formula: σ(Ld) represents the standard deviation of the data processing delay Ld within the monitoring period, and μ(Ld) represents the average value of the data processing delay Ld; this coefficient reflects the stability of the data processing delay, and the smaller the value, the more stable it is.

[0089] S43. Based on the throughput Tp and resource consumption Rc, calculate the cost-benefit ratio Cb, which is obtained by the following formula:

[0090]

[0091] In the formula, k1 and k2 are cost weighting coefficients for different resources, and CPU and Memory are the normalized consumption values ​​of CPU and memory, respectively; this coefficient measures the processing efficiency brought about by a unit of resource input.

[0092] After normalizing the delay jitter coefficient Jd, effective throughput Te, and cost-effectiveness ratio Cb using a trained pipeline performance prediction model, the adaptive performance index AEI is calculated. The adaptive performance index AEI is obtained using the following formula:

[0093]

[0094] In the formula, Wt, Wc, and Wj are all weighting coefficients, representing the degree of importance attached to throughput, cost-effectiveness, and latency jitter, respectively. , as well as These are the normalized values ​​for each indicator.

[0095] S5.1 Generates dynamic optimization instructions based on the evaluation results, the specific content of which is as follows:

[0096] If the Adaptive Performance Index (AEI) is lower than the performance threshold β, it indicates that the current pipeline configuration is not performing well. The system will automatically execute tuning instructions, including switching from batch processing to stream processing mode, increasing resource allocation, or issuing warnings to R&D personnel, suggesting optimization of related dynamic adaptation modules.

[0097] If the Adaptive Performance Index (AEI) is not lower than the performance threshold β, it means that the current pipeline configuration meets the business requirements, and the current configuration will be maintained and continuously monitored.

[0098] Furthermore, the series of steps constitute the core of the data pipeline's execution, monitoring, and closed-loop self-optimization. Its systemic significance lies in endowing the pipeline with self-awareness and autonomous evolution capabilities throughout its lifecycle. Specifically, the system first dynamically selects and links standard or customized containerized dynamic adaptation modules from the module library to assemble the data processing link based on the construction mode.

[0099] During pipeline operation, the system monitors key performance indicators in real time, including data processing latency Ld, which measures end-to-end timeliness; throughput Tp, which reflects processing capacity; and resource consumption Rc, which includes CPU and memory consumption.

[0100] Subsequently, the system uses a Long Short-Term Memory (LSTM) network prediction model to refine these original indicators into more insightful features. For example, it obtains the latency jitter coefficient Jd, which reflects stability, by calculating the ratio of the standard deviation to the mean of latency Ld, and calculates the cost-benefit ratio Cb, which measures efficiency, by calculating the ratio of throughput Tp to the weighted resource cost (k1CPU+k2Memory).

[0101] Finally, the model fits the normalized effective throughput Te_norm, cost-effectiveness ratio Cb_norm, and latency jitter coefficient Jd_norm according to their respective weights Wt, Wc, and Wj to generate a comprehensive adaptive performance index AEI. This AEI is then compared with a preset performance threshold β. If the AEI is lower than β, dynamic optimization commands such as switching processing modes or adjusting resources are automatically triggered, thereby achieving continuous, intelligent, and automated assurance of pipeline performance.

[0102] Example 2

[0103] Please see Figure 2 A data pipeline integration system includes the following modules:

[0104] The task parsing and strategy generation module receives and parses data integration task requests, which include data source characteristics, data objectives, and business scenario requirements. It generates an initial pipeline configuration strategy by performing meta-analysis on the task requests.

[0105] The complexity assessment and pattern decision module, based on the initial pipeline configuration strategy, assesses the non-standard nature of the data source and the real-time requirements of the business scenario, calculates and obtains a customized complexity index (CCI), and compares the customized complexity index CCI with a preset complexity threshold α to determine the pipeline construction pattern.

[0106] The pipeline dynamic assembly and runtime monitoring module dynamically assembles the data pipeline according to the determined pipeline construction mode. The pipeline consists of one or more pluggable dynamic adaptation modules, and selects batch processing or stream processing mode to execute data integration tasks. During pipeline operation, its key performance indicators (KPIs) are monitored in real time, including data processing latency Ld, throughput Tp and resource consumption Rc, in order to construct a pipeline operation status set.

[0107] The pipeline performance analysis and prediction module performs time-series feature analysis on the pipeline operating status set obtained in step S3 to obtain the delay jitter coefficient Jd, effective throughput Te, and cost-benefit ratio Cb in different monitoring periods. It then uses a pre-trained pipeline performance prediction model to fit and calculate the adaptive performance index AEI.

[0108] The closed-loop adaptive tuning module pre-sets a performance threshold β and compares and analyzes the performance threshold β with the adaptive performance index AEI to comprehensively evaluate whether the current data pipeline configuration is optimal. Based on the evaluation results, it generates dynamic tuning instructions to achieve closed-loop self-optimization of the pipeline.

[0109] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A data pipeline integration method, characterized in that, Includes the following steps: S1. Receive and parse the data integration task request. The task request includes data source characteristics, data objectives, and business scenario requirements. Perform meta-analysis on the task request to generate an initial pipeline configuration strategy. S2. Based on the initial pipeline configuration strategy, assess the non-standardization of the data source and the real-time requirements of the business scenario, calculate and obtain the customized complexity index (CCI), and compare the customized complexity index CCI with a preset complexity threshold α to determine the pipeline construction mode; wherein, S2 includes S21, collecting the structural deviation degree Sd, transformation logic complexity Tc, and interface call frequency Fc of the data source as calculation parameters, and calculating the customized complexity index CCI using the following formula: It should be noted that w1, w2, and w3 are weight coefficients for different dimensions, and w1 + w2 + w3 = 1. S3. According to the determined pipeline construction mode, dynamically assemble the data pipeline, which consists of one or more pluggable dynamic adaptation modules, and select batch processing or stream processing mode to execute the data integration task; during pipeline operation, monitor its key performance indicators in real time, including data processing latency Ld, throughput Tp and resource consumption Rc, in order to construct a pipeline operation status set. S4. By performing time-series feature analysis on the pipeline operating status set obtained in step S3, the delay jitter coefficient Jd, effective throughput Te, and cost-effectiveness ratio Cb are obtained for different monitoring periods. An adaptive performance index AEI is then calculated using a pre-trained pipeline performance prediction model. S4 includes S43, where the cost-effectiveness ratio Cb is calculated based on throughput Tp and resource consumption Rc. The cost-effectiveness ratio Cb is obtained using the following formula: In the formula, k1 and k2 are cost weighting coefficients for different resources, and CPU and Memory are the normalized consumption values ​​of CPU and memory, respectively. After normalizing the delay jitter coefficient Jd, effective throughput Te, and cost-effectiveness ratio Cb using a trained pipeline performance prediction model, the adaptive performance index AEI is calculated. The adaptive performance index AEI is obtained using the following formula: In the formula, Wt, Wc, and Wj are all weighting coefficients, representing the degree of importance attached to throughput, cost-effectiveness, and latency jitter, respectively. , as well as These are the normalized values ​​for each indicator; S5. Pre-set the performance threshold β, and compare and analyze the performance threshold β with the adaptive performance index AEI to comprehensively evaluate whether the current data pipeline configuration is optimal, and generate dynamic tuning instructions based on the evaluation results to achieve closed-loop self-optimization of the pipeline.

2. The data pipeline integration method according to claim 1, characterized in that: S11. Data source characteristics include data format, interface type, and data change frequency; Business scenario requirements include latency requirements, data consistency levels, and target applications; S111. Meta-analysis includes analysis of the differences between data source formats and standard libraries, and analysis of the matching degree between business latency requirements and system default processing capabilities, in order to generate an initial pipeline configuration strategy that includes recommended adaptation modules, processing modes, and resource estimates.

3. The data pipeline integration method according to claim 1, characterized in that: S22. Compare the customized complexity index CCI with the complexity threshold α, and determine the pipeline construction mode based on the latency requirements of the business scenario. The specific details are as follows: If the CCI exceeds the complexity threshold α, or the latency requirement is in the second range, it is determined to be a highly customized or real-time scenario. The pipeline construction mode will select a customized dynamic adaptation module and prioritize the stream processing mode. If the CCI does not exceed the complexity threshold α and the latency requirement is in the minute range or lower, it is judged as a standard scenario. The pipeline construction mode will select a standardized dynamic adaptation module and prioritize the batch processing mode to save costs.

4. The data pipeline integration method according to claim 1, characterized in that: S31. Upon receiving a task request, dynamically assemble the data pipeline. The specific details of the dynamic data pipeline assembly are as follows: S311. The dynamic adaptation module is a containerized service that is encapsulated for specific data processing tasks and can be deployed and invoked independently. S312. Based on the pipeline construction mode, automatically select the corresponding standardized or customized DAM from the module library and link them into a complete data processing link in logical order.

5. The data pipeline integration method according to claim 1, characterized in that: S32. During pipeline operation, the monitoring cycle is divided into several discrete monitoring periods to monitor and record key performance indicators in real time. S321, the data processing delay Ld refers to the end-to-end time from the generation of data at the source to its final loading into the target system; S322, The resource consumption Rc includes the CPU, memory and network bandwidth occupied by the pipeline task, and the cost-benefit ratio Cb is calculated based on the throughput Tp and the resource consumption Rc.

6. The data pipeline integration method according to claim 1, characterized in that: S41. Use a long short-term memory network to build the original model, use the set of historical pipeline running states as training data to train the original model, and use the trained model as a pipeline performance prediction model to predict the performance trend in the future under the current configuration. S42. Based on the data processing delay Ld sequence within the continuous monitoring period, calculate and obtain the delay jitter coefficient Jd, which is obtained by the following formula: It should be noted that in this formula: σ(Ld) represents the standard deviation of the data processing delay Ld within the monitoring period, and μ(Ld) represents the average value of the data processing delay Ld.

7. The data pipeline integration method according to claim 1, characterized in that: S5.1 Generates dynamic optimization instructions based on the evaluation results, the specific content of which is as follows: If the Adaptive Performance Index (AEI) is lower than the performance threshold β, it indicates that the current pipeline configuration is not performing well. The system will automatically execute tuning instructions, including switching from batch processing to stream processing mode, increasing resource allocation, or issuing warnings to R&D personnel, suggesting optimization of related dynamic adaptation modules. If the Adaptive Performance Index (AEI) is not lower than the performance threshold β, it means that the current pipeline configuration meets the business requirements, and the current configuration will be maintained and continuously monitored.

8. A data pipeline integration system, comprising a data pipeline integration method according to any one of claims 1-7, characterized in that: The task parsing and strategy generation module receives and parses data integration task requests, which include data source characteristics, data objectives, and business scenario requirements. It generates an initial pipeline configuration strategy by performing meta-analysis on the task requests. The complexity assessment and pattern decision module, based on the initial pipeline configuration strategy, assesses the non-standard nature of the data source and the real-time requirements of the business scenario, calculates and obtains a customized complexity index (CCI), and compares the customized complexity index CCI with a preset complexity threshold α to determine the pipeline construction pattern. The pipeline dynamic assembly and runtime monitoring module dynamically assembles the data pipeline according to the determined pipeline construction mode. The pipeline consists of one or more pluggable dynamic adaptation modules, and selects batch processing or stream processing mode to execute data integration tasks. During pipeline operation, its key performance indicators (KPIs) are monitored in real time, including data processing latency Ld, throughput Tp and resource consumption Rc, in order to construct a pipeline operation status set. The pipeline performance analysis and prediction module performs time-series feature analysis on the pipeline operating status set obtained in step S3 to obtain the delay jitter coefficient Jd, effective throughput Te, and cost-benefit ratio Cb in different monitoring periods. It then uses a pre-trained pipeline performance prediction model to fit and calculate the adaptive performance index AEI. The closed-loop adaptive tuning module pre-sets a performance threshold β and compares and analyzes the performance threshold β with the adaptive performance index AEI to comprehensively evaluate whether the current data pipeline configuration is optimal. Based on the evaluation results, it generates dynamic tuning instructions to achieve closed-loop self-optimization of the pipeline.

Images