Stream and batch adaptive fusion processing method and system for heterogeneous data sources

Abstract

This application relates to the field of data processing technology and discloses a method and system for adaptive fusion processing of streaming and batch data sources for heterogeneous data sources. The method includes: dividing the input data stream into data blocks; extracting arrival rate, second-order change, and disorder degree, and weighted fusion to obtain a turbulence index; mapping the data blocks to obtain processing intent labels using a decision matrix and attaching them to the data blocks to obtain labeled data blocks; reading the execution mode routing to the corresponding engine, and having a fusion coordinator drive cross-engine collaboration to obtain calculation results and actual execution metric data; and after fine-tuning the time-series prediction model online using a confidence-weighted loss function, updating the gradient contribution ratio inversely to the weighted fusion weight coefficients. This application improves the real-time performance and adaptive accuracy of processing mode decisions in heterogeneous data source scenarios.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a method and system for adaptive fusion processing of streaming and batch data sources for heterogeneous data sources. Background Technology

[0002] With the development of big data technology, enterprise data environments exhibit typical heterogeneous characteristics. Data sources cover a variety of heterogeneous data sources, including relational databases, message queues, log files, IoT sensors, and API interfaces. These data sources differ significantly in terms of protocols, formats, throughput, and timeliness. Business scenarios require both low-latency real-time computing and high-throughput offline analysis. Existing technologies mainly address these needs through three approaches: Lambda architecture, Kappa architecture, and split systems. The Lambda architecture maintains two sets of code for both the stream processing layer and the batch processing layer and ultimately merges the results. The Kappa architecture replaces both the stream and batch layers with a unified stream processing layer. The split system independently deploys stream processing clusters and batch processing clusters and coordinates data flow through manual scheduling.

[0003] However, the aforementioned existing technologies all have significant shortcomings. The Lambda architecture requires maintaining two sets of code for the same business logic, resulting in high development and maintenance costs and introducing consistency risks when merging results. The Kappa architecture incurs high costs for reprocessing historical data and lacks sufficient support for complex analysis scenarios. The separate system lacks a unified metadata management and resource coordination mechanism, forming data silos and computing silos between the two clusters, and resources cannot be shared elastically. The more fundamental defect is that the above solutions all rely on a centralized decision engine to passively perceive data characteristics through external polling and then uniformly issue processing modes. The data blocks themselves do not carry any processing intent information, and there is a communication waiting delay in the decision chain. A decision vacuum window is formed between the sudden change in data characteristics and the completion of mode switching by the decision engine, resulting in data backlog or resource waste in high-turbulence scenarios for heterogeneous data sources. Summary of the Invention

[0004] This application provides a streaming and batch adaptive fusion processing method and system for heterogeneous data sources, which solves the problems of decision vacuum window caused by centralized passive perception decision architecture and systematic inaccuracy of label prediction in new turbulent scenarios in existing streaming and batch fusion systems, and improves the real-time performance and adaptive accuracy of processing mode decision in heterogeneous data source scenarios.

[0005] Firstly, this application provides a method for adaptive fusion processing of streaming and batch data sources for heterogeneous data sources, the method comprising: Step S1: Divide the input data stream from the heterogeneous data source into several data blocks according to a preset sampling period. Extract the arrival rate, the second-order change of the arrival rate, and the disorder degree from each data block. After weighted fusion processing, obtain the turbulence index corresponding to each data block. Step S2: For each data block, based on the turbulence index and service level agreement parameters corresponding to the data block, a processing intent label carrying the execution mode, execution parameters and confidence level is obtained through decision matrix mapping. The processing intent label is attached to the corresponding data block to obtain the labeled data block corresponding to each data block. Step S3: Read the execution mode in each of the labeled data blocks, route each of the labeled data blocks to the corresponding stream computing engine or batch computing engine, and have the fusion coordinator perform cross-engine collaborative processing between the stream computing engine and the batch computing engine with the execution parameters as the driving force to obtain the calculation results and actual execution metric data. Step S4: Based on the calculation results, the actual execution metric data, and the confidence score, the time series prediction model is fine-tuned online using the confidence-weighted loss function. The gradient contribution ratio of each feature after fine-tuning is then updated inversely to the weight coefficients of the weighted fusion processing in step S1 to obtain the updated turbulence index calculation weights. The confidence-weighted loss function is a cross-entropy loss function that uses the confidence score as the weight of each training sample. The calculation method is as follows: For each training sample, the actual processing mode corresponding to the calculation result is encoded as a one-hot vector as a label. The cross-entropy is calculated between the label and the probability distribution vector composed of the three prediction probabilities of the batch processing mode, micro-batch processing mode, and streaming processing mode output by the time series prediction model. Then, the cross-entropy is multiplied by the confidence score corresponding to the sample. The weighted cross-entropy of all training samples is summed to obtain the weighted loss value.

[0006] Secondly, this application provides a streaming and batch adaptive fusion processing system for heterogeneous data sources, the streaming and batch adaptive fusion processing system for heterogeneous data sources comprising: The segmentation module is used to segment the input data stream of the heterogeneous data source into several data blocks according to a preset sampling period, extract the arrival rate, the second-order change of the arrival rate and the disorder degree of each data block, and obtain the turbulence index corresponding to each data block through weighted fusion processing. The mapping module is used to, for each data block, obtain a processing intent label carrying the execution mode, execution parameters and confidence level through decision matrix mapping processing based on the turbulence index and service level agreement parameters corresponding to the data block, and attach the processing intent label to the corresponding data block to obtain the labeled data block corresponding to each data block. The processing module is used to read the execution mode in each of the labeled data blocks, route each of the labeled data blocks to the corresponding stream computing engine or batch computing engine, and use the execution parameters as the driving force to perform cross-engine collaborative processing on the stream computing engine and the batch computing engine to obtain the calculation results and actual execution metric data. The update module is used to perform online fine-tuning of the time series prediction model based on the calculation results, the actual execution metric data, and the confidence score, using a confidence-weighted loss function. The gradient contribution ratio of each feature after fine-tuning is then updated inversely to the weight coefficients of the weighted fusion processing in step S1, resulting in updated weights for calculating the turbulence index. The confidence-weighted loss function is a cross-entropy loss function that uses the confidence score as the weight of each training sample. The calculation method is as follows: for each training sample, the actual processing mode corresponding to the calculation result is encoded as a one-hot vector as a label. This label is then cross-entropy calculated between the label and the probability distribution vector formed by the three prediction probabilities of the batch processing mode, micro-batch processing mode, and streaming processing mode output by the time series prediction model. This cross-entropy is then multiplied by the confidence score corresponding to the sample, and the weighted cross-entropy of all training samples is summed to obtain the weighted loss value.

[0007] The technical solution provided in this application synchronously extracts the second-order change and disorder degree of arrival rate during the data sharding stage and calculates the turbulence index through weighted fusion. This allows the system to predict the processing mode before data characteristics undergo abrupt changes. Compared with the existing technology where the centralized decision engine relies on external polling to passively perceive data characteristics and then uniformly issue processing modes, this eliminates the decision vacuum window caused by communication waiting in the decision chain. On this basis, the execution mode, execution parameters, and confidence are encapsulated as processing intent tags and attached to the data blocks. The engine router can directly read the execution mode field in the tag to complete the routing without sending query requests to any external scheduling components. This realizes the transfer of the processing mode decision subject from the centralized engine to the data block itself. The fusion coordinator uses the execution parameters as the driver to perform cross-engine collaborative processing between the stream computing engine and the batch computing engine. The static dimension table generated by the batch computing engine is injected into the memory of the stream computing engine in the form of a side stream. When the stream computing engine detects continuous backlog, it automatically triggers the historical backtracking batch processing task. This realizes deep collaboration between the stream and batch processing modes within the same execution framework and avoids the data silos and resource idleness problems caused by the two independent clusters in the separate system.

[0008] Furthermore, the confidence-weighted loss function integrates the latency achievement status and resource consumption deviation in the actual execution measurement data into a confidence score. The confidence score is used as the weight of the training samples to drive the online fine-tuning of the time series prediction model. This allows samples with inaccurate label predictions to contribute a larger gradient during backpropagation, focusing on correcting the model's predictive ability for inaccurate scenarios. After fine-tuning, the gradient contribution ratio of each feature is updated inversely to the weight coefficients of the weighted fusion processing in step S1. This allows the perception logic of the feature extraction layer to continuously self-correct with the feedback of execution results, forming a complete closed loop from data perception to label generation to execution result backtracking and verification. This enables the system to have continuous adaptive learning capabilities for novel turbulent scenarios that have never appeared before. It solves the fundamental defect of existing online learning methods that only update model parameters and cannot correct the label generation capability from the source of the feature extraction layer. The introduction of algorithm features in this solution is not an isolated model stack, but is deeply integrated with the specific scenario of heterogeneous data source stream-batch fusion processing. Through the confidence score as an intermediate quantity, the business execution results and feature weight coefficients are formed into a quantifiable causal chain, demonstrating the substantial support of algorithm features for the technical contribution of this solution. Attached Figure Description

[0009] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0010] Figure 1 This is a schematic diagram of an embodiment of the adaptive fusion processing method for heterogeneous data sources in this application. Figure 2 This is a schematic diagram illustrating how the confidence score changes with the number of online fine-tuning iterations in an embodiment of this application. Detailed Implementation

[0011] This application provides a streaming and batch adaptive fusion processing method and system for heterogeneous data sources. The terms "first," "second," "third," "fourth," etc. (if present) in the specification, claims, and accompanying drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments described herein can be implemented in a sequence other than that illustrated or described herein. Furthermore, the terms "comprising" or "having" and any variations thereof are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that includes a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0012] For ease of understanding, the specific process of the embodiments of this application is described below. Please refer to [link / reference]. Figure 1 One embodiment of the adaptive fusion processing method for heterogeneous data sources in this application includes: Step S1: Divide the input data stream from the heterogeneous data source into several data blocks according to a preset sampling period. Extract the arrival rate, the second-order change of the arrival rate, and the disorder degree from each data block. After weighted fusion processing, obtain the turbulence index corresponding to each data block. Heterogeneous data sources refer to various data sources with different protocols, formats, throughputs, and timeliness, including relational databases based on the JDBC protocol, streaming data sources based on the Kafka message queue protocol, log data sources based on the file system protocol, API data sources based on the HTTP protocol, and IoT sensor data sources based on the MQTT protocol. The data blocks formed by segmenting the input data stream according to a preset sampling period are the basic processing units for all subsequent feature extraction and label generation. The second-order change in arrival rate is the result of subtracting the first-order change from the arrival rates of adjacent sampling periods, and then subtracting the first-order change from the arrival rates of adjacent sampling periods again; it reflects the acceleration of the data stream rate change rather than the rate itself. Out-of-order tolerance refers to the ratio of the number of records in a data block whose event timestamps are earlier than the current processing time than a preset out-of-order tolerance threshold to the total number of records in that data block, with a value ranging from zero to one. After the above three indicators are processed by maximum and minimum normalization, they are multiplied by their respective weight coefficients and then summed to obtain a turbulence index with a value range of 0 to 1. The sum of the three weight coefficients is 1. The initial value of the weight coefficients is obtained from the statistics of historical execution data and is continuously updated by the gradient contribution ratio in step S4.

[0013] Step S2: For each data block, based on the turbulence index and service level agreement parameters corresponding to the data block, a processing intent label carrying the execution mode, execution parameters and confidence level is obtained through decision matrix mapping. The processing intent label is then attached to the corresponding data block to obtain the labeled data block for each data block. Specifically, the decision matrix refers to a table that determines the correspondence between the execution mode, execution parameters, and confidence level outputs after dividing the turbulence index into low-turbulence, medium-turbulence, and high-turbulence intervals according to a preset grading threshold, and combining the maximum allowable processing latency and result accuracy semantic requirements in the service level agreement parameters, according to a pre-defined mapping rule. The low-turbulence interval corresponds to batch processing mode, the medium-turbulence interval to micro-batch processing mode, and the high-turbulence interval to streaming processing mode. Execution parameters include window size, batch interval, and parallelism, each calculated according to a preset proportional relationship based on the maximum allowable processing latency parameter. The confidence level is the probability value corresponding to the maximum predicted probability of the processing mode output by the time-series prediction model for the current turbulence index output. The processing intent tag encapsulates the above execution mode, execution parameters, and confidence level into a key-value pair structure, attaching it to the corresponding data block as a metadata header field. The tag and data block maintain a binding relationship throughout all subsequent processing stages.

[0014] Step S3: Read the execution mode in each labeled data block, and route each labeled data block to the corresponding stream computing engine or batch computing engine. The fusion coordinator uses the execution parameters as the driving force to perform cross-engine collaborative processing between the stream computing engine and the batch computing engine to obtain the calculation results and actual execution metric data. The actual execution metrics include four quantitative indicators: actual end-to-end processing latency (the difference between the timestamp of the data block entering the router and the timestamp of the corresponding calculation result being written to the output storage); actual CPU resource consumption (the sum of the core times of all execution operators during the processing of the data block); actual peak memory consumption (the maximum memory usage of all execution operators during the processing); and latency compliance indicator (the Boolean judgment result of whether the actual end-to-end processing latency does not exceed the maximum allowable processing latency parameter in the processing intent label). The confidence score is obtained by weighting the latency compliance indicator and the resource consumption deviation according to their respective weight coefficients. The resource consumption deviation is the result of dividing the difference between the actual resource consumption and the preset resource consumption benchmark by the preset resource consumption benchmark, taking the absolute value, and then subtracting by one. The confidence score ranges from zero to one, with a lower confidence score indicating a greater deviation between the predicted and actual execution results of the processing intent label.

[0015] Step S4: Based on the calculation results, actual execution metrics, and confidence scores, the time series prediction model is fine-tuned online using the confidence-weighted loss function. The gradient contribution ratio of each feature after fine-tuning is then updated inversely to the weight coefficients of the weighted fusion processing in Step S1 to obtain the updated weights for calculating the turbulence index. The confidence-weighted loss function is a cross-entropy loss function that uses the confidence score as the weight of each training sample. The calculation method is as follows: For each training sample, the actual processing mode corresponding to the calculation result is encoded as a one-hot vector as a label. The cross-entropy is calculated between the label and the probability distribution vector composed of the three prediction probabilities of the batch processing mode, micro-batch processing mode, and streaming processing mode output by the time series prediction model. This cross-entropy is then multiplied by the confidence score corresponding to the sample. The weighted cross-entropy of all training samples is summed to obtain the weighted loss value.

[0016] Specifically, the confidence-weighted loss function uses the confidence score as the weight of each training sample. The cross-entropy loss between the predicted probability of the processing mode output by the time-series prediction model and the one-hot encoded label of the actual executed processing mode is summed according to the sample weights. Samples with lower confidence scores contribute a larger gradient value to the loss function, driving the time-series prediction model parameters to focus on correcting the scenarios corresponding to the inaccurate samples during backpropagation. After fine-tuning, the absolute gradient values ​​of the arrival rate, the second-order change of the arrival rate, and the disorder degree are calculated with respect to the loss function. The absolute gradient value of each feature is divided by the sum of the absolute gradient values ​​of the three features to obtain the gradient contribution ratio of each feature. This ratio directly covers the weight coefficients corresponding to the three indicators in step S1, so that the calculation of the turbulence index of subsequent data blocks automatically adopts the updated weight coefficients.

[0017] In one specific embodiment, step S1 includes: The input data stream from the heterogeneous data source is divided into several data blocks according to a preset sampling period. The total number of data records arriving within the preset sampling period is counted for each data block. Based on the ratio of the total number of data records to the preset sampling period, the arrival rate of each data block is obtained. The first-order change in arrival rate is obtained by subtracting the arrival rates of adjacent sampling periods, and the second-order change in arrival rate is obtained by subtracting the first-order change in arrival rate of adjacent sampling periods. The disorder degree of a data block is obtained by comparing the number of data records in each data block whose event timestamps are earlier than the current processing time than the preset out-of-order tolerance threshold with the total number of data records. After the arrival rate, the second-order change of arrival rate, and the disorder degree are processed by max-min normalization, they are weighted and summed according to their respective weight coefficients to obtain the turbulence index corresponding to each data block.

[0018] Specifically, the preset sampling period refers to a fixed time interval configured during system initialization, used to divide the continuously input data stream into discrete data blocks. Each data block contains only the data records arriving within that sampling period. The total number of data records refers to the total number of data records received from the heterogeneous data source within a preset sampling period. The arrival rate is calculated by dividing the total number of data records by the duration of the preset sampling period, in units of records per second. The first-order change in arrival rate is obtained by subtracting the arrival rate of the previous sampling period from the arrival rate of the current sampling period, reflecting the absolute change in rate between adjacent periods. The second-order change in arrival rate is obtained by subtracting the first-order change in arrival rate of the previous sampling period from the first-order change in arrival rate of the current sampling period, reflecting the trend of rate change itself. The preset out-of-order tolerance threshold is the upper limit of time deviation configured during system initialization, in milliseconds. Data records whose event timestamp differs from the current processing time by more than this threshold are judged as out-of-order records. The out-of-order degree is calculated by dividing the number of out-of-order records by the total number of data records, with a value ranging from 0 to 1.

[0019] Max-min normalization refers to a linear transformation that maps the arrival rate, the second-order change of the arrival rate, and the disorder level to a range of zero to one. Specifically, the calculation involves subtracting the minimum value of the indicator within a historical sliding window from the current value, then dividing by the difference between the maximum and minimum values ​​within that historical sliding window. The length of the historical sliding window is the preset number of sampling periods. After normalization, each of the three indicators is multiplied by its corresponding weight coefficient and then summed. All weight coefficients are real numbers greater than zero, and the sum of the three weight coefficients is 1. This results in a turbulence index ranging from 0 to 1. A turbulence index closer to 1 indicates a highly turbulent state with high rate, large acceleration, and high disorder; a turbulence index closer to zero indicates a stable, low-speed state.

[0020] In one specific embodiment, in step S2, based on the turbulence index and service level agreement parameters corresponding to the data block, a processing intent label carrying the execution mode, execution parameters, and confidence level is obtained through decision matrix mapping, including: The turbulence index is divided into low turbulence, medium turbulence and high turbulence intervals according to a preset grading threshold. The low turbulence interval corresponds to the batch processing mode, the medium turbulence interval corresponds to the micro-batch processing mode, and the high turbulence interval corresponds to the streaming processing mode, so as to obtain the execution mode corresponding to each data block. Based on the maximum allowable processing delay and result accuracy semantic requirements in the execution mode and business service level agreement parameters, the execution parameters corresponding to each data block are obtained through decision matrix mapping. The predicted probability output by the time series prediction model for the current turbulence index is used as the confidence level for each data block. The execution mode, execution parameters, and confidence level are encapsulated as processing intent tags and attached to the corresponding data blocks to obtain the tagged data blocks for each data block.

[0021] Specifically, the preset grading thresholds refer to two boundary values ​​configured during system initialization, dividing the turbulence index's range of zero to one into three intervals: the interval where the turbulence index is below the first boundary value is the low-turbulence interval, corresponding to batch processing mode; the interval where the turbulence index is between the first and second boundary values ​​is the medium-turbulence interval, corresponding to micro-batch processing mode; and the interval where the turbulence index is above the second boundary value is the high-turbulence interval, corresponding to streaming processing mode. The decision matrix is ​​a mapping table constructed with the execution mode as the row index and the result accuracy semantic requirement as the column index. Each cell in the table records the calculation rules for the execution parameters under the corresponding combination. The execution parameters include three items: window size, batch interval, and parallelism. The window size is calculated by multiplying the maximum allowable processing delay by a preset proportional coefficient; the batch interval is calculated by multiplying the maximum allowable processing delay by another preset proportional coefficient; and the parallelism is calculated by dividing the arrival rate by a preset single-parallelism processing capacity benchmark value and rounding up. The result accuracy semantic requirement is set to exactly once or at least once, with different values ​​corresponding to different rows of the decision matrix in terms of execution parameter calculation rules.

[0022] The time-series prediction model is a model trained on historical data, taking the historical time-series sequence of turbulence index as input and outputting the prediction probabilities corresponding to batch processing mode, micro-batch processing mode, and streaming processing mode, respectively. The sum of the three prediction probabilities is 1. The confidence score is the probability value of the one corresponding to the current execution mode among the three prediction probabilities, reflecting the degree of certainty that the time-series prediction model should adopt the execution mode for the current data block, with a value ranging from 0 to 1. The processing intent tag encapsulates the execution mode, execution parameters, and confidence score in a key-value pair structure and writes it as a metadata header field to the corresponding data block. The tag field and the data block content remain bound throughout the subsequent routing, execution, and backtracking verification stages, without separation. The engine router completes the routing decision by directly reading the execution mode field in the tag, without sending query requests to any external scheduling components.

[0023] In one specific embodiment, in step S3, the fusion coordinator, driven by execution parameters, performs cross-engine collaborative processing between the streaming computing engine and the batch computing engine to obtain computation results and actual execution metric data, including: The fusion coordinator injects the static dimension table generated by the batch computing engine into the stream computing engine in the form of a side stream. The validity period of the static dimension table is set to a preset multiple of the batch processing trigger cycle. After the validity period expires, the batch computing engine is triggered to regenerate the static dimension table. The data records in each labeled data block are associated with the static dimension table to obtain the real-time association calculation results. When the stream computing engine detects that the amount of data backlog exceeds the preset backlog threshold within a preset number of consecutive sampling periods, the fusion coordinator submits a historical backtracking batch processing task to the batch computing engine to perform consistency verification on all data within the preset backtracking time window and obtain the batch verification results. The real-time correlation calculation results and batch verification results are written to a unified status backend to obtain the calculation results and actual execution measurement data.

[0024] Specifically, a static dimension table refers to a key-value mapping table generated by the batch processing engine after performing a full aggregation calculation on a bounded historical dataset. Its content remains unchanged from generation until its expiration date. Sidestream injection refers to the fusion coordinator loading the static dimension table as an independent data channel into the stream processing engine's memory. When processing data records in each labeled data block, the stream processing engine directly queries the static dimension table in memory using the association key field of the data record, concatenating the fields of the data record and the static dimension table to obtain the real-time association calculation result. This query process is completed in the stream processing engine's memory without accessing any external storage. The validity period of the static dimension table is calculated by multiplying the batch processing trigger period by a preset multiple. The batch processing trigger period is the time interval between two consecutive full aggregation calculations performed by the batch processing engine. After the validity period expires, the fusion coordinator sends a regeneration command to the batch processing engine. The batch processing engine then performs a new round of full aggregation calculation, generating an updated static dimension table, which is then injected into the stream processing engine again in a sidestream form, replacing the original static dimension table in memory.

[0025] Data backlog refers to the total number of data records received but not yet processed in the stream processing engine's input queue. At the end of each sampling period, the stream processing engine calculates the current data backlog and compares it with a preset backlog threshold. When the data backlog calculated for a preset number of consecutive sampling periods exceeds the preset backlog threshold, the fusion coordinator submits a historical backtracking batch processing task to the batch processing engine. The preset backtracking time window refers to the time range extending from the current moment in the historical direction, determined by the backtracking time window parameter in the business service level agreement parameters. The batch processing engine performs cross-source consistency checks on data records from all heterogeneous data sources within this time range. The check verifies the consistency of the field values ​​corresponding to the same business key across different data sources, resulting in batch check results. The unified state backend refers to the state management component that provides a shared key-value storage space for stream computing engines and batch computing engines. Real-time correlation calculation results and batch verification results are written to the unified state backend using business keys as indexes. The two types of engines read and write intermediate states using the same key space. The actual execution measurement data includes four indicators: actual end-to-end processing latency, actual CPU resource consumption, actual peak memory consumption, and latency compliance flag. All of these are collected synchronously during execution and written to the unified state backend.

[0026] In one specific embodiment, step S4 involves online fine-tuning of the time series prediction model based on the calculation results, actual execution metric data, and confidence level using a confidence-weighted loss function. This includes: The actual processing delay in the actual execution measurement data is compared with the maximum allowable processing delay in the service level agreement parameters to obtain the delay compliance indicator. The difference between the actual resource consumption in the actual execution measurement data and the preset resource consumption benchmark value is normalized to obtain the resource consumption deviation. Based on the delayed achievement criteria and the deviation in resource consumption, the confidence score is obtained by weighting and summing them according to their respective weight coefficients. Using the confidence score as the sample weight, and based on the calculation results and the predicted probability output by the time series prediction model, the weighted loss value is obtained by calculating the confidence-weighted loss function.

[0027] Specifically, the latency compliance flag is a Boolean value obtained by comparing the actual processing latency with the maximum allowable processing latency. The flag is set to one when the actual processing latency does not exceed the maximum allowable latency, and zero otherwise. Actual resource consumption refers to two indicators: actual CPU resource consumption and actual peak memory consumption. The preset resource consumption benchmark value is a reference value for resource consumption under the corresponding processing mode, calculated based on historical execution data during system initialization. Difference normalization involves dividing the difference between the actual resource consumption and the preset resource consumption benchmark value by the preset benchmark value, taking the absolute value, and then subtracting this absolute value from one to obtain a resource consumption deviation ranging from 0 to 1. The closer the resource consumption deviation is to 1, the closer the actual resource consumption is to the benchmark value; the closer it is to 0, the larger the deviation. The confidence score is obtained by multiplying the latency compliance flag by its corresponding weight coefficient and the resource consumption deviation by its corresponding weight coefficient, and then summing the results. Both weight coefficients are real numbers greater than zero, and their sum is 1. The confidence score ranges from 0 to 1.

[0028] The confidence-weighted loss function is a cross-entropy loss function that uses the confidence score as the weight of each training sample. It is calculated as follows: For each training sample, the actual processing mode corresponding to the calculation result is encoded as a one-hot vector as a label. This label is then cross-entropy calculated between the label and the probability distribution vector formed by the three prediction probabilities of batch processing mode, micro-batch processing mode, and streaming processing mode output by the time series prediction model. This cross-entropy is then multiplied by the confidence score corresponding to the sample. The weighted cross-entropy of all training samples is summed to obtain the weighted loss value. Samples with low confidence scores correspond to scenarios where the label prediction is inaccurate, and their weighted cross-entropy contributes a larger value to the summation, making the weighted loss value more sensitive to inaccurate scenarios. This drives the time series prediction model parameters to be updated in the direction of correcting inaccurate scenarios during subsequent backpropagation.

[0029] Figure 2This is a schematic diagram illustrating how the confidence score changes with the number of online fine-tuning iterations in an embodiment of this application. Figure 2 As shown, the horizontal axis represents the number of online fine-tuning iterations, and the vertical axis represents the confidence score. The figure illustrates the convergence process of the confidence score in two typical scenarios.

[0030] Among them, conventional turbulent scenarios refer to data flow states that have occurred multiple times in the system's historical operation, and for which the time-series prediction model has accumulated sufficient historical samples during the training phase. Examples include sudden increases in data flow rate during periodic business peaks and regular increases in disorder when known batch tasks are triggered. Novel turbulent scenarios refer to data flow states that have never occurred or have occurred with extremely low frequency in the system's historical operation, and for which the time-series prediction model lacks corresponding historical samples to support them. Examples include sudden traffic anomalies and entirely new data arrival patterns generated after heterogeneous data sources are connected to new types of sensors. Because the time-series prediction model has insufficient initial prediction capability for novel turbulent scenarios, the confidence scores of its corresponding processing intent labels are generally low in the initial stage, which is precisely the target that the confidence-weighted loss function needs to focus on correcting.

[0031] In both scenarios, the confidence scores continuously increased with the number of online fine-tuning iterations. The confidence score for the conventional turbulence scenario exceeded the threshold of 0.85 after approximately 15 iterations, while the confidence score for the novel turbulence scenario exceeded the threshold of 0.85 after approximately 25 iterations. Both curves eventually converged above the threshold. These results indicate that the confidence-weighted loss function focuses on inaccurate samples with low confidence scores, driving the time-series prediction model parameters to continuously update towards correcting inaccurate scenarios. This ensures that the prediction quality of the intent label is consistently stable with increasing iterations in both scenarios. The convergence speed of the novel turbulence scenario is slower than that of the conventional turbulence scenario because the number of historical training samples corresponding to the novel scenario is smaller. The model has insufficient gradient accumulation from similar samples during backpropagation, thus requiring more iterations for effective correction. This perfectly aligns with the objective law that historical samples are scarce in novel scenarios, requiring more iterations for model correction.

[0032] In one specific embodiment, in step S4, the gradient contribution ratios of each feature after fine-tuning are updated inversely to the weight coefficients of the weighted fusion process in step S1 to obtain the updated turbulence index calculation weights, including: The weighted loss value is input into the time series prediction model. The time series prediction model takes the historical time series sequence of turbulence index as the input sequence and the probability distribution of processing mode as the output. The network parameters of the time series prediction model are updated by backpropagation based on the weighted loss value to obtain the fine-tuned time series prediction model. Based on the absolute gradient values ​​of each feature with respect to the weighted loss value in the fine-tuned time series prediction model, the gradient contribution ratio of each feature is obtained by summing the absolute gradient values ​​of each feature and then performing ratio processing. The gradient contribution ratio of each feature is updated inversely to the weight coefficients corresponding to the arrival rate, the second-order change of the arrival rate, and the disorder degree in the weighted fusion processing in step S1, so as to obtain the updated turbulence index calculation weight.

[0033] Specifically, the time-series prediction model employs a Long Short-Term Memory (LSTM) network, which consists of an input layer, an LSM layer, and an output layer. The input layer receives historical time-series sequences of turbulence indices within a preset time window length. The time window length refers to the number of consecutive sampling periods configured during system initialization. The input layer feeds this sequence into the LSM layer step by step. The LSM layer extracts long-term dependencies from the historical time-series sequences through three gating units: an input gate, a forget gate, and an output gate. The input gate determines how much input information from the current time step is written to the memory unit; the forget gate determines how much of the memory unit state from the previous time step is retained in the current time step; and the output gate determines how much of the current memory unit state is output as the current hidden state. Finally, the output layer receives the hidden state from the last time step, processes it through a fully connected layer, and outputs three prediction probabilities: batch processing mode, micro-batch processing mode, and streaming processing mode. The sum of these three prediction probabilities is one, and the processing mode corresponding to the largest of the three prediction probabilities is the processing mode probability distribution output for the current data block.

[0034] For example, if the time window length is set to twenty sampling periods, the input layer receives a sequence of twenty consecutive turbulence index values. The long short-term memory layer processes the turbulence index values ​​at each time step in the sequence step by step. After processing the twentieth time step, the final hidden state is output. The output layer maps the hidden state to three predicted probabilities. If the actual execution processing mode corresponding to the current data block is the streaming processing mode, the streaming processing mode is encoded as a one-hot vector. The cross-entropy is calculated between the vector and the probability distribution vector composed of the three predicted probabilities, and then multiplied by the confidence score corresponding to the data block to obtain the weighted cross-entropy contribution value of the sample.

[0035] Backpropagation update refers to taking the partial derivatives of the weighted loss value with respect to all trainable parameters in the Long Short-Term Memory network, and updating the parameter values ​​in the direction of gradient descent. This causes the weighted loss value to continuously decrease on subsequent training samples, resulting in a fine-tuned time-series prediction model. The absolute value of the gradient of each feature with respect to the weighted loss value is the result of taking the absolute value of the partial derivatives of the weighted loss value with respect to the arrival rate, the second-order change of the arrival rate, and the disorder degree, respectively. It reflects the degree of contribution of each feature to the change of the weighted loss value. The gradient contribution ratio of each feature is calculated by dividing the absolute value of the gradient of each feature by the sum of the absolute values ​​of the gradients of the three features, and the sum of the three gradient contribution ratios is one. The three gradient contribution ratios are respectively covered by the weight coefficients corresponding to the arrival rate, the second-order change of the arrival rate, and the disorder degree in step S1. This ensures that when subsequent data blocks are processed by weighted fusion, features that contribute more to the weighted loss value have a higher weight in the turbulence index calculation, while features that contribute less to the weighted loss value have a lower weight. The weight coefficients are updated once after each online fine-tuning to ensure that the weighted fusion calculation in step S1 always uses weight coefficients that are compatible with the current data scenario.

[0036] In one specific embodiment, step S4 involves inputting the weighted loss value into a time-series prediction model. The time-series prediction model takes the historical time-series sequence of the turbulence index as its input sequence and the probability distribution of the processing mode as its output, and includes: The time series prediction model is a long short-term memory network, which includes an input layer, a long short-term memory layer, and an output layer. The input layer receives the historical time series sequence of turbulence index within a preset time window length, and inputs the historical time series sequence of turbulence index into the long short-term memory layer for time series feature extraction processing to obtain the hidden state sequence; The hidden state sequence input and output layers are fully connected to obtain the prediction probabilities corresponding to the batch processing mode, micro-batch processing mode, and streaming processing mode. The sum of the three is one, and the processing mode corresponding to the maximum value of the prediction probability is output as the processing mode probability distribution of the current data block.

[0037] Specifically, Long Short-Term Memory (LSTM) networks are recurrent neural networks with gating mechanisms. Each computational unit in an LSM layer consists of four components: an input gate, a forget gate, an output gate, and a memory unit. The forget gate receives the input of the current time step and the hidden state of the previous time step, and outputs a gating value between zero and one after passing through a sigmoid activation function. This value determines how much information from the memory unit in the previous time step is retained in the current time step. Similarly, the input gate receives the input of the current time step and the hidden state of the previous time step, outputs a gating value after passing through a sigmoid activation function, and multiplies this value by the candidate memory value processed by a tanh activation function. This determines how much new information is written to the memory unit in the current time step. The output gate receives the input of the current time step and the hidden state of the previous time step, outputs a gating value after passing through a sigmoid activation function, and multiplies this value by the current memory unit state processed by a tanh activation function to obtain the hidden state output for the current time step. The input layer feeds the historical time series of turbulence index within the preset time window length into the long short-term memory layer step by step. The long short-term memory layer performs the above gating calculation on each time step in the sequence in turn, and outputs the corresponding hidden state for each time step. The hidden states of all time steps constitute the hidden state sequence.

[0038] The output layer receives the hidden state of the last time step in the hidden state sequence, performs a linear transformation through a fully connected layer, and outputs a three-dimensional vector. Then, a softmax function maps each component of the three-dimensional vector to the interval between zero and one, with the sum of the three components being one. These three components correspond to the prediction probabilities of batch processing mode, micro-batch processing mode, and streaming processing mode, respectively. The preset time window length refers to the number of consecutive sampling periods configured during system initialization, determining the length of the historical time sequence of turbulence indices received by the input layer each time. This length determines the total number of time steps for the long short-term memory layer to perform gating calculations. The processing mode probability distribution output takes the processing mode corresponding to the largest of the three predicted probabilities as the execution mode judgment result for the current data block. This result is written to the execution mode field of the processing intent label in step S2, and the corresponding predicted probability value is written to the confidence field of the processing intent label.

[0039] The above describes the streaming and batch adaptive fusion processing method for heterogeneous data sources in the embodiments of this application. The following describes the streaming and batch adaptive fusion processing system for heterogeneous data sources in the embodiments of this application. One embodiment of the streaming and batch adaptive fusion processing system for heterogeneous data sources in the embodiments of this application includes: The segmentation module is used to segment the input data stream of the heterogeneous data source into several data blocks according to a preset sampling period, extract the arrival rate, the second-order change of the arrival rate and the disorder degree of each data block, and obtain the turbulence index corresponding to each data block through weighted fusion processing. The mapping module is used to, for each data block, obtain a processing intent label carrying the execution mode, execution parameters and confidence level through decision matrix mapping processing based on the turbulence index and service level agreement parameters corresponding to the data block, and attach the processing intent label to the corresponding data block to obtain the labeled data block corresponding to each data block. The processing module is used to read the execution mode in each of the labeled data blocks, route each of the labeled data blocks to the corresponding stream computing engine or batch computing engine, and use the execution parameters as the driving force to perform cross-engine collaborative processing on the stream computing engine and the batch computing engine to obtain the calculation results and actual execution metric data. The update module is used to perform online fine-tuning of the time series prediction model based on the calculation results, the actual execution metric data, and the confidence score, using a confidence-weighted loss function. The gradient contribution ratio of each feature after fine-tuning is then updated inversely to the weight coefficients of the weighted fusion processing in step S1, resulting in updated weights for calculating the turbulence index. The confidence-weighted loss function is a cross-entropy loss function that uses the confidence score as the weight of each training sample. The calculation method is as follows: for each training sample, the actual processing mode corresponding to the calculation result is encoded as a one-hot vector as a label. This label is then cross-entropy calculated between the label and the probability distribution vector formed by the three prediction probabilities of the batch processing mode, micro-batch processing mode, and streaming processing mode output by the time series prediction model. This cross-entropy is then multiplied by the confidence score corresponding to the sample, and the weighted cross-entropy of all training samples is summed to obtain the weighted loss value.

[0040] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for stream and batch adaptive fusion processing towards heterogeneous data sources, characterized in that, The method comprises: Step S1: dividing the input data stream of the heterogeneous data source into a plurality of data blocks according to a preset sampling period, extracting the arrival rate, the second-order change amount of the arrival rate and the out-of-order degree of each data block, and obtaining the turbulence degree index corresponding to each data block through weighted fusion processing; Step S2: for each data block, according to the turbulence degree index corresponding to the data block and the service level agreement parameter, through decision matrix mapping processing, a processing intention label carrying the execution mode, the execution parameter and the confidence is obtained, the processing intention label is attached to the corresponding data block, and the labeled data block corresponding to each data block is obtained; Step S3: reading the execution mode in each labeled data block, routing each labeled data block to a corresponding stream computing engine or batch computing engine, and driving the stream computing engine and the batch computing engine through the execution parameter to perform cross-engine cooperative processing through the fusion coordinator, to obtain the calculation result and the actual execution metric data; Step S4: according to the calculation result, the actual execution metric data and the confidence, an online fine-tuning processing is performed on the time series prediction model through a confidence weighted loss function, and the gradient contribution proportion of each feature after fine-tuning is updated to the weight coefficient of the weighted fusion processing in step S1, to obtain the updated turbulence degree index calculation weight, wherein the confidence weighted loss function is a cross-entropy loss function taking the confidence score as the weight of each training sample, and the calculation method is: for each training sample, the actual processing mode corresponding to the calculation result is encoded as a one-hot vector as a label, and the cross-entropy between the label and the probability distribution vector composed of the three prediction probabilities of batch processing mode, micro-batch processing mode and stream processing mode output by the time series prediction model is calculated, and then multiplied by the confidence score corresponding to the sample. The weighted cross-entropy of all training samples is summed to obtain the weighted loss value. 2.The method of claim 1, wherein, The step S1 comprises: Divide the input data stream of the heterogeneous data source into a plurality of data blocks according to a preset sampling period, and count the total number of data records arrived in the preset sampling period for each data block. Based on the ratio of the total number of data records to the preset sampling period, the arrival rate of each data block is obtained; The first-order change amount of the arrival rate is obtained by differencing the arrival rates of adjacent sampling periods, and the second-order change amount of the arrival rate is obtained by differencing the first-order change amounts of the arrival rates of adjacent sampling periods; The out-of-order degree of each data block is obtained by taking the ratio of the number of data records with event timestamps earlier than the current processing time exceeding a preset out-of-order tolerance threshold to the total number of data records in the data block; After the arrival rate, the second-order change amount of the arrival rate and the out-of-order degree are respectively processed by maximum and minimum normalization, the weighted sum is obtained according to the corresponding weight coefficients, and the turbulence degree index corresponding to each data block is obtained. 3.The method of claim 1, wherein, In step S2, according to the turbulence degree index corresponding to the data block and the service level agreement parameter, through decision matrix mapping processing, a processing intention label carrying the execution mode, the execution parameter and the confidence is obtained, which comprises: The turbulence index is divided into a low turbulence interval, a medium turbulence interval and a high turbulence interval according to a preset classification threshold, the low turbulence interval corresponds to a batch processing mode, the medium turbulence interval corresponds to a micro-batch processing mode, and the high turbulence interval corresponds to a stream processing mode, to obtain an execution mode corresponding to each data block; Based on the execution mode, the maximum allowed processing delay in the service level agreement parameter and the result accuracy semantic requirement, a decision matrix mapping process is performed to obtain an execution parameter corresponding to each data block; The prediction probability output by the time series prediction model for the current turbulence index is taken as the confidence of each data block; The execution mode, the execution parameter and the confidence are packaged as a processing intent label and attached to the corresponding data block to obtain a labeled data block corresponding to each data block. 4.The method of claim 1, wherein, In step S3, the fusion coordinator drives the stream computing engine and the batch computing engine to perform cross-engine collaboration processing based on the execution parameter, to obtain a calculation result and actual execution metric data, including: The fusion coordinator injects a static dimension table generated by the batch computing engine into the stream computing engine in a side stream form, the validity period of the static dimension table is set to a preset multiple of the batch processing trigger period, and the batch computing engine is triggered to regenerate the static dimension table when the validity period is exceeded, the data records in each labeled data block are associated with the static dimension table to obtain a real-time association calculation result; When the stream computing engine detects that the data backlog exceeds a preset backlog threshold within a continuous preset number of sampling periods, the fusion coordinator submits a historical backtracking batch processing task to the batch computing engine, performs consistency checking processing on the full amount of data within a preset backtracking time window to obtain a batch checking result; The real-time association calculation result and the batch checking result are written into a unified state backend to obtain a calculation result and actual execution metric data.

5. The method of claim 1, wherein, In step S4, according to the calculation result, the actual execution metric data and the confidence, an online fine-tuning process is performed on the time series prediction model through a confidence weighted loss function, including: The actual processing delay in the actual execution metric data is compared with the maximum allowed processing delay in the service level agreement parameter to obtain a delay compliance flag, and the actual resource consumption in the actual execution metric data is normalized by subtracting a preset resource consumption reference value to obtain a resource consumption deviation; Based on the delay compliance flag and the resource consumption deviation, weighted summation is performed according to the respective weight coefficients to obtain a confidence score; The confidence score is taken as a sample weight, and based on the calculation result and the prediction probability output by the time series prediction model, a weighted loss value is calculated through a confidence weighted loss function. 6.The method of claim 5, wherein, In step S4, the gradient contribution proportion of each feature after fine-tuning is updated in reverse to the weight coefficient of the weighted fusion process in step S1 to obtain an updated turbulence index calculation weight, including: inputting the weighted loss value into a time series prediction model, the time series prediction model taking a historical time series of the turbulence index as an input sequence and taking a processing mode probability distribution as an output, and performing back propagation update on network parameters of the time series prediction model based on the weighted loss value to obtain a fine-tuned time series prediction model; performing ratio processing on gradient absolute values of each feature after summing the gradient absolute values of each feature based on the gradient absolute values of each feature with respect to the weighted loss value in the fine-tuned time series prediction model to obtain a gradient contribution proportion of each feature; performing reverse update of the gradient contribution proportion of each feature to weight coefficients of the arrival rate, the second-order change of the arrival rate, and the out-of-order degree respectively corresponding to the weight fusion processing in step S1 to obtain updated turbulence index calculation weights.

7. The method of claim 6, wherein, In the step S4, the weighted loss value is input into a time series prediction model, the time series prediction model taking a historical time series of the turbulence index as an input sequence and taking a processing mode probability distribution as an output, and the step S4 includes the following steps. The time series prediction model is a long short-term memory network, and the long short-term memory network includes an input layer, a long short-term memory layer, and an output layer. The input layer receives a historical time series of the turbulence index in a preset time series window length, inputs the historical time series of the turbulence index into the long short-term memory layer for time series feature extraction processing to obtain a hidden state sequence. The hidden state sequence is input into the output layer for full connection processing to obtain predicted probabilities of a batch processing mode, a micro-batch processing mode, and a stream processing mode respectively, and the sum of the three is one. The processing mode corresponding to the maximum value in the predicted probabilities is taken as a processing mode probability distribution of the current data block and is output.

8. A system for stream and batch adaptive fusion processing towards heterogeneous data sources, characterized in that, The system for implementing the method for adaptive fusion processing of streams and batches for heterogeneous data sources according to any one of claims 1-7 includes the following modules. A segmentation module is configured to segment an input data stream of a heterogeneous data source into a plurality of data blocks according to a preset sampling period, extract an arrival rate, a second-order change of the arrival rate, and an out-of-order degree for each data block, and obtain a turbulence index corresponding to each data block through weighted fusion processing. A mapping module is configured to, for each data block, obtain a processing intent label carrying an execution mode, an execution parameter, and a confidence level through decision matrix mapping processing based on the turbulence index corresponding to the data block and a service level agreement parameter, attach the processing intent label to the corresponding data block to obtain a labeled data block corresponding to each data block. A processing module is configured to read an execution mode in each labeled data block, route each labeled data block to a corresponding stream computing engine or batch computing engine, and perform cross-engine collaboration processing on the stream computing engine and the batch computing engine by a fusion coordinator driven by the execution parameter to obtain a calculation result and actual execution metric data. An updating module is configured to perform online fine-tuning processing on the time series prediction model according to the calculation result, the actual execution metric data and the confidence, and a confidence-weighted loss function, and to reversely update a gradient contribution proportion of each feature after fine-tuning to a weight coefficient of the weight fusion processing in step S1 to obtain an updated turbulence index calculation weight.

9. The system of claim 8, wherein, The input data stream of the heterogeneous data source is divided into a plurality of data blocks according to a preset sampling period, and the arrival rate, the second-order change amount of the arrival rate and the disorder degree of each data block are extracted, and the turbulence index corresponding to each data block is obtained through weighted fusion processing, including: The input data stream of the heterogeneous data source is divided into a plurality of data blocks according to a preset sampling period, and the arrival rate, the second-order change amount of the arrival rate and the disorder degree of each data block are extracted, and the turbulence index corresponding to each data block is obtained through weighted fusion processing, including: The input data stream of the heterogeneous data source is divided into a plurality of data blocks according to a preset sampling period, and the arrival rate, the second-order change amount of the arrival rate and the disorder degree of each data block are extracted, and the turbulence index corresponding to each data block is obtained through weighted fusion processing, including: The arrival rate, the second-order change amount of the arrival rate and the disorder degree are respectively processed through maximum-minimum normalization, and then weighted summation is performed according to the respective weight coefficients to obtain the turbulence index corresponding to each data block. According to the turbulence index corresponding to the data block and the service level agreement parameter, a decision matrix mapping processing is performed to obtain a processing intention label carrying an execution mode, an execution parameter and a confidence, including:

10. The system of claim 8, wherein, The turbulence index is divided into a low turbulence interval, a medium turbulence interval and a high turbulence interval according to a preset classification threshold, the low turbulence interval corresponds to a batch processing mode, the medium turbulence interval corresponds to a micro-batch processing mode, and the high turbulence interval corresponds to a stream processing mode, to obtain the execution mode corresponding to each data block; Based on the execution mode and the maximum allowed processing delay and result accuracy semantic requirement in the service level agreement parameter, a decision matrix mapping processing is performed to obtain the execution parameter corresponding to each data block; The prediction probability output by the time series prediction model for the current turbulence index is taken as the confidence corresponding to each data block; The execution mode, the execution parameter and the confidence are packaged as a processing intention label and attached to the corresponding data block to obtain a labeled data block corresponding to each data block. ​

Images