Self-adaptive real-time processing scheduling method for cross-platform data stream

Abstract

The invention relates to the technical field of data scheduling, and provides a self-adaptive real-time processing scheduling method for cross-platform data streams, which comprises the following steps of: S1, establishing a multi-protocol adaptation platform, performing intelligent format conversion on data, and performing standardized adaptation and secure access on multi-platform data in combination with edge access and real-time monitoring; s2, collecting a multi-source heterogeneous data stream in real time, and performing marginalization processing; and S3, performing multi-dimensional feature fusion on the processed data, and dynamically calculating the scheduling priority. The method comprises the steps of accurately collecting three-dimensional core features of service attributes, system states and resource requirements, generating trend coefficients, fusing the trend coefficients into priority calculation, realizing prospective adaptation of flow fluctuation, calculating a priority score through a multi-dimensional weighted fusion formula, emphasizing resource requirement dimensions when the load is too high, preferentially guaranteeing high-emergency-degree services when the load is relatively low, and effectively improving the service quality. And the accuracy and the adaptive capability of cross-platform data flow scheduling are remarkably improved.

Description

Technical Field

[0001] This invention relates to the field of data scheduling technology, and in particular to an adaptive real-time processing scheduling method for cross-platform data streams. Background Technology

[0002] In cross-platform data stream processing scenarios, existing technologies typically achieve multi-source data access by building basic protocol conversion interfaces, calculate scheduling priorities using fixed rules or single-dimensional indicators, manage node operation permissions through centralized permission configuration, and rely on static resource allocation strategies between edge and core nodes to complete data processing. Compliance audits are also conducted through manual verification or simple log recording. This approach can meet basic processing needs when the data scale is small and the scenario is simple, making it a common method for cross-platform data interaction.

[0003] However, existing technologies have many limitations and are difficult to adapt to complex and ever-changing cross-platform data processing scenarios: First, priority calculation is one-sided, failing to comprehensively consider core factors such as business attributes, system status, and resource requirements, and lacks the ability to predict traffic change trends, resulting in rigid scheduling strategies that cannot prioritize data processing with high urgency and low latency requirements; Second, permission management is mainly based on static configuration, unable to dynamically adjust permissions according to the data processing stage, and the data processing chain lacks a reliable traceability mechanism, making it difficult to deal with permission chaos in cross-platform scenarios; Third, compliance auditing relies on manual verification after the fact, which is inefficient and prone to omissions; Fourth, resource allocation lacks flexibility, failing to achieve dynamic collaboration between edge nodes and core nodes, easily leading to load imbalance, and protocol adaptation suffers from fragmentation issues, which can easily cause data access interruptions during interface iterations.

[0004] Therefore, we make improvements by proposing an adaptive real-time processing scheduling method for cross-platform data streams. Summary of the Invention

[0005] (I) The technical problem to be solved by this invention is to solve the problems of one-sided scheduling priority calculation dimension and lack of traffic trend prediction and inability of permissions to dynamically adapt to the processing stage in cross-platform multi-source heterogeneous data stream processing, and to overcome the limitations of existing solutions in terms of processing efficiency, stability, security and compliance.

[0006] (II) Technical Solution

[0007] To achieve the above-mentioned objectives, this invention provides an adaptive real-time processing and scheduling method for cross-platform data streams, comprising the following steps:

[0008] S1. Establish a multi-protocol adaptation platform to intelligently convert data formats, and combine edge access and real-time monitoring to standardize and securely access data from multiple platforms.

[0009] S2. Real-time acquisition of multi-source heterogeneous data streams and edge processing;

[0010] S3. Perform multi-dimensional feature fusion on the processed data and dynamically calculate the scheduling priority;

[0011] S4. Based on priority scores and real-time resource monitoring data, virtualization and software-defined networking technologies are used to dynamically allocate resources between edge nodes and core nodes.

[0012] S5. Design smart contracts to dynamically generate JWT permission tokens with a validity period of 30 seconds and update them in real time to adapt to permissions at each stage of data processing.

[0013] S6. Real-time monitoring of data processing quality indicators, timely identification of anomalies, and synchronous automatic adjustment of processing strategies to ensure continuous high quality and stable and smooth cross-platform data processing.

[0014] S7. Collect feedback on cross-platform data processing results and analyze problems, iteratively optimize scheduling strategies, and improve the efficiency, stability, and adaptability of data processing.

[0015] Preferably, in step S1, the multi-protocol adaptation platform is established based on a general data exchange protocol, and detailed interface specifications are formulated in combination with industry characteristics. At the same time, an intelligent protocol gateway is built to automatically identify heterogeneous protocols.

[0016] Preferably, in step S2, the multi-source heterogeneous data stream is edge-processed using the Flink streaming processing framework for data cleaning and format standardization conversion, sensitive fields are initially encrypted using the ChaCha20 lightweight encryption algorithm, and only the processed core data is transmitted to the core network to reduce bandwidth loss and transmission latency.

[0017] Preferably, in step S3, dynamically calculating the scheduling priority includes the following steps:

[0018] S31. In response to the characteristics of cross-platform data flow, collect three-dimensional core feature parameters of business attributes, system status and resource requirements to ensure that the data dimensions are comprehensive and quantifiable.

[0019] S32. Using the Min-Max standardization method, all parameters are mapped to the range of [0-1] to reduce the difference in the magnitude of parameters in each dimension and eliminate the influence of dimensions.

[0020] S33. Use the LSTM model to analyze the characteristics of historical data streams, predict the trend of traffic changes in the next 10 seconds, and generate trend coefficients.

[0021] S34. The standardized three-dimensional features and traffic trend coefficients are integrated and the priority score is calculated through multi-dimensional weighted integration. All cross-platform data are sorted in descending order according to the final scheduling priority score value. The higher the score, the higher the scheduling priority.

[0022] Preferably, in step S34, a reinforcement learning algorithm is introduced simultaneously to dynamically adjust the weight coefficients according to the real-time system load, ensuring that the scheduling strategy adapts to system changes.

[0023] Preferably, in step S4, for tasks with high priority and low latency requirements, edge nodes are dynamically allocated first. By utilizing the low latency characteristics of edge computing, large-scale and highly complex tasks are decomposed into several sub-tasks according to the principles of functional decoupling and parallel execution, and then distributed to idle nodes for parallel processing.

[0024] Preferably, in step S5, designing the smart contract includes the following steps:

[0025] S51. Construct a blockchain evidence storage network with a consortium blockchain architecture to adapt to cross-platform data processing scenarios;

[0026] S52. Write the entire processing information of cross-platform data streams into the blockchain in real time to achieve tamper-proof traceability. At the same time, set the timing of uploading to the chain. When each link is completed, immediately upload the information of that link to the chain to form a complete processing traceability chain.

[0027] S53. Deploy smart contracts on the blockchain, define a cross-platform data processing permission matrix, and dynamically adapt permissions;

[0028] S54. Utilize the immutability of blockchain to achieve compliance auditing, and automate the execution of compliance results through smart contracts.

[0029] Preferably, in step S6, the quality indicators of data processing include latency rate, data loss rate, and format error rate, and a time series prediction model is used to provide early warnings of potential faults such as node overload and interface anomalies. When an anomaly occurs, a response mechanism is automatically triggered.

[0030] Preferably, in step S7, the iterative optimization involves constructing a scheduling effect evaluation system, collecting data on processing delay, priority satisfaction rate, resource utilization rate, and anomaly recovery time, and then iteratively updating the priority weight coefficient, resource allocation threshold, and task splitting rules based on a reinforcement learning algorithm with the goal of achieving optimal long-term adaptive efficiency.

[0031] (III) Beneficial Effects

[0032] The adaptive real-time processing and scheduling method for cross-platform data streams provided by this invention has the following advantages:

[0033] 1. By accurately collecting three core features of business attributes, system status, and resource requirements, the system covers key influencing factors in cross-platform data processing. Quantitative grading ensures parameter operability. Simultaneously, an LSTM model is introduced to predict traffic change trends in the next 10 seconds, generating trend coefficients that are incorporated into priority calculations, enabling proactive adaptation to traffic fluctuations. Furthermore, priority scores are calculated using a multi-dimensional weighted fusion formula, and reinforcement learning algorithms are combined to dynamically adjust weight coefficients based on real-time system load. When the load is too high, the focus is on resource requirements; when the load is low, priority is given to high-urgency services, significantly improving the accuracy and adaptability of cross-platform data flow scheduling.

[0034] 2. By building a consortium blockchain evidence storage network, setting up three types of nodes—core, verification, and audit—and adopting the PBFT consensus mechanism, the network's fault tolerance and throughput meet the cross-platform processing requirements. Simultaneously, an on-chain data structure containing key fields such as BlockID and DataHash is defined to achieve tamper-proof traceability of information throughout the entire data processing chain. An innovative mechanism of "on-chain upon completion of each step" is adopted, using the SHA-256 algorithm to generate unique identifiers for timestamps, data digests, and other information, ensuring data integrity and source credibility. Furthermore, based on smart contracts, a dynamic permission matrix is ​​defined with nodes, steps, and operations as the core, generating 30-second-expired JWT tokens to achieve real-time permission adaptation. It also supports the automated execution of the "right to be forgotten," triggering the entire process of data marking, deletion, log on-chaining, and result verification through smart contracts. This solves the pain points of chaotic permissions and difficulty in implementing compliant audits in cross-platform data processing, constructing a trustworthy, secure, and compliant processing environment. Attached Figure Description

[0035] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0036] Figure 1 A schematic diagram of the overall process of an adaptive real-time processing scheduling method for cross-platform data streams provided in this application;

[0037] Figure 2 A schematic diagram illustrating the dynamic calculation scheduling priority process of an adaptive real-time processing scheduling method for cross-platform data streams provided in this application;

[0038] Figure 3 This application provides a schematic diagram of the smart contract process for designing an adaptive real-time processing and scheduling method for cross-platform data streams. Detailed Implementation

[0039] The specific embodiments of the present invention will be further described in detail below with reference to the accompanying drawings and examples. The following examples are for illustrative purposes only and should not be construed as limiting the scope of the invention.

[0040] like Figures 1-3 As shown, this embodiment proposes an adaptive real-time processing scheduling method for cross-platform data streams, including the following steps:

[0041] S1. Establish a multi-protocol adaptation platform to intelligently convert data formats, and combine edge access and real-time monitoring to standardize and securely access data from multiple platforms.

[0042] S2. Real-time acquisition of multi-source heterogeneous data streams and edge processing;

[0043] S3. Perform multi-dimensional feature fusion on the processed data and dynamically calculate the scheduling priority;

[0044] S4. Based on priority scores and real-time resource monitoring data, virtualization and software-defined networking technologies are used to dynamically allocate resources between edge nodes and core nodes.

[0045] S5. Design smart contracts to dynamically generate JWT permission tokens with a validity period of 30 seconds and update them in real time to adapt to permissions at each stage of data processing.

[0046] S6. Real-time monitoring of data processing quality indicators, timely identification of anomalies, and synchronous automatic adjustment of processing strategies to ensure continuous high quality and stable and smooth cross-platform data processing.

[0047] S7. Collect feedback on cross-platform data processing results and analyze problems, iteratively optimize scheduling strategies, and improve the efficiency, stability, and adaptability of data processing.

[0048] In this embodiment, in step S1, a multi-protocol adaptation platform is established based on a general data exchange protocol, and detailed interface specifications are formulated in combination with industry characteristics. At the same time, an intelligent protocol gateway is built to automatically identify heterogeneous protocols.

[0049] Specifically, it automatically identifies heterogeneous protocols and converts the private interfaces of each platform into standard interfaces through the adaptation layer component. At the same time, it presets an interface version compatibility mechanism and a dynamic update response module. When the platform interface iterates, it automatically adapts to the new protocol format, avoids data access interruption, and solves the problem of cross-platform protocol fragmentation.

[0050] In this embodiment, in step S2, the Flink streaming processing framework is used to perform edge processing on multi-source heterogeneous data streams for data cleaning and format standardization conversion. The ChaCha20 lightweight encryption algorithm is used to initially encrypt sensitive fields, and only the processed core data is transmitted to the core network to reduce bandwidth loss and transmission latency.

[0051] In this embodiment, step S3, dynamically calculating the scheduling priority, includes the following steps:

[0052] S31. In response to the characteristics of cross-platform data flow, collect three-dimensional core feature parameters of business attributes, system status and resource requirements to ensure that the data dimensions are comprehensive and quantifiable.

[0053] Specifically, the business attribute parameters include business urgency level, SLA timeliness threshold, and data importance coefficient. The business urgency level ranges from 1 to 5, with 5 being the highest. For example, medical emergency data is set to level 5. The SLA timeliness threshold is in milliseconds. For example, the SLA timeliness threshold for industrial control data is set to less than or equal to 10 milliseconds. The importance sensitivity coefficient ranges from 0.1 to 1.0. For example, the sensitivity coefficient for financial transactions is set to 1.0.

[0054] S32. Using the Min-Max standardization method, all parameters are mapped to the range of [0-1] to reduce the difference in the magnitude of parameters in each dimension and eliminate the influence of dimensions.

[0055] Specifically, the Min-Max standardized calculation formula is as follows: In the formula, x′ is the standardized feature value, x is the original feature value, and x' is the standardized feature value. min and x max These are the historical minimum and maximum values ​​for this feature, respectively. The business urgency and importance sensitivity coefficients are positively standardized, while the SLA timeliness threshold is negatively standardized. Ensure that data requiring low latency receives higher weight;

[0056] S33. Use the LSTM model to analyze the characteristics of historical data streams, predict the trend of traffic changes in the next 10 seconds, and generate trend coefficients.

[0057] Specifically, the calculation formula for the LSTM model is: P = σ(LSTM(X) ist ))×1.0+0.5, where P is the trend coefficient, σ is the Sigmoid activation function, and X istThe data consists of historical traffic sequence data from the past 5 minutes, including data volume and transmission frequency. The LSTM model takes the time-series features of historical traffic as input and outputs the probability of traffic changes. The final calculated trend coefficient ranges from 0.5 to 1.5. A value less than 1 indicates a decrease in traffic, a value greater than 1 indicates an increase in traffic, and a value greater than 1.2 indicates a sudden increase in traffic, in which case resources will be reserved in advance during subsequent scheduling.

[0058] S34. The standardized three-dimensional features and traffic trend coefficients are integrated and the priority score is calculated through multi-dimensional weighted integration. All cross-platform data are sorted in descending order according to the final scheduling priority score value. The higher the score, the higher the scheduling priority.

[0059] Specifically, the priority score calculation formula is: S = w1 × A ′ +w2×S ′ +w3×R ′ +w4×P, where S is the final scheduling priority score, A ′ The standardized fusion value is the business attribute value, and A' = 0.4E' + 0.3T'. SLA +0.3I′, where E′ and T′ SLA I and I' are the standardized values ​​of the business urgency level, SLA timeliness threshold, and importance sensitivity coefficient, respectively. ′ Let S' be the standardized fusion value of the system state, and S' = 0.3(1-C') + 0.3(1-M') + 0.2(1-B') + 0.2A' node Among them, C', M', B' and A' node R represents the standardized values ​​of CPU utilization, memory usage, network bandwidth load, and node availability, respectively, and R' is the standardized fusion value of resource requirements, where R' = 0.5(1-C'). comp )+0.3(1-S' data )+0.2(1-T' data ), where C' comp S' data and T' data These are the standardized values ​​of computational complexity, storage usage, and data transfer volume, respectively, with w1, w2, w3, and w4 being the corresponding weighting coefficients.

[0060] In this embodiment, in step S34, a reinforcement learning algorithm is introduced simultaneously to dynamically adjust the weight coefficients according to the real-time system load, ensuring that the scheduling strategy adapts to system changes.

[0061] Specifically, the calculation formula for the reinforcement learning algorithm is: w i '=w i ×(1+k×(Load-Load0)), where w i 'and wi Here, are the weight coefficients of the i-th dimension before and after adjustment, respectively; k is the weight adjustment coefficient with a value of 0.02, which is iteratively optimized by the reinforcement learning agent through a reward mechanism; and Load is the average system load, where Load = 0.5C. avg +0.5M avg , where C avg and M avg These are the average CPU and memory utilization rates of all nodes, respectively. Load0 is the baseline load threshold, with a value of 60%. The adjustment rule is: when Load > Load0, increase w3 and decrease w1; when Load < 40%, increase w1 to prioritize high-urgency services.

[0062] This invention accurately collects three core features—business attributes, system status, and resource requirements—covering key influencing factors in cross-platform data processing. It ensures parameter operability through quantitative grading and introduces an LSTM model to predict traffic trends over the next 10 seconds, generating trend coefficients that are incorporated into priority calculations. This enables proactive adaptation to traffic fluctuations. Furthermore, it calculates priority scores using a multi-dimensional weighted fusion formula and dynamically adjusts weight coefficients based on real-time system load using a reinforcement learning algorithm. When the load is too high, it prioritizes resource requirements; when the load is low, it prioritizes high-urgency services, significantly improving the accuracy and adaptability of cross-platform data flow scheduling.

[0063] In this embodiment, in step S4, for tasks with high priority and low latency requirements, edge nodes are dynamically allocated first. By utilizing the low latency characteristics of edge computing, large data volume and high complexity tasks are split into several sub-tasks according to the principles of functional decoupling and parallel execution, and then distributed to idle nodes for parallel processing.

[0064] Specifically, when the load on core nodes exceeds the threshold, non-core tasks are automatically migrated to edge nodes to ensure elastic resource scaling and load balancing.

[0065] In this embodiment, step S5, designing the smart contract includes the following steps:

[0066] S51. Construct a blockchain evidence storage network with a consortium blockchain architecture to adapt to cross-platform data processing scenarios;

[0067] Specifically, the data processing scenarios include node deployment, on-chain data structure definition, and network communication optimization. Node deployment involves setting up three types of nodes: core nodes, verification nodes, and audit nodes, and adopting the PBFT consensus mechanism to achieve a fault tolerance rate of more than one-third, thereby ensuring a network throughput of more than 1000 TPS. On-chain data structure definition uses BlockID to identify blocks, PrevHash to maintain the blockchain chain association, TimeStamp to record operation time, DataHash to verify the integrity of processed data, NodeID to locate processing nodes, and Signature to ensure the trustworthiness of data sources. Network communication optimization ensures the security of on-chain data transmission by encrypting all communication between nodes.

[0068] S52. Write the entire processing information of cross-platform data streams into the blockchain in real time to achieve tamper-proof traceability. At the same time, set the timing of uploading to the chain. When each link is completed, immediately upload the information of that link to the chain to form a complete processing traceability chain.

[0069] Specifically, the end-to-end information includes the timestamp of data processing, the data digest before and after processing, the node representation of the processing operation, and the processing operation type. A unique hash identifier is then generated using a hash algorithm. The calculation formula is: DataHash = SHA-256(TimeStamp + DataContent + NodeID + Operation), where DataHash is the unique hash identifier of the data processing information, SHA-256 is the hash algorithm that outputs a 256-bit hash value, TimeStamp is the data processing timestamp, accurate to milliseconds, DataContent is the data digest before and after processing (sensitive data must use an anonymized digest), NodeID is the node identifier that performed the processing operation, and Operation is the processing operation type.

[0070] S53. Deploy smart contracts on the blockchain, define a cross-platform data processing permission matrix, and dynamically adapt permissions;

[0071] Specifically, when deploying smart contracts developed based on OAuth2.0 and the ABAC model, a permission matrix is ​​first defined with processing node + data stage + operation type as the core dimensions. This clarifies the allowed operations of different nodes in the corresponding data stages and the 30-second permission validity period. Then, the contract is linked with the on-chain notarization module. When a node initiates a data operation request, the contract reads the current processing stage and node identity information on the chain, matches the permission matrix, and automatically generates the corresponding permission JWT token. The token expires automatically after its validity period expires. At the same time, the contract also supports the execution of permission revoke operations corresponding to the right to be forgotten, thereby achieving dynamic adaptation of permissions for cross-platform data processing.

[0072] S54. Utilize the immutability of blockchain to achieve compliance auditing, and automate the execution of compliance results through smart contracts;

[0073] Specifically, the smart contract is automatically triggered when a request for the right to be forgotten is received, and the calculation formula is as follows: In the formula, Compliance represents the compliance execution result, with 1 indicating full compliance and 0 indicating a violation. The execution logic is as follows: first, mark the corresponding data hash on the chain as pending deletion; then, send a data deletion instruction to all processing nodes; next, record the deletion operation log and upload it to the chain to ensure compliance traceability; finally, verify the deletion results of all nodes, and trigger an alarm if there are any nodes that have not deleted the data.

[0074] This invention establishes a consortium blockchain evidence storage network, sets up three types of nodes (core, verification, and audit), and adopts the PBFT consensus mechanism to ensure that the network's fault tolerance and throughput meet the cross-platform processing requirements. It also defines an on-chain data structure containing key fields such as BlockID and DataHash, enabling tamper-proof traceability of information throughout the entire data processing chain. An innovative mechanism of "on-chain upon completion of each step" is adopted, using the SHA-256 algorithm to generate unique identifiers for timestamps, data digests, and other information, ensuring data integrity and source credibility. Furthermore, based on smart contracts, a dynamic permission matrix is ​​defined with nodes, steps, and operations at its core, generating 30-second-expired JWT tokens for real-time permission adaptation. It also supports automated execution of the "right to be forgotten," triggering the entire process of data marking, deletion, log on-chaining, and result verification through smart contracts. This solves the pain points of chaotic permissions and difficulties in implementing compliant audits in cross-platform data processing, constructing a trustworthy, secure, and compliant processing environment.

[0075] In this embodiment, the quality indicators of data processing in step S6 include latency rate, data loss rate and format error rate, and the model provides early warning of potential faults such as node overload and interface anomalies based on time series prediction model. When an anomaly occurs, the response mechanism is automatically triggered.

[0076] Specifically, the response mechanism includes three levels of fault response conditions. Level 1 fault is a single node failure, and the response measure is to switch to a redundant node. Level 2 fault is network congestion, and the response measure is to downgrade high-complexity algorithms to lightweight models. Level 3 fault is a core node anomaly, and the response measure is to suspend low-priority tasks, release resources to ensure the execution of core tasks, and ensure processing continuity.

[0077] In this embodiment, in step S7, the iterative optimization constructs a scheduling effect evaluation system, collects data on processing delay, priority satisfaction rate, resource utilization rate, and anomaly recovery time, and then, based on the reinforcement learning algorithm, iteratively updates the priority weight coefficient, resource allocation threshold, and task splitting rules with the goal of achieving optimal long-term adaptive efficiency.

[0078] Specifically, during the iterative optimization process, stress tests are conducted regularly to optimize adaptation layer components and caching strategies for high-frequency bottlenecks, such as protocol conversion of high-concurrency data, such as adding a Redis caching layer, and continuously improve the system's adaptability to complex scenarios.

[0079] The above embodiments are for illustrative purposes only and are not intended to limit the scope of the invention. Although the invention has been described in detail with reference to the embodiments, those skilled in the art should understand that various combinations, modifications, or equivalent substitutions of the technical solutions of the invention do not depart from the spirit and scope of the invention and should be covered within the scope of the claims of the invention.

Claims

1. An adaptive real-time processing and scheduling method for cross-platform data streams, characterized in that, Includes the following steps: S1. Establish a multi-protocol adaptation platform to intelligently convert data formats, and combine edge access and real-time monitoring to standardize and securely access data from multiple platforms. S2. Real-time acquisition of multi-source heterogeneous data streams and edge processing; S3. Perform multi-dimensional feature fusion on the processed data and dynamically calculate the scheduling priority; S4. Based on priority scores and real-time resource monitoring data, virtualization and software-defined networking technologies are used to dynamically allocate resources between edge nodes and core nodes. S5. Design smart contracts to dynamically generate JWT permission tokens with a validity period of 30 seconds and update them in real time to adapt to permissions at each stage of data processing. S6. Real-time monitoring of data processing quality indicators, timely identification of anomalies, and synchronous automatic adjustment of processing strategies to ensure continuous high quality and stable and smooth cross-platform data processing. S7. Collect feedback on cross-platform data processing results and analyze problems, iteratively optimize scheduling strategies, and improve the efficiency, stability, and adaptability of data processing.

2. The adaptive real-time processing and scheduling method for cross-platform data streams according to claim 1, characterized in that, In step S1, a multi-protocol adaptation platform is established based on a general data exchange protocol, and detailed interface specifications are formulated in combination with industry characteristics. At the same time, an intelligent protocol gateway is built to automatically identify heterogeneous protocols.

3. The adaptive real-time processing and scheduling method for cross-platform data streams according to claim 1, characterized in that, In step S2, edge processing of multi-source heterogeneous data streams is performed using the Flink streaming processing framework for data cleaning and format standardization conversion. Sensitive fields are initially encrypted using the ChaCha20 lightweight encryption algorithm, and only the processed core data is transmitted to the core network to reduce bandwidth loss and transmission latency.

4. The adaptive real-time processing and scheduling method for cross-platform data streams according to claim 1, characterized in that, In step S3, dynamically calculating the scheduling priority includes the following steps: S31. In response to the characteristics of cross-platform data flow, collect three-dimensional core feature parameters of business attributes, system status and resource requirements to ensure that the data dimensions are comprehensive and quantifiable. S32. Using the Min-Max standardization method, all parameters are mapped to the range of [0-1] to reduce the difference in the magnitude of parameters in each dimension and eliminate the influence of dimensions. S33. Use the LSTM model to analyze the characteristics of historical data streams, predict the trend of traffic changes in the next 10 seconds, and generate trend coefficients. S34. The standardized three-dimensional features and traffic trend coefficients are integrated and the priority score is calculated through multi-dimensional weighted integration. All cross-platform data are sorted in descending order according to the final scheduling priority score value. The higher the score, the higher the scheduling priority.

5. The adaptive real-time processing and scheduling method for cross-platform data streams according to claim 4, characterized in that, In step S34, a reinforcement learning algorithm is introduced simultaneously to dynamically adjust the weight coefficients according to the real-time system load, ensuring that the scheduling strategy adapts to system changes.

6. The adaptive real-time processing and scheduling method for cross-platform data streams according to claim 4, characterized in that, In step S4, for tasks with high priority and low latency requirements, edge nodes are dynamically allocated first. Taking advantage of the low latency characteristics of edge computing, large data volume and high complexity tasks are split into several sub-tasks according to the principles of functional decoupling and parallel execution, and then distributed to idle nodes for parallel processing.

7. The adaptive real-time processing and scheduling method for cross-platform data streams according to claim 4, characterized in that, In step S5, designing the smart contract includes the following steps: S51. Construct a blockchain evidence storage network with a consortium blockchain architecture to adapt to cross-platform data processing scenarios; S52. Write the entire processing information of cross-platform data streams into the blockchain in real time to achieve tamper-proof traceability. At the same time, set the timing of uploading to the chain. When each link is completed, immediately upload the information of that link to the chain to form a complete processing traceability chain. S53. Deploy smart contracts on the blockchain, define a cross-platform data processing permission matrix, and dynamically adapt permissions; S54. Utilize the immutability of blockchain to achieve compliance auditing, and automate the execution of compliance results through smart contracts.

8. The adaptive real-time processing and scheduling method for cross-platform data streams according to claim 7, characterized in that, In step S6, the quality indicators of data processing include latency rate, data loss rate and format error rate, and based on the time series prediction model, it provides early warning of potential faults such as node overload and interface anomalies. When an anomaly occurs, a response mechanism is automatically triggered.

9. The adaptive real-time processing and scheduling method for cross-platform data streams according to claim 8, characterized in that, In step S7, iterative optimization involves constructing a scheduling effect evaluation system, collecting data on processing delay, priority satisfaction rate, resource utilization, and anomaly recovery time, and then iteratively updating priority weight coefficients, resource allocation thresholds, and task splitting rules based on reinforcement learning algorithms with the goal of achieving optimal long-term adaptive efficiency.

Images