A multi-system dynamic scheduling processing method for real-time streaming data of public services

Abstract

The application relates to data processing technology in the field of public services and discloses a multi-system dynamic scheduling processing method for real-time streaming data of public services, which comprises the following steps: collecting data from different data sources by a distributed data collection framework through a data collection and preprocessing layer, and performing preliminary format unification and standardization processing; the stream-batch integrated processing of a real-time stream computing engine and a batch processing system in a processing and analysis layer can reduce data redundancy and improve analysis timeliness; the dynamic scheduling in a multi-system scheduling and cooperation layer can adapt to business priority changes to provide scheduling flexibility; and the application and service layer provides an intuitive and convenient service interface for users, assists the users in scientific decision-making and resource optimization configuration, and promotes cross-department and cross-field data fusion innovation. When complex system load and variable business requirements are responded to, the method can improve the real-time performance, reliability and flexibility of data processing.

Description

Technical Field

[0001] This invention relates to data processing technology in the field of public services, and relates to, but is not limited to, a multi-system dynamic scheduling and processing method for real-time streaming data of public services. Background Technology

[0002] In modern society, public service sectors such as traffic management, healthcare, and urban security face the demand for processing massive amounts of real-time streaming data. This data comes from diverse sources, is varied in format, and is frequently updated, requiring rapid and accurate transfer and processing between multiple systems to support real-time decision-making and efficient services. However, traditional data processing architectures and scheduling methods struggle to meet the high-performance requirements of such complex scenarios. Summary of the Invention

[0003] In view of this, embodiments of the present invention provide a multi-system dynamic scheduling and processing method for real-time streaming data of public services, aiming to overcome the shortcomings of the prior art and improve the real-time performance, reliability and flexibility of data processing.

[0004] The specific technical solutions of this invention are as follows:

[0005] This invention provides a method for dynamic scheduling and processing of real-time streaming data from multiple systems in public services, the method comprising:

[0006] Data Acquisition and Preprocessing Layer: This layer acquires real-time streaming data from multiple data sources through a distributed data acquisition framework, performs preprocessing, and transmits the data; these multiple data sources include various public service data sources.

[0007] Processing and Analysis Layer: Utilizes a real-time stream computing engine to perform real-time stream computing on preprocessed data, while combining with a batch processing system to perform in-depth analysis of historical data and trigger data processing tasks;

[0008] Multi-system scheduling and collaboration layer: Through the task scheduling center, a dynamic scheduling strategy combining machine learning-based load prediction and multi-objective optimization scheduling model is adopted to dynamically allocate the data processing tasks to different subsystems, and realize the collaborative work between multiple subsystems through system adaptation interfaces and data routing and forwarding mechanisms.

[0009] Application and Service Layer: Based on the processing and analysis results of each subsystem, it provides business applications for specific public service areas, as well as decision support systems and data sharing and open platforms.

[0010] In some embodiments, the process of collecting and preprocessing real-time streaming data from multiple data sources using a distributed data acquisition framework includes: using Kafka as a message queue to collect real-time streaming data from different data sources and performing preliminary format unification and standardization processing; designing an intelligent edge computing task allocation algorithm based on the characteristics and service requirements of the data sources, dynamically deploying edge computing nodes to perform real-time preprocessing on some data; and caching the preprocessed data into different caching systems according to data type and uploading it to the data center.

[0011] In some embodiments, the process of using a real-time stream computing engine to perform real-time stream computing on preprocessed data, while simultaneously combining a batch processing system to perform in-depth analysis of historical data and triggering a data processing task, includes: defining event patterns using Flink's CEP library to detect key events conforming to specific patterns from the continuous data stream transmitted from the data acquisition and preprocessing layers; using Flink's window functions to perform aggregation statistics on the real-time streaming data corresponding to the key events within a time window; using Hadoop / Hive to analyze historical incident data, predict risk areas, and obtain batch processing results; and based on the Kappa architecture, unifying the real-time stream and batch processing results through middleware to obtain a unified stream and batch data processing task.

[0012] In some embodiments, the step of designing an intelligent edge computing task allocation algorithm based on the characteristics and service requirements of the data source, and dynamically deploying edge computing nodes to perform real-time preprocessing of some data, includes: for first-level tasks, processing is prioritized on edge computing nodes; for second-level tasks, the results or key data are uploaded to the central system for further analysis after preliminary processing on edge computing nodes; for third-level tasks, the data is directly uploaded to the central system for processing; wherein, the real-time requirements of the first-level tasks, the second-level tasks, and the third-level tasks decrease sequentially while the computational load increases sequentially.

[0013] In some embodiments, the method further includes: integrating machine learning libraries and AI model training and inference tools into edge computing nodes, and using historical and real-time data to predict, classify, and detect anomalies in public service scenarios.

[0014] In some embodiments, the dynamic scheduling strategy combining machine learning-based load prediction and multi-objective optimization scheduling models dynamically allocates the data processing tasks to different subsystems, including: training a load prediction model and predicting the load trend of each system based on historical task execution data and real-time streaming data; constructing a multi-objective optimization scheduling model with the objectives of minimizing response time, maximizing resource utilization, load balancing index, and task priority, while satisfying resource capacity, task dependency, and QoS constraints; generating a Pareto optimal scheduling scheme using the non-dominated sorting genetic algorithm NSGA-III; and dynamically adjusting the allocation of data processing tasks according to the predicted load through a sliding window mechanism.

[0015] In some embodiments, the method further includes: real-time monitoring of environmental changes and task execution status, triggering model updates and strategy adjustments.

[0016] In some embodiments, the collaborative work between multiple subsystems through system adaptation interfaces and data routing and forwarding mechanisms includes: developing standardized interfaces for different subsystems to support task submission, status query, and resource request operations; using Kafka message queues to achieve asynchronous transmission of task metadata and intermediate results; describing task dependencies through directed acyclic graphs and using the Airflow scheduling framework to achieve cascading task triggering.

[0017] The beneficial effects of the technical solutions provided in the embodiments of the present invention include at least the following:

[0018] 1) A unified stream-batch data processing model is proposed, combining real-time stream computing with batch processing. A real-time stream computing engine performs complex event processing and window aggregation calculations on real-time streaming data to extract its real-time value; simultaneously, a batch processing system is used to perform in-depth analysis of historical data, achieving unified stream-batch processing and providing more comprehensive data support for public service decision-making.

[0019] 2) A multi-system collaborative architecture was constructed, including a data source layer, a data acquisition and preprocessing layer, a data processing and analysis layer, a multi-system scheduling and collaboration layer, and an application and service layer. Each layer has a clear division of labor, and through system adaptation interfaces and data routing and forwarding mechanisms, seamless flow and interaction between different heterogeneous systems are achieved, ensuring that data can be efficiently processed and utilized across multiple systems.

[0020] 3) A dynamic scheduling strategy combining machine learning-based load prediction and a multi-objective optimization scheduling model is adopted. Machine learning technology is used to accurately predict future system load. Then, considering multiple objectives such as real-time data processing, resource utilization, system throughput, and task priority, a multi-objective optimization scheduling model is constructed. Intelligent optimization algorithms are used to solve for the optimal task scheduling scheme, thereby achieving dynamic resource allocation and efficient task execution.

[0021] 4) This method can process various types of data, including structured, semi-structured, and unstructured data from different data sources such as traffic sensors, medical IoT devices, and video surveillance cameras. Furthermore, the solution of this invention is not only applicable to specific public service areas such as traffic management, healthcare, and urban security, but can also be expanded in function and upgraded in performance according to actual needs, adapting to constantly changing business requirements and technological developments. Attached Figure Description

[0022] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort, wherein:

[0023] Figure 1 A flowchart illustrating the multi-system dynamic scheduling and processing method for real-time streaming data of public services provided in this embodiment of the invention. Detailed Implementation

[0024] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. The following embodiments are used to illustrate the present invention, but are not intended to limit the scope of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0025] In the following description, references are made to “some embodiments,” which describe a subset of all possible embodiments. However, it is understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0026] It should be noted that the terms "first, second, and third" used in the embodiments of the present invention are only used to distinguish similar objects and do not represent a specific ordering of objects. It is understood that "first, second, and third" can be interchanged in a specific order or sequence where permitted, so that the embodiments of the present invention described herein can be implemented in an order other than that illustrated or described herein.

[0027] It will be understood by those skilled in the art that, unless otherwise defined, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art to which these embodiments of the invention pertain. It should also be understood that terms such as those defined in general dictionaries should be understood to have the same meaning as in the context of the prior art and should not be interpreted in an idealized or overly formal sense unless specifically defined as herein.

[0028] Figure 1 A flowchart illustrating a multi-system dynamic scheduling and processing method for real-time streaming data of public services provided in an embodiment of the present invention is shown below. Figure 1 As shown, the method includes at least the following steps:

[0029] Step S110, Data Acquisition and Preprocessing Layer: Real-time streaming data is acquired from multiple data sources through a distributed data acquisition framework and then preprocessed and transmitted.

[0030] This step aims to extract high-quality streaming data from multi-source heterogeneous data. The multiple data sources include various public service data sources, namely various systems in the public service sector, such as traffic monitoring cameras, environmental monitoring sensors, government service interfaces, weather stations, medical equipment, and social media APIs. These data sources continuously generate real-time streaming data in different data formats (structured, semi-structured, and unstructured) and transmission protocols, characterized by large data volumes, diverse formats, and high real-time requirements.

[0031] The key technical aspects include the following:

[0032] 1) Distributed Acquisition Framework: Used to build the data acquisition and preprocessing layer, supporting high-throughput, low-latency data access. Furthermore, a "data splitting" strategy is adopted during acquisition: high-priority data (such as emergency alarm information) is transmitted through a low-latency channel, while ordinary data is transmitted through a batch queue.

[0033] Common distributed data acquisition frameworks include:

[0034] Apache Kafka: A high-throughput, distributed, and scalable message queue that supports real-time data stream processing. Apache Flume: A distributed, reliable, and highly available log collection system suitable for real-time log data acquisition. Apache Nifi: A visualization tool that supports data stream management and is easy to configure and manage. Logstash: An open-source data collection engine that supports real-time acquisition and preprocessing of various data sources.

[0035] During implementation, appropriate tools are selected to build the data acquisition and preprocessing layer and processing framework based on business needs (e.g., Kafka for high-throughput scenarios, Flink for complex computing scenarios). Simultaneously, the appropriate method for data access is chosen based on the type of data source.

[0036] Log files (including server logs, application logs, access logs, etc.): Use Flume or Logstash to collect log files, supporting multiple log formats (such as JSON, CSV).

[0037] Message queues (Kafka, RabbitMQ, Pulsar, etc.): Use Kafka as a data source or target through Kafka Connect or other client libraries.

[0038] Databases (relational and NoSQL databases): Use tools such as Debezium to capture database change logs (CDC) to achieve real-time data collection.

[0039] API interface: Obtain data in real time via HTTP requests or WebSocket.

[0040] 2) Preprocessing operations: After data acquisition, the following real-time preprocessing is performed to meet the needs of subsequent analysis or storage.

[0041] Data cleaning: Removing noise, outliers, and other invalid data (such as null values, duplicate values, and GPS drift data); formatting data (such as timestamp conversion and field mapping). Data transformation: Aggregating data (statistics by time window, such as traffic flow statistics for a certain area within 5 minutes, reducing downstream computational pressure); calculating derived fields (such as average and maximum values). Data filtering: Filtering data according to business rules (such as retaining only specific types of events). For example, a city traffic monitoring system collects data from 1000+ cameras via Kafka; after preprocessing, the raw data volume is reduced by 30%, and accident-prone road sections are marked in real time.

[0042] Efficiently connecting to various data sources, including but not limited to sensor networks, log files, and database change logs, and automatically identifying and parsing different data formats and transmission protocols to ensure the integrity and accuracy of data acquisition, is fundamental for subsequent dynamic scheduling and processing across multiple systems. This invention aims to acquire data from multiple heterogeneous data sources in a low-latency, high-throughput manner through a distributed data acquisition framework, supporting subsequent real-time processing and analysis.

[0043] For example, scenario 1: traffic flow monitoring, with the corresponding data source being traffic sensors.

[0044] Processing flow: Sensor data (such as GPS coordinates, traffic flow, and speed) is pushed to Kafka via MQTT or CoAP protocol. Flink consumes the Kafka data and calculates the traffic flow per minute. The results are stored in PostgreSQL and displayed as a heatmap using Grafana.

[0045] Scenario 2: Patient health monitoring, with the corresponding data source being medical IoT devices.

[0046] Processing flow: Patient vital signs (heart rate, blood pressure) and device status are pushed to Kafka via HTTP API or WebSocket. Flink consumes the Kafka data and detects abnormal heart rate or blood pressure. When an anomaly is detected, an alarm is triggered to notify the doctor.

[0047] Scenario 3: Video surveillance analysis, with the corresponding data source being video surveillance cameras.

[0048] Processing flow: The camera runs an object detection model and pushes keyframes to Kafka. Flink consumes the Kafka data and counts the number of vehicles detected per minute. The results are stored in Elasticsearch, supporting historical queries.

[0049] Step S120, Processing and Analysis Layer: The preprocessed data is processed in real time using a real-time stream computing engine, while the historical data is analyzed in depth using a batch processing system, triggering data processing tasks.

[0050] This step aims to jointly analyze real-time streaming data with historical data, achieving integrated stream and batch processing. Preprocessed data is input into real-time stream computing engines, such as Apache Flink and Spark Streaming, for stream processing to extract real-time value. For tasks requiring in-depth analysis of historical data, batch processing systems (such as Hadoop and Hive) are used to analyze historical data, uncovering long-term trends or correlations, working collaboratively with the stream computing engine. Finally, the Kappa architecture unifies real-time and offline computing logic, reducing data redundancy and achieving integrated stream and batch processing. Custom middleware merges real-time and batch processing results into a unified view, and the analysis results are pushed to downstream applications (such as early warning systems and decision-making platforms).

[0051] The technical implementation involves the following key points:

[0052] 1) Real-time stream processing engine: Uses Flink / Spark Streaming to process complex events (such as "three consecutive speeding events trigger an alert") and window aggregation (such as "calculate the peak traffic flow within one hour").

[0053] 2) Batch Processing System: This system combines Hadoop / Hive to analyze historical incident data and predict risk areas. For example, it analyzes user behavior logs from the past month to uncover user preferences and predict churn risk. Simultaneously, it trains machine learning models (such as user profiling models and recommendation systems) based on batch processing tasks and feeds the results back to the real-time stream processing system to improve the accuracy of real-time decision-making.

[0054] 3) Unified Streaming and Batch Processing: Based on the Kappa architecture, it unifies real-time streaming and batch processing logic, and achieves data fusion through middleware (such as Kafka Streams) to reduce data redundancy. The working principle is as follows:

[0055] Existing data processing models are relatively singular, with some studies focusing solely on real-time streaming data processing or batch data processing. When faced with the complex and ever-changing data processing needs of public services, such singular processing models may fail to fully leverage the value of the data. This invention proposes an integrated stream-batch data processing model that combines real-time stream computing with batch processing. A real-time stream computing engine performs stream computing operations on real-time streaming data to extract its real-time value; simultaneously, a batch processing system performs in-depth analysis of historical data, achieving integrated stream-batch processing and providing more comprehensive data support for public service decision-making.

[0056] Step S130, Multi-system scheduling and collaboration layer: The task scheduling center adopts a dynamic scheduling strategy that combines machine learning-based load prediction and multi-objective optimization scheduling model to dynamically allocate data processing tasks to different subsystems, and realizes collaborative work between multiple systems through system adaptation interfaces and data routing and forwarding mechanisms.

[0057] This step aims to dynamically allocate tasks based on resources to achieve efficient collaboration among multiple systems. A dynamic scheduling strategy combining machine learning-based load prediction and a multi-objective optimization scheduling model is employed to achieve dynamic resource allocation and efficient task execution. The key technical aspects involved are as follows:

[0058] 1) Task Scheduling Center: Responsible for dynamically allocating data processing tasks based on data characteristics and other factors, and monitoring and managing the task execution process. Preferably, this invention implements resource-aware scheduling based on Kubernetes / YARN, taking into account system load and task priority.

[0059] 2) Machine learning-based load prediction: Collect historical system load data and business traffic patterns, and use time series analysis, deep learning and other technologies to accurately predict the system load in the future, providing a forward-looking basis for task scheduling.

[0060] 3) Multi-objective optimization scheduling model: Taking into account multiple objectives such as real-time data processing, resource utilization, system throughput, and task priority, a mathematical optimization model is constructed, and the optimal task scheduling scheme is solved through intelligent optimization algorithms.

[0061] 3) System adaptation interface: Provides RESTful API / gRPC interface to support rapid access of subsystems (such as early warning system, data analysis platform, historical database).

[0062] 4) Data Routing and Forwarding: Determine the data transmission path according to the task scheduling strategy and transmit the data to the corresponding subsystem to ensure the security and reliability of data transmission. Preferably, NSQ / RabbitMQ is used for data distribution to ensure low latency (<100ms).

[0063] Step S140, Application and Service Layer: Based on the processing and analysis results of each subsystem, provide business applications for specific public service areas, as well as decision support systems and data sharing and open platforms.

[0064] This step aims to transform data value into business capabilities. Based on the data processing and analysis results obtained from the preceding steps, various business applications will be developed for specific public service sectors, such as intelligent traffic control systems, remote medical diagnosis platforms, and urban security monitoring and command systems, providing users with intuitive and convenient service interfaces.

[0065] Among them, specific public service applications include real-time traffic management systems that can push congestion warnings to navigation apps via WebSocket; medical equipment anomaly monitoring systems that can trigger automatic alarms to hospital systems; and pollution warning systems that can automatically trigger traffic restriction measures when PM2.5 levels exceed the standard.

[0066] Among them, the decision support system provides public service managers and decision-makers with tools such as data visualization, report generation, and trend analysis to assist them in making scientific decisions and optimizing resource allocation. For example, a traffic flow prediction model: based on LSTM to predict traffic flow for the next hour.

[0067] The data sharing and open platform, in accordance with relevant laws, regulations, and data security standards, enables the secure sharing and open access to public service data, promoting cross-departmental and cross-sectoral data integration and innovation. It provides standardized APIs to support third-party applications (such as urban planning departments) in accessing the data.

[0068] This invention, through the aforementioned steps, achieves efficient processing and multi-system collaboration of real-time streaming data for public services, providing technical support for smart city management. Specifically, the processing and analysis layer combines a real-time streaming computing engine with a batch processing system for integrated streaming and batch processing, reducing data redundancy and improving analysis timeliness. The multi-system scheduling and collaboration layer provides flexibility through dynamic scheduling that adapts to changes in business priorities. The application and service layer offers users an intuitive and convenient service interface, assisting them in making scientific decisions and optimizing resource allocation, and promoting cross-departmental and cross-domain data fusion and innovation.

[0069] In some embodiments, step S110 above, "collecting real-time streaming data from multiple data sources and preprocessing it using a distributed data acquisition framework," is further implemented through the following process: using Kafka as a message queue to collect real-time streaming data from different data sources and perform preliminary format unification and standardization processing; designing an intelligent edge computing task allocation algorithm based on the characteristics and service requirements of the data sources, dynamically deploying edge computing nodes to perform real-time preprocessing on some data; and caching the preprocessed data into different caching systems according to the data type and uploading it to the data center.

[0070] Here, the system first identifies and interfaces with various data sources, automatically parsing data in different formats and transmission protocols. Traffic sensors and medical IoT devices use JSON or Avro formats. Video streams from video surveillance are directly stored in object storage (such as S3), while keyframe data (such as motion detection results) is pushed to Kafka in JSON format.

[0071] To unify data from different formats into a standard format for easier subsequent processing and analysis, the following strategies can be adopted: 1) Define a unified data model: Based on business needs, define a unified data model, including data structure, field types, naming conventions, etc. 2) Data format conversion tools: Develop or use existing data format conversion tools to convert data from different formats into a unified data model. 3) Data quality verification: Add a data quality verification step during the data conversion process to ensure the accuracy and integrity of the data.

[0072] Then, based on the characteristics and service requirements of the data source, an intelligent edge computing task allocation algorithm is designed and deployed close to the data source to optimize overall system performance. For some data with high real-time requirements, real-time preprocessing, such as data cleaning, filtering, and compression, is performed through the deployed edge computing nodes, thereby reducing invalid data transmission and lowering the processing pressure on the central system.

[0073] The characteristics of data sources mainly include: data type (such as sensor data, video stream data, log data, etc.); data generation frequency: high frequency (such as hundreds of times per second) or low frequency (such as once per hour); data size (ranging from KB to GB); data real-time requirements (such as real-time monitoring, early warning, control, etc.); and data sensitivity (data involving privacy, security, or trade secrets requires special handling).

[0074] Service requirements mainly include: Response time requirements: such as millisecond, second, or minute-level response times. Processing accuracy requirements: such as high-precision calculations and approximate calculations. Resource consumption limitations: such as limitations on the use of resources such as CPU, memory, and storage. Reliability requirements: such as no data loss and accurate calculation results.

[0075] Finally, high-performance caching technologies (such as Hadoop HDFS, Elasticsearch, relational databases, and NoSQL databases) are used to temporarily cache the collected data to cope with data surges and system failure recovery scenarios. Hadoop HDFS is suitable for large-scale data storage; Elasticsearch supports full-text search and real-time analysis; relational databases such as MySQL and PostgreSQL are suitable for structured data; and NoSQL databases such as MongoDB and Cassandra are suitable for semi-structured or unstructured data.

[0076] In this embodiment, distributed message queue technology is used to efficiently collect data from different data sources and perform preliminary format unification and standardization processing. Deploying edge computing nodes reduces invalid data transmission and lowers the processing pressure on the central system. Simultaneously, a data caching mechanism is provided to handle scenarios such as data surges and system failure recovery.

[0077] In some embodiments, step S120 above, "using a real-time stream computing engine to perform real-time stream computing on preprocessed data, while combining a batch processing system to perform in-depth analysis of historical data and trigger a data processing task," is further implemented through the following process: defining event patterns using Flink's CEP library, detecting key events that conform to specific patterns from the continuous data stream transmitted from the data acquisition and preprocessing layers; using Flink's window functions to perform aggregation statistics on the real-time streaming data corresponding to the key events within a time window; using Hadoop / Hive to analyze historical accident data, predict risk areas, and obtain batch processing results; and based on the Kappa architecture, unifying the real-time stream and batch processing results through middleware (Kafka Streams) to obtain a unified stream and batch data processing task.

[0078] Here, in stream processing, complex events are processed using tools such as Flink / Spark Streaming. For example, in traffic monitoring scenarios, if a vehicle experiences three consecutive speeding events (speed exceeding 100km / h) with an interval of less than 5 minutes, a "speeding" warning is triggered. In financial risk control scenarios, if an account experiences three unauthorized logins within 5 minutes, an anomaly warning is triggered.

[0079] Aggregate and statistically analyze streaming data based on time windows (such as sliding windows or session windows). For example, in traffic monitoring scenarios, Flink's window functions (such as TumblingWindow) are used to aggregate and analyze video keyframes to calculate the peak traffic flow within one hour; in e-commerce scenarios, the number of clicks and conversion rates of a product page are counted every minute, and real-time dashboard data is dynamically updated.

[0080] By combining batch processing systems such as Hadoop / Hive with historical accident data analysis, long-term trends or correlations can be uncovered. For example, when analyzing traffic data in real time, historical accident data (such as T+1 day data) can be combined to dynamically adjust warning thresholds (e.g., lowering the speeding threshold to 80km / h in rainy weather). In financial risk control scenarios, user behavior logs from the past month can be analyzed to uncover user preferences and predict churn risk.

[0081] In this embodiment, low-latency real-time computing is achieved through complex event processing and window aggregation computation, while in-depth analysis of historical data is performed through batch processing. This integrated stream-batch data processing model combines real-time stream computing with batch processing results, achieving a balance between real-time response and deep insights, ensuring business timeliness while unlocking the long-term value of data.

[0082] In some embodiments, step S130 above, "designing an intelligent edge computing task allocation algorithm based on the characteristics and service requirements of the data source, and dynamically deploying edge computing nodes to perform real-time preprocessing of some data," includes: for first-level tasks, processing is preferentially completed on edge computing nodes; for second-level tasks, after preliminary processing on edge computing nodes, the results or key data are uploaded to the central system for further analysis; for third-level tasks, the data is directly uploaded to the central system for processing; wherein, the real-time requirements of the first-level tasks, the second-level tasks, and the third-level tasks decrease sequentially, while the computational load increases sequentially.

[0083] Here, the first level of tasks are those with high real-time requirements and low computational load, such as real-time monitoring and early warning of sensor data; the second level of tasks are those with moderate real-time requirements and high computational load, such as preliminary analysis of video stream data; and the third level of tasks are those with low real-time requirements and huge computational load, such as in-depth analysis of large-scale log data.

[0084] Based on task classification and edge node capability assessment, the following intelligent task allocation strategy is designed: First-level tasks, such as traffic congestion warning and accident detection, are prioritized for completion at edge nodes, and corresponding control actions are immediately triggered to reduce network latency and central system load; Second-level tasks, such as traffic flow statistics and vehicle behavior analysis, can be initially processed at edge nodes, and then the results or key data are uploaded to the central system for further analysis and visualization; Third-level tasks, such as long-term traffic trend prediction and road maintenance plan formulation, can be directly uploaded to the central system for processing to fully utilize the central system's computing resources and data storage capabilities.

[0085] In this embodiment, by designing an intelligent edge computing task allocation algorithm, it is possible to dynamically determine which data processing operations are completed at the edge nodes and which need to be uploaded to the central system based on the characteristics of the data source and service requirements, thereby achieving optimal overall system performance.

[0086] In some embodiments, the method further includes: integrating machine learning libraries and AI model training and inference tools into edge computing nodes, and using historical and real-time data to predict, classify, and detect anomalies in public service scenarios.

[0087] Here, machine learning libraries such as TensorFlow and PyTorch are used. At edge computing nodes, an intelligent data processing and analysis system is built by integrating advanced machine learning libraries and efficient AI model training and inference tools. This system can fully utilize accumulated historical data and the continuous influx of real-time data to conduct in-depth mining and analysis of various public service scenarios. Specifically, it can accurately predict various situations in public services, such as predicting peak passenger flow periods in public transportation and predicting fluctuations in energy consumption; simultaneously, it can scientifically classify different service objects or events, such as classifying the types of faults in urban facilities, in order to take targeted maintenance measures; furthermore, it can efficiently complete anomaly detection tasks, promptly identifying abnormal conditions in public services, such as abnormal water quality or traffic flow, thereby providing strong data support for the optimization and decision-making of public services.

[0088] In some embodiments, the dynamic scheduling strategy combining machine learning-based load prediction and multi-objective optimization scheduling models dynamically allocates the data processing tasks to different subsystems, including: training a load prediction model and predicting the load trend of each system based on historical task execution data and real-time streaming data; constructing a multi-objective optimization scheduling model with the objectives of minimizing response time, maximizing resource utilization, load balancing index, and task priority, while satisfying resource capacity, task dependency, and QoS constraints; generating a Pareto optimal scheduling scheme using the NSGA-III algorithm; and dynamically adjusting the allocation of data processing tasks according to the predicted load through a sliding window mechanism.

[0089] Here, the training and trend prediction process of the load forecasting model includes:

[0090] Data Acquisition: Two types of core data are collected: first, historical task execution data, covering task type (such as compute-intensive, I / O-intensive), execution duration (second-level precision), and resource consumption (CPU / memory / storage usage per unit time); second, real-time streaming data, i.e. system status data, including CPU / memory utilization (sampled per second) and network bandwidth (Mbps-level dynamic monitoring).

[0091] Feature engineering: In the data preprocessing stage, one-hot encoding is performed on task types, and execution time and resource consumption are standardized. Statistics are aggregated for real-time streaming data by time window (e.g., 5 minutes).

[0092] Model Construction and Training: The load prediction model employs a hybrid LSTM-GRU architecture. Preprocessed data is input into an LSTM layer (50 units) to capture time-series dependencies, and then a GRU layer (30 units) is used to enhance short-term dynamic features. A two-layer fully connected network (ReLU activation) maps the features to the load trend space. The model outputs predicted load values ​​(expressed as a percentage of resource usage) for each subsystem (e.g., compute, storage, network) within the next 15 minutes. The mean squared error (MSE) loss function is used during training.

[0093]

[0094] Among them, y i For real load, These are predicted values.

[0095] The process of constructing a multi-objective optimization scheduling model includes:

[0096] Objective function to minimize response time Where M is the number of tasks. Let be the delay time of task j; the objective function is to maximize resource utilization. Where K is the number of subsystems. For the resources already used by subsystem k, The objective function of the load balancing index is: Total Resources; Where L k For the load of subsystem k, The objective function is: average load; task priority. w j For priority weights, P j The task is considered complete (0 or 1). Constraints include: resource capacity constraints. Service quality constraints: Task dependency constraint: If task j depends on task i, then the start time of task j is later than the end time of task i.

[0097] The process of generating the Pareto optimal scheduling scheme using the NSGA-III algorithm is as follows:

[0098] S1, Population initialization: Randomly generate N scheduling schemes, each scheme containing a task allocation matrix (M×K);

[0099] S2, Non-dominated sorting: The scheme is stratified according to the objective function values ​​f1, f2, f3, f4;

[0100] S3, Reference point generation: Generate uniformly distributed reference points using the Das-Dennis method;

[0101] S4, Environment Selection: Select the next generation of population based on the reference point, preserving the Pareto front solution.

[0102] The final output is the Pareto optimal scheduling scheme, which includes task allocation, resource allocation, and latency prediction, and supports dynamic adjustment to cope with sudden load fluctuations.

[0103] A sliding window mechanism, for example, updates the scheduling scheme every 5 minutes. When the predicted load exceeds a threshold (e.g., 80%), it triggers a re-evaluation of the scheduling scheme, migrating tasks from high-load subsystems to low-load subsystems. At the same time, it fine-tunes the load prediction model parameters based on the actual task execution results (latency, resource utilization), forming a closed-loop optimization.

[0104] Through this embodiment, the multi-system dynamic scheduling method provided by the present invention can dynamically balance response time, resource utilization and load balancing, while satisfying task priority and resource constraints, and achieve efficient multi-system collaborative scheduling.

[0105] In some embodiments, the method further includes: real-time monitoring of environmental changes and task execution status, triggering model updates and strategy adjustments.

[0106] Here, the system monitors changes in the external environment (such as dynamic data fluctuations, abnormal event triggering, etc.) and the internal task execution status (such as progress deviations, resource utilization, etc.) in real time, automatically identifies key thresholds or pattern deviations, and immediately triggers model parameter optimization and strategy dynamic adjustment mechanisms to ensure the timeliness and adaptability of the system response.

[0107] In some embodiments, the collaborative work between multiple subsystems through system adaptation interfaces and data routing and forwarding mechanisms includes: developing standardized interfaces for different subsystems to support task submission, status query, and resource request operations; using Kafka message queues to achieve asynchronous transmission of task metadata and intermediate results; describing task dependencies through directed acyclic graphs and using the Airflow scheduling framework to achieve cascading task triggering.

[0108] Here, a standardized set of interfaces has been developed for different subsystems. This set of interfaces has high versatility and compatibility, and can fully support key operations such as task submission, status query, and resource request. The task submission interface ensures that each subsystem can easily and accurately transmit task information to the system core; the status query interface allows subsystems to monitor the processing progress and current status of tasks in real time; and the resource request interface ensures that subsystems can obtain the necessary computing, storage, and other resources in a timely manner during operation.

[0109] For data transmission, Kafka message queues were introduced. Kafka, with its high throughput, low latency, and scalability, enables asynchronous transmission of task metadata and intermediate results. Task metadata includes basic task information and configuration parameters, while intermediate results are the stage data generated during task execution. Through Kafka message queues, this data can flow efficiently and stably between various subsystems, avoiding blocking and latency during data transmission and improving the overall system response speed.

[0110] To clearly describe the dependencies between tasks, a Directed Acyclic Graph (DAG) is used for modeling. DAGs intuitively display the sequence and dependencies between tasks, making task scheduling and execution more rational and efficient. Based on this, the Airflow scheduling framework is used. Airflow has powerful task scheduling and monitoring capabilities, enabling cascading task triggering based on the task dependencies described by the DAG. When a task completes, Airflow automatically triggers its dependent subsequent tasks, ensuring the entire task flow proceeds in a predetermined logical order, thereby achieving efficient collaboration between multiple subsystems.

[0111] The following describes the multi-system dynamic scheduling and processing method for real-time streaming data of public services using a specific embodiment. However, it is worth noting that this specific embodiment is only for better illustrating the present invention and does not constitute an improper limitation of the present invention.

[0112] This invention, in its specific embodiments, constructs a multi-system collaborative overall architecture, including a data source layer, a data acquisition and preprocessing layer, a processing and analysis layer, a multi-system scheduling and collaboration layer, and an application and service layer. Each layer has a clear division of labor, and through system adaptation interfaces and data routing and forwarding mechanisms, seamless flow and interaction between different heterogeneous systems are achieved, ensuring that data can be efficiently processed and utilized across multiple systems.

[0113] First, set up the experimental environment.

[0114] Hardware platform: Construct an experimental cluster consisting of multiple servers, edge computing devices, and simulated data sources. The servers are equipped with high-performance CPUs, large-capacity memory, and high-speed network interfaces, while the edge computing devices are embedded hardware with certain computing capabilities.

[0115] Software environment: Deploy various open-source software and technical frameworks involved in the overall architecture mentioned above, including message queues, stream computing engines, batch processing systems, machine learning platforms, etc., to build a complete public service real-time streaming data processing experimental platform.

[0116] Data Simulation Generator: Develop data simulation generators for different public service scenarios, which can generate simulated real-time streaming data according to preset data formats, generation rates and business logic for experimental testing and performance evaluation.

[0117] Secondly, design the experimental plan.

[0118] Performance metrics selection: A series of key performance metrics, such as data acquisition latency, data processing throughput, task scheduling response time, system resource utilization, and data consistency accuracy, are determined to quantitatively evaluate the performance of the proposed solution.

[0119] Comparative experiment setup: The proposed solution is compared with traditional static scheduling and single-system processing methods. Under the same experimental environment, for different data scales, data types and business loads, the performance index data are recorded to analyze the advantages and improvement effects of the proposed solution.

[0120] Finally, the experimental results and analysis are presented.

[0121] Regarding the improvement of data processing performance: Experimental results show that the proposed solution has significant effects over traditional methods in reducing data acquisition latency, increasing data processing throughput, and shortening task scheduling response time. This is mainly due to the reasonable allocation and optimization of resources by the dynamic scheduling algorithm and the acceleration effect of edge computing on data preprocessing.

[0122] Regarding efficient utilization of system resources: Through the flexible resource allocation strategy, the system resource utilization rate has increased by an average of 70%. When the business load fluctuates greatly, the system can automatically adjust resource allocation to avoid resource waste and performance bottlenecks, thereby reducing operating costs.

[0123] This invention provides a multi-system dynamic scheduling and processing method for real-time streaming data in public services. It can handle various types of data, including structured, semi-structured, and unstructured data from different data sources such as traffic sensors, medical IoT devices, and video surveillance cameras. Furthermore, this solution is not only applicable to specific public service areas such as traffic management, healthcare, and urban security, but can also be expanded and upgraded in performance according to actual needs, adapting to constantly changing business requirements and technological advancements.

[0124] It should be understood that the phrase "one embodiment" or "an embodiment" throughout the specification means that a specific feature, structure, or characteristic related to the embodiment is included in at least one embodiment of the invention. Therefore, "in one embodiment" or "in an embodiment" appearing throughout the specification does not necessarily refer to the same embodiment. Furthermore, these specific features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. It should be understood that in the various embodiments of the invention, the sequence numbers of the above-described processes do not imply a sequential order of execution; the execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the invention. The sequence numbers of the above-described embodiments of the invention are merely descriptive and do not represent the superiority or inferiority of the embodiments.

[0125] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0126] In the several embodiments provided by this invention, it should be understood that the disclosed methods can be implemented in other ways. The methods disclosed in the several method embodiments provided by this invention can be arbitrarily combined without conflict to obtain new method embodiments. The features disclosed in the several method embodiments provided by this invention can be arbitrarily combined without conflict to obtain new method embodiments.

[0127] The above description is merely an embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method for dynamic scheduling and processing of real-time streaming data of public services across multiple systems, characterized in that, include: Data Acquisition and Preprocessing Layer: This layer acquires real-time streaming data from multiple data sources through a distributed data acquisition framework, performs preprocessing, and transmits the data; these multiple data sources include various public service data sources. Processing and Analysis Layer: Utilizes a real-time stream computing engine to perform real-time stream computing on preprocessed data, while combining with a batch processing system to perform in-depth analysis of historical data and trigger data processing tasks; Multi-system scheduling and collaboration layer: Through the task scheduling center, a dynamic scheduling strategy combining machine learning-based load prediction and multi-objective optimization scheduling model is adopted to dynamically allocate the data processing tasks to different subsystems, and realize the collaborative work between multiple subsystems through system adaptation interfaces and data routing and forwarding mechanisms. Application and Service Layer: Based on the processing and analysis results of each subsystem, it provides business applications for specific public service areas, as well as decision support systems and data sharing and open platforms; The process involves using a real-time stream computing engine to perform real-time stream computing on preprocessed data, while simultaneously combining a batch processing system to perform in-depth analysis of historical data, triggering data processing tasks. These tasks include: defining event patterns using Flink's CEP library to detect key events conforming to specific patterns from the continuous data stream transmitted from the data acquisition and preprocessing layers; using Flink's window functions to perform aggregation and statistics on the real-time streaming data corresponding to key events within a time window; using Hadoop / Hive to analyze historical incident data, predict risk areas, and obtain batch processing results; and based on the Kappa architecture, unifying the real-time stream and batch processing results through middleware to obtain a unified stream-batch data processing task. The dynamic scheduling strategy, which combines machine learning-based load prediction and a multi-objective optimization scheduling model, dynamically allocates data processing tasks to different subsystems. This includes: training a load prediction model based on historical task execution data and real-time streaming data to predict the load trends of each system; constructing a multi-objective optimization scheduling model with the objectives of minimizing response time, maximizing resource utilization, load balancing index, and task priority, while simultaneously satisfying resource capacity, task dependency, and QoS constraints; generating a Pareto optimal scheduling scheme using the non-dominated sorting genetic algorithm NSGA-III; and dynamically adjusting the allocation of data processing tasks based on the predicted load through a sliding window mechanism.

2. The multi-system dynamic scheduling processing method according to claim 1, characterized in that, The process of acquiring real-time streaming data from multiple data sources and preprocessing it using a distributed data acquisition framework includes: Kafka is used as a message queue to collect real-time streaming data from different data sources and perform preliminary format unification and standardization processing. Based on the characteristics and service requirements of the data source, an intelligent edge computing task allocation algorithm is designed to dynamically deploy edge computing nodes to perform real-time preprocessing of some data. Based on data type, the preprocessed data is cached in different caching systems and then uploaded to the data center.

3. The multi-system dynamic scheduling processing method according to claim 2, characterized in that, Based on the characteristics and service requirements of the data source, an intelligent edge computing task allocation algorithm is designed to dynamically deploy edge computing nodes to perform real-time preprocessing of some data, including: For Level 1 tasks, processing should be prioritized at edge computing nodes; For the second-level tasks, after preliminary processing at the edge computing nodes, the results or key data are uploaded to the central system for further analysis. For Level 3 tasks, the data is directly uploaded to the central system for processing. The real-time requirements for Level 1, Level 2, and Level 3 tasks decrease sequentially, while the computational load increases sequentially.

4. The multi-system dynamic scheduling processing method according to claim 3, characterized in that, The method further includes: By integrating machine learning libraries and AI model training and inference tools into edge computing nodes, historical and real-time data can be used to predict, classify, and detect anomalies in public service scenarios.

5. The multi-system dynamic scheduling processing method according to claim 1, characterized in that, The method further includes: Real-time monitoring of environmental changes and task execution status triggers model updates and strategy adjustments.

6. The multi-system dynamic scheduling processing method according to any one of claims 1 to 5, characterized in that, The system achieves collaborative operation between multiple subsystems through system adaptation interfaces and data routing and forwarding mechanisms, including: Develop standardized interfaces for different subsystems to support task submission, status query, and resource request operations; Asynchronous transmission of task metadata and intermediate results is achieved using Kafka message queues; Task dependencies are described using a directed acyclic graph, and task cascading is implemented using the Airflow scheduling framework.

Images