A data service system based on the Internet

Abstract

The present invention discloses an Internet-based data service system, which relates to the field of data management technology. A data acquisition module collects distributed sensor data in real time. A network resource management module ensures efficient transmission through dynamic congestion monitoring and path optimization. A priority-aware task scheduling module adjusts response priority according to task importance and evaluates real-time performance. A cloud processing and storage module combines high-performance computing with distributed storage to efficiently process and store data. A comprehensive analysis module comprehensively analyzes dynamic congestion and response priority offsets to evaluate the impact of network congestion on task delays. An optimization module dynamically adjusts task transmission paths and scheduling priorities based on the analysis results, significantly reducing the risk of congestion caused by large-scale concurrent traffic and improving the real-time performance and reliability of data transmission and task execution, thereby ensuring the integrity of analysis and the accuracy of decision-making.

Description

Technical Field

[0001] The present invention relates to the technical field of data management, and in particular to an Internet-based data service system. Background Art

[0002] Internet-based data services are technical solutions that provide data storage, management, transmission, and processing over the Internet. They rely on technologies such as cloud computing, big data processing, the Internet of Things (IoT), and APIs to centrally store dispersed data and make it available to users via standardized protocols. Existing technologies include data acquisition, real-time transmission, cloud storage, intelligent analysis, and various distribution methods (such as HTTP, WebSocket, or proprietary protocols). These services typically utilize distributed storage and high-performance computing to support high-concurrency and large-scale data access.

[0003] Typical implementations include cloud storage services (such as AWS S3), streaming data analytics (such as Apache Kafka), and API gateway technologies (such as RESTful APIs), which support data sharing and collaboration across regions and devices. Users can access this data directly through browsers, mobile apps, or other clients, without having to worry about the underlying hardware and infrastructure. For example, smart cities rely on sensor networks to collect real-time data on traffic flow, air quality, and energy consumption. This data is then uploaded to a cloud-based processing platform via the Internet of Things. Internet-based data services analyze this data in real time and push it to traffic management departments for use in optimizing traffic light control or issuing congestion warnings.

[0004] The existing technology has the following shortcomings:

[0005] During peak periods, sensor networks may generate massive amounts of data (such as traffic flow and air quality monitoring data). When millions of sensors transmit data to the cloud simultaneously, network congestion and transmission delays may occur, and the data may not be uploaded to the processing platform in a timely manner. Especially in scenarios that rely on public networks (such as 5G or Wi-Fi), network traffic overload may cause partial data loss or large-scale service interruptions. In addition, if the peak demand for large-scale concurrent data traffic cannot be effectively handled, peak network congestion may cause the permanent loss of data generated by some sensors, directly weakening the integrity and accuracy of the analysis and having a long-term negative impact on subsequent management decisions. Summary of the Invention

[0006] The purpose of the present invention is to provide an Internet-based data service system to address the deficiencies in the background technology.

[0007] In order to achieve the above-mentioned object, the present invention provides the following technical solutions: an Internet-based data service system, comprising a data acquisition module, a network resource management module, a priority-aware task scheduling module, a cloud processing and storage module, a comprehensive analysis module, and an optimization module;

[0008] Data acquisition module, used to collect raw data from various distributed sensors in real time;

[0009] Network resource management module, used to monitor and optimize data transmission paths, and perform real-time path adjustments by calculating dynamic congestion;

[0010] Priority-aware task scheduling module, which dynamically adjusts the response priority of tasks according to their importance and evaluates the real-time performance of task execution by calculating the service response priority offset;

[0011] Cloud processing and storage module, including distributed storage system and high-performance computing platform, is used to process and store uploaded dynamic congestion data and response priority offset data;

[0012] Comprehensive analysis module, used to conduct comprehensive analysis based on dynamic congestion and service response priority offset to determine the severity of the impact of network congestion on task response delay;

[0013] The optimization module immediately switches the transmission of high-priority tasks to the low DCL path if the severity is high, and inserts high-priority tasks into the task scheduling queue first, delaying the processing of low-priority tasks; if the severity is low, reduces the data transmission rate of low-priority tasks and improves the sorting position of high-priority tasks in the queue.

[0014] Preferably, in the network resource management module, n paths are set, and the network performance of each path is described by k indicators to form a data matrix: Among them, X is an n×k matrix, representing the k network performance indicators of n paths, x nk Represents the kth indicator value of the nth path, normalizing each column of data to zero mean and unit variance: Among them, μ j is the mean of the jth index, σ j is the standard deviation of the j-th indicator, and the standardized data matrix is Z: The covariance matrix Σ is used to describe the correlation between different indicators: Where Σ is a k×k matrix, and the elements σ in the matrix are ij Represents the covariance of the i-th indicator and the j-th indicator, and performs eigenvalue decomposition on the covariance matrix Σ: Σv i =λ i v i ; Among them, λi is the i-th eigenvalue, representing the proportion of the explained variance of the corresponding principal component, v i is the i-th eigenvector, representing the direction of the i-th principal component. Arranged in descending order of eigenvalues, the eigenvector v1 corresponding to the largest eigenvalue is selected as the first principal component, and the dynamic congestion degree is calculated. The expression is: In the formula, DCL is the dynamic congestion degree, v 1j is the j-th eigenvector component corresponding to the first principal component.

[0015] Preferably, in the network resource management module, a DCL threshold is set, and the paths are classified according to the calculation results: Low congestion path: DCL ≤ 0.5; Moderate congestion path: 0.5 < DCL ≤ 0.8; High congestion path: DCL > 0.8; Obtain the real-time DCL data of all available paths in the system, compare the DCL values, and preferentially select the path with the lowest DCL as the alternative. High-priority tasks must be assigned to low-congestion paths. If there are no low-congestion paths, the backup network is enabled; Medium / low-priority tasks are assigned to moderate-congestion paths or sub-optimal paths, and traffic limiting is performed on high-congestion paths to gradually release bandwidth resources.

[0016] Preferably, in the priority-aware task scheduling module, after analyzing the service response priority offset situation, a response priority offset index is generated to evaluate the real-time performance of task execution. The method for obtaining the response priority offset index is:

[0017] Set the actual response time as T actual,i , representing the actual completion time of the task, set the expected response time as T expected,i , representing the target time when the task should be completed within the specified time, set the service response offset value SRPD i , the offset value calculated according to the task response time, and the calculation expression is: SRPD i = T actual,i - T expected,i ; The higher the priority of the task, the greater the weight. Set the allocation rule of the priority P weight w as: P = 3 high priority, w = 0.6; P = 2 medium priority, w = 0.3; P = 1 low priority, w = 0.1; According to the task priority and the corresponding offset value, calculate the total weight W total of each task, and the expression is: In the formula, w i is the task priority weight, n is the total number of tasks. Summing up the weighted offset values of all tasks and dividing by the total weight, the response priority offset index is obtained, and the expression is: In the formula, SRPDI is the response priority offset index.

[0018] Preferably, in the comprehensive analysis module, a comprehensive analysis is performed in combination with the dynamic congestion degree and the service response priority offset to determine the severity of the impact of network congestion on task response delay, specifically:

[0019] The dynamic congestion degree and response priority offset index are converted into comprehensive feature vectors, which are used as inputs of the machine learning model. The machine learning model uses each set of comprehensive feature vectors to predict the severity value label of the impact of network congestion on task response delay as the prediction target, and takes minimizing the sum of the prediction errors of all severity value labels of the impact of network congestion on task response delay as the training target. The machine learning model is trained until the sum of the prediction errors reaches convergence, and the model training is stopped. The severity value of the impact of network congestion on task response delay is determined based on the model output results. The machine learning model is a polynomial regression model.

[0020] Preferably, in the optimization module, the acquired severity value of the impact of network congestion on task response delay is compared with the severity reference threshold set according to historical data. If the severity value of the impact of network congestion on task response delay is greater than or equal to the set severity reference threshold, it means that the severity of the impact of network congestion on task response delay is high. At this time, an early warning signal is generated, and the transmission of high-priority tasks is immediately switched to a low DCL path, and high-priority tasks are preferentially inserted into the task scheduling queue. If the severity value of the impact of network congestion on task response delay is less than the set severity reference threshold, it means that the severity of the impact of network congestion on task response delay is low. At this time, no early warning signal is generated. At this time, the data transmission rate of low-priority tasks should be reduced and the sorting position of high-priority tasks in the queue should be increased.

[0021] Preferably, the transmission of high-priority tasks is switched to the low DCL path, and high-priority tasks are preferentially inserted into the task scheduling queue, specifically:

[0022] Switch the transmission path of the high-priority task to the path with the lowest dynamic congestion level DCL to ensure that its transmission delay is minimized. The formula is: Where, P opt is the optimal transmission path, P is the set of all available paths, and DCL(P) is the dynamic congestion degree of path P; collect the DCL(P) values of all paths in the current path set P, and calculate the congestion degree of the path P according to the public Select the path P with the lowest dynamic congestion opt , the high priority task T high The data flow is transferred to path P opt :

[0023] The high priority task T highInsert the priority position of the task scheduling queue Q to ensure the real-time execution of high-priority tasks. The formula is: Q = Insert (Q, T high , priority); Q is the task scheduling queue, T high For high-priority tasks that need to be inserted, priority is the priority attribute used for sorting adjustment. The queue sorting rule is: Sort(Q)=argmax T∈ Q priority(T); where priority(T) is the priority value of task T. The higher the value, the higher the task priority. Define the priority weight priority(T) and set the priority weight for each task T: w high >w low Indicates that the weight of high priority tasks is greater than that of low priority tasks, w high 、w low Represents the weights of high priority and low priority tasks respectively. According to the task priority, T high Insert it into the priority position in the queue and sort the task queue Q from high to low according to the priority value.

[0024] Preferably, the data transmission rate of low-priority tasks is reduced and the sorting position of high-priority tasks in the queue is improved, specifically:

[0025] Adjust the data transfer rate R of low priority tasks low , ensuring the real-time performance of high-priority tasks and the new transmission rate of low-priority tasks: Where, is the adjusted low priority task data transmission rate, is the data transmission rate of the low-priority task before adjustment, α is the rate reduction coefficient, and the value range is 0<α<1. The transmission rate of the low-priority task is dynamically adjusted according to the calculation formula, and the high-priority task T is adjusted by reordering the task scheduling queue Q. high Move to the front position, prioritize computing resources and execution time, and schedule queue reordering rule: Sort(Q) = argmax T∈Q priority(T); Task priority definition function: w high 、w mid 、w low is the task priority weight, where w high >w mid >w low , T mid For medium priority tasks; define the priority weight w for each task high 、w mid 、wlow , sort the task queue Q according to the priority value, put high priority tasks at the front, and generate a new task scheduling queue.

[0026] In the above technical solution, the technical effects and advantages provided by the present invention are:

[0027] 1. The present invention realizes real-time acquisition of multi-source data through the data acquisition module, the network resource management module combines dynamic congestion level (DCL) to perform real-time path adjustment, the priority-aware task scheduling module dynamically adjusts the task response priority according to the task importance and evaluates the real-time performance, the cloud processing and storage module adopts distributed storage and high-performance computing to ensure efficient data processing and reliable storage, the comprehensive analysis module combines DCL and response priority offset index to analyze the severity of the impact of network congestion on task delay, and the optimization module dynamically adjusts the transmission path and task scheduling strategy according to the analysis results, thereby improving the stability and real-time performance of the system in high-concurrency scenarios.

[0028] 2. This invention significantly reduces transmission delays and the risk of data loss caused by network traffic overload, ensures timely response to high-priority tasks, optimizes system resource utilization and the fairness of task scheduling. The system can dynamically adapt to changes in the network environment, effectively cope with peak loads, and improve overall performance through multi-level data optimization strategies. It provides reliable and scalable technical support for smart cities, the industrial internet, and other fields, and has extremely high practical value and promotion prospects. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments described in the present invention. For ordinary technicians in this field, other drawings can also be obtained based on these drawings.

[0030] Figure 1 It is a system module diagram of the present invention. DETAILED DESCRIPTION

[0031] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0032] For examples, see Figure 1As shown, the Internet-based data service system described in this embodiment includes a data acquisition module, a network resource management module, a priority-aware task scheduling module, a cloud processing and storage module, a comprehensive analysis module, and an optimization module;

[0033] Data acquisition module, used to collect raw data from various distributed sensors in real time;

[0034] Network resource management module, used to monitor and optimize data transmission paths, and perform real-time path adjustments by calculating dynamic congestion;

[0035] Priority-aware task scheduling module, which dynamically adjusts the response priority of tasks according to their importance and evaluates the real-time performance of task execution by calculating the service response priority offset;

[0036] Cloud processing and storage module, including distributed storage system and high-performance computing platform, is used to process and store uploaded dynamic congestion data and response priority offset data;

[0037] Comprehensive analysis module, used to conduct comprehensive analysis based on dynamic congestion and service response priority offset to determine the severity of the impact of network congestion on task response delay;

[0038] The optimization module immediately switches the transmission of high-priority tasks to the low DCL path if the severity is high, and inserts high-priority tasks into the task scheduling queue first, delaying the processing of low-priority tasks; if the severity is low, reduces the data transmission rate of low-priority tasks and improves the sorting position of high-priority tasks in the queue.

[0039] In the data acquisition module, the sensor interface unit communicates with distributed sensors, supporting various hardware interfaces and communication protocols. Supported protocols include: Lightweight protocols: MQTT and CoAP (suitable for low-power devices); Traditional network protocols: HTTP / HTTPS (for general internet devices); Industrial Internet protocols: Modbus, Zigbee, and LoRa (suitable for industrial environments and long-distance, low-power communication). Adapter design: A built-in protocol converter ensures that data from different devices and protocols can be parsed by the system in a standardized format.

[0040] The data filtering unit performs preliminary screening of the collected raw data, removing redundant or obviously erroneous data to improve data quality. Methods include: data integrity verification (e.g., checksum validation), rule-based filtering (e.g., removing invalid data packets and noisy data), and preliminary validity checks (e.g., value range checks).

[0041] Time Synchronization Unit: Ensures consistent timestamps for data collected from distributed sensors, supporting time series analysis and comparison. Technical Implementation: Synchronizes with sensors based on NTP (Network Time Protocol). Local timestamps are added to data transmission to ensure time consistency of uploaded data.

[0042] The data encoding unit standardizes sensor data in various formats to facilitate subsequent transmission and analysis. Encoding methods include: JSON, Protobuf (lightweight, cross-platform support), and custom binary formats (suitable for efficient communication scenarios).

[0043] The data security unit encrypts data during collection and transmission to prevent data leakage or tampering during transmission. Technical implementation: Uses the TLS protocol for data link encryption. Data signing and verification (such as HMAC-based integrity verification) are also implemented.

[0044] Supports simultaneous connection of multiple sensor types (such as traffic flow sensors, air quality monitoring equipment, and energy consumption meters). Merges data streams from multiple sensors to generate a unified real-time data stream. Achieves millisecond-level data collection cycles to meet high-real-time requirements (such as traffic light optimization). Supports on-demand adjustment of collection frequency to accommodate different scenarios (such as high-frequency traffic data versus low-frequency environmental data).

[0045] After data collection, some non-core data is filtered and compressed locally to reduce the amount of data uploaded to the cloud and improve transmission efficiency. Example: Extract metadata (such as vehicle counts) from a traffic video stream without uploading the entire video.

[0046] The data acquisition module establishes communication with distributed sensors through the sensor interface unit. It receives real-time data packets from the sensors. The data filtering unit verifies the data packets, filtering out noise and invalid data. This ensures that the uploaded data is complete, accurate, and meaningful. It adds a unified timestamp to the data to ensure temporal consistency across multiple sources. It encodes the data into a standard format (such as JSON) for easy transmission and processing. It encrypts the data and appends integrity verification information. The encrypted data is then uploaded to the network layer.

[0047] Asynchronous I / O technologies (such as libuv and Netty) enable simultaneous connection and data collection from millions of sensors, ensuring system performance in high-concurrency scenarios. If a sensor connection is lost, the module automatically reconnects and resumes data collection. In case of network instability, data is temporarily cached and uploaded after the network is restored to avoid data loss.

[0048] The network resource management module is used to monitor and optimize data transmission paths and perform real-time path adjustments by calculating dynamic congestion.

[0049] Dynamic congestion calculation quantifies the load on network paths in real time by comprehensively analyzing network performance indicators (such as traffic utilization, latency, and packet loss rate). This unified metric enables the network resource management module to quickly identify high-load paths, avoiding transmission delays and data loss caused by path congestion, thereby improving system transmission efficiency and responsiveness. Furthermore, dynamic congestion provides data support for optimizing multi-path transmission, enabling the system to intelligently select the optimal path among multiple available paths, effectively reducing the risk of network outages and service quality degradation.

[0050] Dynamic congestion calculation is particularly critical in large-scale concurrent scenarios, providing a quantitative basis for real-time traffic control and resource allocation decisions. For example, in intelligent transportation systems, dynamic congestion can be used to dynamically adjust the transmission paths of high-priority incident data, ensuring that critical data is transmitted preferentially along less congested paths, reducing latency. Furthermore, dynamic congestion can be used to monitor the long-term operational status of the network. By analyzing congestion trends, it can help operators identify potential bottlenecks, optimize network architecture, and improve overall network service quality and reliability.

[0051] Assume there are n paths, and the network performance of each path is described by k indicators, forming a data matrix: Among them, X is an n×k matrix, representing the k network performance indicators of n paths, x nk Indicates the kth metric value of the nth path. Common metrics include: x1 is the link traffic utilization, x2 is the link delay ratio, and x3 is the packet loss rate P.

[0052] In order to eliminate the influence of different indicator dimensions, each column of data is standardized to zero mean and unit variance: Among them, μ j is the mean of the jth index, σ j is the standard deviation of the j-th indicator, and the standardized data matrix is Z: The covariance matrix Σ is used to describe the correlation between different indicators: Where Σ is a k×k matrix, and the elements σ in the matrix are ij Represents the covariance of the i-th indicator and the j-th indicator, and performs eigenvalue decomposition on the covariance matrix Σ: Σv i =λ i v i ; Among them, λ i is the i-th eigenvalue, indicating the proportion of explained variance of the corresponding principal component, v i is the i-th eigenvector, indicating the direction of the i-th principal component. Arrange them from large to small according to their eigenvalues, select the eigenvector v1 corresponding to the largest eigenvalue as the first principal component, and calculate the dynamic congestion degree. The expression is: where DCL is the dynamic congestion level, and v 1j is the j-th eigenvector component corresponding to the first principal component.

[0053] Set the DCL threshold, and classify the paths according to the calculation results: Low congestion path: DCL ≤ 0.5; Moderate congestion path: 0.5 < DCL ≤ 0.8; High congestion path: DCL > 0.8;

[0054] Obtain the real-time DCL data of all available paths in the system. Compare the DCL values, and preferentially select the path with the lowest DCL as the alternative. High-priority tasks: Must be assigned to low congestion paths. If there is no low congestion path, the backup network (such as a dedicated communication channel) can be enabled. Medium / low-priority tasks: Be assigned to moderate congestion paths or sub-optimal paths. Perform traffic limiting on high congestion paths to gradually release bandwidth resources.

[0055] Use a network controller (such as an SDN controller) to modify the routing table, and switch the task flow from high congestion paths to low congestion paths: Modify the forwarding table entries of the target path so that the traffic is forwarded according to the new path. Preferentially process high-priority data packets to ensure the real-time performance of latency-sensitive data. Shunt the remaining data streams: Assign high-priority tasks to backup paths or low congestion paths. Low-priority tasks adopt a multi-path transmission strategy to disperse the data into multiple paths. Use load balancing algorithms (such as weighted round-robin, least connections) to dynamically adjust the data stream allocation ratio of each path.

[0056] Reserve dedicated bandwidth for tasks with different priorities: High-priority tasks: Reserve more bandwidth to avoid being preempted by low-priority traffic. Low-priority tasks: Limit their maximum bandwidth occupancy to reduce their impact on path resources. Adopt flow control algorithms (such as Token Bucket or Leaky Bucket) to control the transmission rate of high congestion paths and gradually relieve the congestion state. After the adjustment is completed, continue to monitor the DCL data of the paths to verify the congestion mitigation effect. If the expected effect is still not achieved after the path switch: Re-evaluate other alternative paths. Dynamically adjust the path switch frequency and traffic allocation strategy. Store the adjustment records in the system log for subsequent training of the machine learning model to improve the path optimization strategy.

[0057] A priority-aware task scheduling module, which is used to dynamically adjust the response priority of tasks according to the importance of the tasks, and evaluate the real-time performance of task execution by calculating the service response priority offset.

[0058] Based on their importance and timeliness, tasks are categorized into three priority levels: high, medium, and low. High-priority tasks: Tasks requiring real-time response, such as emergency response and disaster warning. Medium-priority tasks: Routine monitoring data (such as traffic flow monitoring). Low-priority tasks: Periodic or batch tasks (such as uploading environmental data). Task priorities are adjusted in real time based on system load and task characteristics.

[0059] By quantifying the deviation between actual and expected response times: Small deviations indicate that task scheduling is normal and the response meets expectations. Large deviations indicate that task processing may be affected by network or resource bottlenecks and requires optimization. Dynamically allocate system resources (bandwidth, computing resources, etc.) based on task priority, prioritizing high-priority tasks. Implement throttling, delaying, or batch scheduling for low-priority tasks.

[0060] Static priority scheduling: The task priority is determined when the task is submitted, and the priority remains unchanged during scheduling. Dynamic priority scheduling: The priority is adjusted in real time according to the task response, time limit or system load: Preemptive scheduling: High-priority tasks can interrupt the execution of low-priority tasks. Non-preemptive scheduling: The executed tasks are not interrupted, but the high-priority tasks are given priority in resource allocation. The offset SRPD calculation formula is: SRPD = T actual -T expected ;T actual is the actual response time of the task, T expected is the expected response time of the task.

[0061] Resources are dynamically adjusted according to the priority allocation ratio, for example: high priority: 60% resources, medium priority: 30% resources, low priority: 10% resources.

[0062] After analyzing the service response priority deviation, a response priority deviation index is generated to evaluate the real-time performance of task execution. The response priority deviation index is obtained as follows:

[0063] Set the actual response time to T actual,i , represents the actual completion time of the task, and sets the expected response time as T expected,i , indicating the target time that the task should be completed within the specified time. Set the service response offset value SRPD i , the offset value calculated based on the task response time, the calculation expression is: SRPD i =T actual,i -T expected,iThe higher the priority, the greater the weight of the task, reflecting its impact on the overall performance of the system. The weight can be set manually according to the specific scenario, or a dynamic adjustment strategy can be adopted. The allocation rule of setting priority P weight w is: P = 3 (high priority), w = 0.6; P = 2 (medium priority), w = 0.3; P = 1 (low priority), w = 0.1; Based on the task priority and the corresponding offset value, calculate the total weight W of each task total , the expression is: Where w i is the task priority weight, reflecting the importance of the task, n is the total number of tasks, the weighted offset values of all tasks are summed and divided by the total weight to obtain the response priority offset index, which is expressed as: Where SRPDI is the response priority deviation index.

[0064] A large response priority deviation index (SRPDI) indicates that the actual response time of tasks in the system generally deviates significantly from the expected time, with high-priority tasks experiencing significant response delays. This typically indicates poor real-time performance of task execution, potentially due to network congestion, uneven resource allocation, or an imperfect task scheduling mechanism. A large deviation index can also lead to a decrease in overall system efficiency, especially in scenarios with a high proportion of critical tasks. Real-time performance issues can directly impact task execution stability and service quality.

[0065] When the response priority deviation index (SRPDI) is small, it indicates that the actual response time of tasks in the system is close to the expected time, especially that high-priority tasks are processed promptly. This indicates that tasks are executed in real time and that the system scheduling mechanism can effectively cope with multi-task concurrency and resource competition. Furthermore, a small deviation index also reflects the rationality of the resource allocation strategy, with low delays for high-priority tasks and controllable delays for low-priority tasks, helping to maintain efficient and stable system operation.

[0066] The cloud processing and storage module, including a distributed storage system and a high-performance computing platform, is used to process and store uploaded dynamic congestion data and response priority offset data.

[0067] Distributed storage system: used to store dynamic congestion data, response priority offset data, and other related metadata. Ensure high availability, scalability, and persistence of data. Multi-copy storage: Multiple copies of each data item are stored on different nodes to prevent single point failures. Distributed file systems: such as HDFS (Hadoop Distributed File System) and Ceph, used to manage massive amounts of data. Key-value storage: such as Apache Cassandra, used to quickly store and query dynamic indicator data (such as DCL and SRPD). Tiered storage of hot and cold data: DCL and SRPD data required for real-time analysis is stored in high-performance storage, and historical data is stored in low-cost storage devices.

[0068] High-performance computing platform: Processes uploaded data in real time, calculates new dynamic metrics (such as comprehensive DCL and SRPDI), and performs large-scale batch analysis (such as task latency trend prediction).

[0069] Streaming computing frameworks, such as Apache Flink and Apache Kafka Streams, are used to calculate the global average of dynamic congestion or the latency distribution of high-priority tasks in real time. Parallel computing engines, such as Apache Spark, are used for batch processing and analysis of response offset data for large-scale tasks. Containerized deployments, such as Kubernetes, enable dynamic scalability to meet concurrent processing needs.

[0070] Receive DCL and SRPD data streams uploaded by the network resource management module and priority scheduling module through a message queue (such as Apache Kafka). Remove duplicate, erroneous, or abnormal data to ensure data quality. Calculate the global DCL distribution and identify highly congested paths. Calculate the SRPDI to assess the overall real-time performance of the system. Calculation results are temporarily stored in a high-performance in-memory database (such as Redis) for real-time access by other modules.

[0071] Analyze long-term latency trends for high-priority tasks using historical data to optimize resource allocation strategies. Metric aggregation: Calculate the average DCL and SRPDI for each path for overall system performance evaluation. Write the latest DCL and SRPDI data to a distributed database (such as Apache Cassandra or MongoDB). Transfer historical data to low-cost storage (such as Amazon S3 or Google Cloud Storage) for long-term analysis and model training.

[0072] The comprehensive analysis module is used to conduct a comprehensive analysis based on the dynamic congestion level and service response priority offset to determine the severity of the impact of network congestion on task response delay.

[0073] The dynamic congestion degree and response priority offset index are converted into comprehensive feature vectors, which are used as inputs of the machine learning model. The machine learning model uses each set of comprehensive feature vectors to predict the severity value label of the impact of network congestion on task response delay as the prediction target, and takes minimizing the sum of the prediction errors of all severity value labels of the impact of network congestion on task response delay as the training target. The machine learning model is trained until the sum of the prediction errors reaches convergence, and the model training is stopped. The severity value of the impact of network congestion on task response delay is determined based on the model output results. The machine learning model is a polynomial regression model.

[0074] The method for obtaining the severity value of the impact of network congestion on task response delay is as follows: from the comprehensive feature vector training data of the trained machine learning model, the corresponding function expression is obtained: LQ = F(DCL, SRPDI); where F is the output function of the model, DCL is the dynamic congestion level, SRPDI is the response priority offset index, and LQ is the severity value of the impact of network congestion on task response delay.

[0075] The optimization module immediately switches the transmission of high-priority tasks to the low DCL path if the severity is high, and inserts high-priority tasks into the task scheduling queue first, delaying the processing of low-priority tasks; if the severity is low, reduces the data transmission rate of low-priority tasks and improves the sorting position of high-priority tasks in the queue.

[0076] The obtained severity value of the impact of network congestion on task response delay is compared with the severity reference threshold set according to historical data. If the severity value of the impact of network congestion on task response delay is greater than or equal to the set severity reference threshold, it means that the severity of the impact of network congestion on task response delay is high. At this time, an early warning signal is generated, and the transmission of high-priority tasks is immediately switched to the low DCL path, and high-priority tasks are preferentially inserted into the task scheduling queue. If the severity value of the impact of network congestion on task response delay is less than the set severity reference threshold, it means that the severity of the impact of network congestion on task response delay is low. At this time, no early warning signal is generated. At this time, the data transmission rate of low-priority tasks should be reduced and the sorting position of high-priority tasks in the queue should be increased.

[0077] Switch the transmission of high-priority tasks to the low DCL path and give priority to inserting high-priority tasks into the task scheduling queue. Specifically:

[0078] Switch the transmission path of the high-priority task to the path with the lowest dynamic congestion level DCL to ensure that its transmission delay is minimized. The formula is: Where, P optis the optimal transmission path, P is the set of all available paths, and DCL(P) is the dynamic congestion degree of path P.

[0079] Collect the DCL(P) values of all paths in the current path set P, and Select the path P with the lowest dynamic congestion opt , the high priority task T high The data flow is transferred to path P opt :

[0080] The high priority task T high Insert the priority position of the task scheduling queue Q to ensure the real-time execution of high-priority tasks. The formula is: Q = Insert (Q, T high , priority); Q is the task scheduling queue, T high For high-priority tasks that need to be inserted, priority is the priority attribute used for sorting adjustment. The queue sorting rule is: Sort(Q)=argmax T∈ Q priority(T); where priority(T) is the priority value of task T. The higher the value, the higher the task priority. Define the priority weight priority(T) and set the priority weight for each task T: w high >w low Indicates that the weight of high priority tasks is greater than that of low priority tasks, w high 、w low Represents the weights of high priority and low priority tasks respectively. According to the task priority, T high Insert it into the priority position in the queue and sort the task queue Q from high to low according to the priority value.

[0081] Reduce the data transfer rate of low-priority tasks and improve the ranking position of high-priority tasks in the queue, specifically:

[0082] Adjust the data transfer rate R of low priority tasks low , freeing up more bandwidth resources for high-priority tasks, ensuring the real-time performance of high-priority tasks, and the new transmission rate of low-priority tasks: Where, is the adjusted low priority task data transmission rate, is the data transmission rate of the low-priority task before adjustment, α is the rate reduction coefficient, the value range is 0<α<1, α=0.5 means the transmission rate is halved. The transmission rate of the low-priority task is dynamically adjusted according to the calculation formula. By reordering the task scheduling queue Q, the high-priority task T highMove to the front position, prioritize computing resources and execution time, and schedule queue reordering rule: Sort(Q) = argmax T∈Q priority(T); Task priority definition function: w high 、w mid 、w low is the task priority weight, where w high >w mid >w low , T mid For medium priority tasks; define the priority weight w for each task high 、w mid 、w low , sort the task queue Q according to the priority value, put high priority tasks at the front, and generate a new task scheduling queue.

[0083] In this embodiment, the data acquisition module is used to collect raw data from multiple distributed sensors in real time, providing basic data support for task execution. The network resource management module optimizes network resource allocation by monitoring data transmission paths, calculating the dynamic congestion level (DCL), and adjusting transmission paths in real time. The priority-aware task scheduling module dynamically adjusts response priorities based on task importance and evaluates the real-time performance of tasks by calculating the service response priority offset (SRPDI). The cloud processing and storage module combines a distributed storage system with a high-performance computing platform to process and store uploaded dynamic congestion level data and response priority offset data, ensuring data reliability and efficiency. The comprehensive analysis module comprehensively analyzes the dynamic congestion level and service response priority offset to determine the severity of the impact of network congestion on task response delay. Based on the analysis results, the optimization module immediately switches high-priority tasks to low-DCL paths if the severity is high, and prioritizes high-priority tasks in the scheduling queue, delaying low-priority tasks. If the severity is low, the transmission rate of low-priority tasks is reduced and the queue order of high-priority tasks is increased, thereby dynamically optimizing resource allocation and improving the real-time performance and efficiency of the system.

[0084] The above formulas are all dimensionless and numerical calculations. The formulas are obtained by collecting a large amount of data and performing software simulation to obtain the most recent real situation. The preset parameters in the formulas are set by technicians in this field according to actual conditions.

[0085] The above embodiments can be implemented in whole or in part by software, hardware, firmware or any other combination. When implemented using software, the above embodiments can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer program are loaded or executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center via a wired (e.g., infrared, wireless, microwave, etc.) method. The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or data center that contains one or more available media sets. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium. The semiconductor medium can be a solid-state drive.

[0086] The above is only a specific implementation method of the present application, but the scope of protection of the present application is not limited thereto. Any technician familiar with this technical field can easily think of changes or replacements within the technical scope disclosed in this application, which should be covered by the scope of protection of the present application.

Claims

1. An Internet-based data service system, characterized by: It includes a data acquisition module, a network resource management module, a priority-aware task scheduling module, a cloud processing and storage module, a comprehensive analysis module, and an optimization module; The data acquisition module is used to collect raw data in real time from a variety of distributed sensors; The network resource management module is used to monitor and optimize the data transmission path, and perform real-time path adjustment by calculating the dynamic congestion degree; The priority-aware task scheduling module is used to dynamically adjust the response priority of tasks according to their importance and evaluate the real-time performance of task execution by calculating the service response priority offset. Specifically, the module generates a response priority offset index after analyzing the service response priority offset. The expression of the response priority offset index is: Where, Response priority offset index, n is the total number of tasks, It represents the actual response time of the i-th task minus the expected response time. Represents the pre-set weight coefficient of the i-th task priority, represents the total weight of the tasks; The cloud processing and storage module includes a distributed storage system and a high-performance computing platform, and is used to process and store the uploaded dynamic congestion degree data and response priority offset data; The comprehensive analysis module is used to conduct a comprehensive analysis by combining the dynamic congestion degree and the service response priority offset, and determine the severity of the impact of network congestion on task response latency, including converting the dynamic congestion degree and the response priority offset index into a comprehensive feature vector, and using the comprehensive feature vector as the input of a machine learning model; In the optimization module, if the severity is high, immediately switch the transmission of high-priority tasks to a low-DCL path, and preferentially insert high-priority tasks into the task scheduling queue, and delay the processing of low-priority tasks; if the severity is low, reduce the data transmission rate of low-priority tasks and increase the sorting position of high-priority tasks in the queue.

2. The Internet-based data service system according to claim 1, wherein: In the network resource management module, n paths are set, and the network performance of each path is described by k indicators, forming a data matrix: Where X is an n×k matrix, representing the k network performance indicators of n paths, Represents the k-th indicator value of the n-th path, normalizing each column of data to the mean and unit variance: in, is the mean value of the jth indicator, is the standard deviation of the j-th indicator, and the standardized data matrix is Z: The covariance matrix Σ is used to describe the correlation between different indicators: ; Where Σ is a k×k matrix, and the elements in the matrix are Represents the covariance of the i-th indicator and the j-th indicator, and performs eigenvalue decomposition on the covariance matrix Σ: ;in, is the i-th eigenvalue, indicating the proportion of explained variance of the corresponding principal component, is the i-th eigenvector, indicating the direction of the i-th principal component, arranged from large to small according to the eigenvalue, and the eigenvector corresponding to the largest eigenvalue is selected , as the first principal component, calculate the dynamic congestion degree, the expression is: ; Where DCL is the dynamic congestion degree, is the jth eigenvector component corresponding to the first principal component.

3. The Internet-based data service system according to claim 2, wherein: In the network resource management module, set the DCL threshold, and classify the paths according to the calculation results: low congestion path: DCL ≤ 0.5; medium congestion path: 0.5 < DCL ≤ 0.8; high congestion path: DCL > 0.8; Obtain the real-time DCL data of all available paths in the system, compare the DCL values, and preferentially select the path with the lowest DCL as the alternative. High-priority tasks must be assigned to low congestion paths. If there is no low congestion path, enable the backup network; medium / low-priority tasks are assigned to medium congestion paths or sub-optimal paths, and traffic limiting is performed on high congestion paths to gradually release bandwidth resources.

4. The Internet-based data service system according to claim 1, wherein: In the priority-aware task scheduling module, after analyzing the service response priority offset situation, generate a response priority offset index to evaluate the real-time performance of task execution. The method for obtaining the response priority offset index is as follows: Set the actual response time to , represents the actual completion time of task i, and sets the expected response time to , which indicates the target time that task i should be completed within the specified time, and sets the service response offset value , the offset value calculated based on the task response time is calculated as follows: The higher the priority, the greater the weight of the task. The rule for assigning priority P and weight w is: P=3 high priority, w=0.6; P=2 medium priority, w=0.3; P=1 low priority, w=0.1; Calculate the total weight of each task based on the task priority and the corresponding offset value. , the expression is: Where, is the task priority weight, n is the total number of tasks, the weighted offset values of all tasks are summed and divided by the total weight to obtain the response priority offset index, which is expressed as: Where, The response priority offset index.

5. The Internet-based data service system according to claim 4, characterized in that: In the comprehensive analysis module, conduct a comprehensive analysis by combining the dynamic congestion degree and the service response priority offset, and determine the severity of the impact of network congestion on task response latency, specifically: Convert the dynamic congestion degree and the response priority offset index into a comprehensive feature vector, and use the comprehensive feature vector as the input of a machine learning model. The machine learning model takes predicting the severity value label of the impact of network congestion on task response latency for each group of comprehensive feature vectors as the prediction target, and takes minimizing the sum of the prediction errors of all severity value labels of the impact of network congestion on task response latency as the training target, and trains the machine learning model until the sum of the prediction errors reaches convergence and then stops the model training. Determine the severity value of the impact of network congestion on task response latency according to the model output result. Among them, the machine learning model is a polynomial regression model.

6. The Internet-based data service system according to claim 1, characterized in that: In the optimization module, the obtained severity value of the impact of network congestion on task response delay is compared with the severity reference threshold set according to historical data. If the severity value of the impact of network congestion on task response delay is greater than or equal to the set severity reference threshold, it means that the severity of the impact of network congestion on task response delay is high. At this time, an early warning signal is generated, and the transmission of high-priority tasks is immediately switched to the low DCL path, and high-priority tasks are preferentially inserted into the task scheduling queue. If the severity value of the impact of network congestion on task response delay is less than the set severity reference threshold, it means that the severity of the impact of network congestion on task response delay is low. At this time, no early warning signal is generated. At this time, the data transmission rate of low-priority tasks should be reduced and the sorting position of high-priority tasks in the queue should be increased.

7. The Internet-based data service system according to claim 6, characterized in that: Switch the transmission of high-priority tasks to the low DCL path and give priority to inserting high-priority tasks into the task scheduling queue. Specifically: Switch the transmission path of the high-priority task to the path with the lowest dynamic congestion level DCL to ensure that its transmission delay is minimized. The formula is: Where, is the optimal transmission path, is the set of all available paths, DCL(P) is the dynamic congestion of path P; collect the current path set The DCL(P) values of all paths in Select the path with the lowest dynamic congestion , high priority tasks The data flow is transferred to the path :High priority tasks Insert the priority position of the task scheduling queue Q to ensure the real-time execution of high-priority tasks. The formula is: ; Q is the task scheduling queue, For high-priority tasks that need to be inserted, priority is the priority attribute used for sorting adjustment. The queue sorting rules are: ; In the formula, priority(T) is the priority value of task T. The higher the value, the higher the task priority. Define the priority weight priority(T) and set the priority weight for each task T: ; > Indicates that the weight of high priority tasks is greater than that of low priority tasks. Represents the weights of high priority and low priority tasks respectively. According to the task priority, Insert it into the priority position in the queue and sort the task queue Q from high to low according to the priority value.

8. The Internet-based data service system according to claim 7, characterized in that: Reduce the data transfer rate of low-priority tasks and improve the ranking position of high-priority tasks in the queue, specifically: Adjust the data transfer rate of low priority tasks , ensuring the real-time performance of high-priority tasks and the new transmission rate of low-priority tasks: Where, is the adjusted low priority task data transmission rate, is the data transmission rate of the low-priority task before adjustment, α is the rate reduction coefficient, and the value range is 0<α<1. The transmission rate of the low-priority task is dynamically adjusted according to the calculation formula, and the high-priority task is scheduled to be sent to the queue Q by reordering the task. Move to the front position, give priority to computing resources and execution time, and reorder the scheduling queue rules: ;Task priority definition function: ; is the task priority weight, where For medium priority tasks; define the priority weight of each task , sort the task queue Q according to the priority value, put high priority tasks at the front, and generate a new task scheduling queue.

Images