Intelligent communication data processing and analyzing system based on edge computing

Abstract

The invention discloses an intelligent communication data processing and analysis system based on edge computing, which belongs to the technical field of communication optimization, and specifically comprises a data acquisition module, a data processing module, a flow distribution module and a long-term monitoring module: the data acquisition module is deployed at the edge of equipment and collects and preprocesses data packet information; the data processing module is used for calculating the link average delay, the maximum bandwidth utilization rate and the average packet loss rate according to the collected data packet information; the flow distribution module is used for adjusting the flow distribution proportion based on the link average delay, the maximum bandwidth utilization rate and the average packet loss rate calculation result; the long-term monitoring module is used for continuously monitoring link performance and dynamically adjusting flow distribution according to a preset threshold value; according to the invention, intelligent management and optimization of the communication network are realized, and the data transmission efficiency and the network stability are improved.

Description

Technical Field

[0001] The present invention relates to the field of communication optimization technology, and in particular to an intelligent communication data processing and analysis system based on edge computing. Background Art

[0002] With the rapid development of information technology, especially the widespread application of technologies such as the Internet of Things (IoT), cloud computing, big data and artificial intelligence, global data has shown an explosive growth trend. The generation, transmission and processing of massive data have posed unprecedented challenges to the existing network architecture and data processing capabilities.

[0003] In the traditional centralized network architecture, all data needs to be transmitted through the network to a remote data center for processing and storage. This not only increases the transmission burden of the network, but may also cause data transmission delays and affect user experience.

[0004] Especially in hot spots, large-scale data will consume more bandwidth resources in network transmission, which may cause network congestion and affect data transmission speed. Therefore, in order to improve data transmission speed, the optimal data transmission path is often required. As a commonly used routing decision-making method, the OSPF protocol is usually used to build a network topology map, and the SPF algorithm is used to build a shortest path tree to calculate the shortest path, so as to obtain the optimal data transmission path; and building a complete network topology map requires obtaining all link status information, resulting in a waste of bandwidth resources. At the same time, routers face huge pressure when processing large amounts of data, and the SPF algorithm does not support flexible load balancing, which limits the network's data processing capabilities and transmission speed under high traffic conditions, affecting user experience. Summary of the invention

[0005] The purpose of the present invention is to provide an intelligent communication data processing and analysis system based on edge computing to solve the following technical problems:

[0006] Routers face tremendous pressure when processing large amounts of data, and the SPF algorithm does not support flexible load balancing, which limits the network's data processing capabilities and transmission speed under high traffic conditions.

[0007] The purpose of the present invention can be achieved by the following technical solutions:

[0008] An intelligent communication data processing and analysis system based on edge computing, comprising:

[0009] The data acquisition module is used to deploy nodes at the edge of the device in the hot spot area, obtain the link set from the source address to the destination address of each data packet, count the total number of data packets N, the number of lost data packets P and each data packet information in each link every acquisition cycle, and perform preprocessing. The data packet information includes the length L of the data packet.i , Sending time T senμ_i , receiving time T recu_i , obtain the total bandwidth B of the link transmission every collection period, and the total transmission time T of the statistical data;

[0010] The data processing module is used to process each data packet according to its sending time T senμ_i and receiving time T recu_i And the total number of data packets N calculates the average delay time of each link, according to the length of each data packet L i The maximum bandwidth utilization of each link is calculated based on the total number of packets N and the total bandwidth B, and the average loss rate of each link is calculated based on the total number of packets N and the number of lost packets P;

[0011] The traffic distribution module is used to calculate the weight value of each link according to the average delay time, maximum bandwidth utilization and average packet loss rate of all data packets in each link, and reset the traffic distribution ratio of each link according to the weight value;

[0012] The long-term monitoring module is used to set the monitoring period of each link according to the weight value of each link, and continuously monitor the average delay time, maximum bandwidth utilization, and average packet loss rate of all data packets in each link. When any of the indicators is detected to be greater than the preset threshold, the distribution ratio of the traffic on each link is adjusted again.

[0013] As a further solution of the present invention: in the traffic distribution module, the weight value μ of each link is j The calculation formula is:

[0014]

[0015] Among them, w1, w2 and w3 are preset coefficients, and w1+w2+w3=1.

[0016] As a further solution of the present invention: in the data acquisition module, the preprocessing process is:

[0017] Verify the data packet information collected from each node, supplement missing values ​​or outliers, convert date and time fields from strings to a unified format, detect and remove duplicate data records, remove outliers and invalid characters in the data, and unify the values ​​of different types of data.

[0018] As a further solution of the present invention: in the long-term monitoring module, the process of setting the monitoring period of each link is:

[0019] Get the weight value sequence μ1, μ2, ..., μ of all links m , m represents the number of links, and the weight value is normalized, the formula is:

[0020]

[0021] In the formula, γ is the correction value, and γ < 1, then the weight of the link with the largest weight is 1, the monitoring frequency of the link with the largest weight is set to f, and the monitoring frequency of each link is fZ j , then the monitoring period of each link is 1 / fZ j .

[0022] As a further solution of the present invention: in the long-term monitoring module, the process of adjusting the allocation ratio again is:

[0023] Collect performance data of each link in real time, including but not limited to average delay time, maximum bandwidth utilization and average packet loss rate; compare the collected performance data with the preset performance threshold; when it is detected that the performance index of a link exceeds the preset threshold, trigger an alarm and record the event; calculate a new weight value μ according to the degree of exceeding the threshold and historical performance trends j '; Use the new weight value μ j 'Update the traffic distribution ratio and monitoring period of the link 1 / fZ j '.

[0024] As a further solution of the present invention: the data acquisition module also includes an API interface module, which is used to integrate and interact with the data source, import and export data, remotely call functions, and access third-party services through the API interface. The communication protocols supported by the API interface module include HTTP / HTTPS, WebSocket, MQTT, and CoAP.

[0025] As a further solution of the present invention: the long-term monitoring module stores the collected data packet information, the processed alarm results and the historical records of traffic distribution in the cloud service end.

[0026] Beneficial effects of the present invention:

[0027] The present invention reduces the delay of data transmission to the central server and improves the response speed of the system by deploying nodes at the edge of the equipment in the hot spot area to collect and process data packet information in real time; it can record the sending time, receiving time and length of each data packet in detail, so as to accurately calculate the average delay time, maximum bandwidth utilization and average packet loss rate of each link, and provide a scientific basis for traffic allocation; by calculating the weight value of each link and dynamically adjusting the traffic allocation ratio according to these weight values, it can optimize the use of network resources and improve the overall network performance; it can not only continuously monitor the performance indicators of each link, but also automatically adjust the traffic allocation ratio when anomalies are detected to ensure the stability and reliability of the network; it can trigger an alarm and record events when it is detected that the link performance indicators exceed the preset threshold, reducing the need for manual intervention and reducing operation and maintenance costs. In summary, this patent provides an efficient, intelligent and flexible communication data processing and analysis system, which helps to improve network performance, optimize resource allocation and reduce operation and maintenance costs. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] The present invention will be further described below in conjunction with the accompanying drawings.

[0029] Figure 1 It is a module schematic diagram of the present invention. DETAILED DESCRIPTION

[0030] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0031] See also Figure 1 As shown, the present invention is an intelligent communication data processing and analysis system based on edge computing, comprising:

[0032] The data collection module plays the role of an information collector for the system. It deploys collection nodes at the edge of devices in the key nodes of the network, that is, hot spots. These nodes are responsible for capturing each data packet during the transmission from the source address to the destination address. Specifically, this module not only records the basic information of each data packet, such as length, sending time, and receiving time, but also counts the total number of data packets, the number of lost data packets, and the total bandwidth usage of each link within the set collection period. In addition, it also monitors the total time consumed for data transmission to provide comprehensive basic data for subsequent data analysis.

[0033] Data processing module, which is the intelligent analysis engine of the system. This module uses the collected data to calculate the key performance indicators of each link through precise algorithms: including the average delay time based on the time difference between sending and receiving data packets, the maximum bandwidth utilization calculated based on the length of the data packet, the total data volume and the bandwidth, and the average packet loss rate determined by comparing the total number of data packets with the number of lost packets. These indicators together depict the health status and efficiency of the network link.

[0034] The traffic distribution module is the policy executor of the system. Based on the link performance indicators provided by the data processing module, it uses a specific weight calculation formula (involving parameters such as average delay, maximum bandwidth utilization, and average packet loss rate) to assign a weight value to each link. This weight value is directly related to the adjustment of the traffic distribution strategy, ensuring that resources can be dynamically optimized and allocated according to the actual performance of the link, thereby improving the overall network throughput and user experience.

[0035] The long-term monitoring module is responsible for continuously tracking the network status and implementing adaptive adjustments. This module intelligently sets the monitoring cycle according to the weight changes of each link, and continuously evaluates key performance indicators such as the average delay, bandwidth utilization and packet loss rate of the link. Once any indicator is detected to exceed the preset safety threshold, the system will immediately trigger an alarm and automatically record the event details. At the same time, the link weight will be recalculated according to the current situation and historical trends, and the traffic distribution ratio and monitoring frequency will be adjusted to form a closed-loop control mechanism to ensure the efficiency and stability of network operation.

[0036] In summary, the intelligent communication data processing and analysis system of the present invention provides strong support for data flow in complex network environments through refined data collection, in-depth data analysis, intelligent traffic scheduling, and continuous performance monitoring and optimization, and significantly improves the utilization efficiency and service quality of network resources.

[0037] In another preferred embodiment of the present invention, in the traffic distribution module, the weight value μ of each link is j The calculation formula is:

[0038]

[0039] Among them, w1, w2 and w3 are preset coefficients, and w1+w2+w3=1.

[0040] In another preferred embodiment of the present invention, in the data acquisition module, the preprocessing process is:

[0041] Data verification and missing value processing: First, the data collected from each node is thoroughly verified to check the integrity and rationality of the data. For missing values ​​or outliers found, the system uses advanced data interpolation technology or context-based inference methods to supplement them to ensure that each data record is complete and meaningful.

[0042] Date and time standardization: Given that the date and time information in the data packet may come from different devices or systems and have various formats, the system converts all date and time fields into a standard format (such as ISO8601) so that the time difference can be accurately compared and calculated during subsequent processing and analysis, which is particularly important for calculating key indicators such as latency.

[0043] Duplicate data detection and removal: During high-speed data transmission, duplicate data records may be generated. The system detects these duplicates through efficient algorithms and removes them from the data set, avoiding redundancy and deviation during data analysis and ensuring the accuracy of the analysis results.

[0044] Outliers and invalid data processing: To improve the accuracy of data analysis, the system will identify and remove outliers in the data, which may be caused by accidental errors or extreme situations, and they may have an adverse effect on statistical analysis. At the same time, the system will also remove any invalid characters or incorrectly formatted entries in the data to ensure data cleanliness.

[0045] Data type standardization: Given that data from different sources may have inconsistent types, such as numerical data may be recorded as strings or in different units, the system will standardize such data. This includes converting numbers in string form to appropriate numerical types and converting all measurement units to consistent standards to facilitate comparison and comprehensive analysis across data sets.

[0046] In another preferred embodiment of the present invention, in the long-term monitoring module, the process of setting the monitoring period of each link is:

[0047] Get the weight value sequence μ1, μ2, ..., μ of all links m , m represents the number of links, and the weight value is normalized, the formula is:

[0048]

[0049] In the formula, γ is the correction value, and γ < 1, then the weight of the link with the largest weight is 1, the monitoring frequency of the link with the largest weight is set to f, and the monitoring frequency of each link is fZ j , then the monitoring period of each link is 1 / fZ j .

[0050] In another preferred embodiment of the present invention, in the long-term monitoring module, the process of re-adjusting the allocation ratio is:

[0051] Collect performance data of each link in real time, including but not limited to average delay time, maximum bandwidth utilization and average packet loss rate; compare the collected performance data with the preset performance threshold; when it is detected that the performance index of a link exceeds the preset threshold, trigger an alarm and record the event; calculate a new weight value μ according to the degree of exceeding the threshold and historical performance trends j '; Use the new weight value μ j 'Update the traffic distribution ratio and monitoring period of the link 1 / fZ j '.

[0052] In another preferred embodiment of the present invention, the data acquisition module also includes an API interface module for integrating and interacting with the data source, importing and exporting data, remotely calling functions, and accessing third-party services through the API interface, and the communication protocols supported by the API interface module include HTTP / HTTPS, WebSocket, MQTT, and CoAP. Through the API interface module, the system can easily integrate and interact with various data sources, support multiple communication protocols, and enhance the compatibility and scalability of the system.

[0053] In another preferred embodiment of the present invention, the long-term monitoring module stores the collected data packet information, the processed alarm results and the historical records of traffic distribution to the cloud service end, which is convenient for subsequent data analysis and mining, and provides support for enterprise decision-making.

[0054] The above is a detailed description of an embodiment of the present invention, but the content is only a preferred embodiment of the present invention and cannot be considered to limit the scope of implementation of the present invention. All equivalent changes and improvements made within the scope of the present invention should still fall within the scope of the patent coverage of the present invention.

Claims

1. An intelligent communication data processing and analysis system based on edge computing, characterized in that: include: The data acquisition module is used to deploy nodes at the edge of the device in the hot spot area, obtain the link set from the source address to the destination address of each data packet, count the total number of data packets N, the number of lost data packets P and each data packet information in each link every acquisition cycle, and perform preprocessing. The data packet information includes the length L of the data packet. i , Sending time T senμ_i , receiving time T recu_i , obtain the total bandwidth B of the link transmission every collection period, and the total transmission time T of the statistical data; The data processing module is used to process each data packet according to its sending time T senμ_i and receiving time T recu_i And the total number of data packets N calculates the average delay time of each link, according to the length of each data packet L i The maximum bandwidth utilization of each link is calculated based on the total number of packets N and the total bandwidth B, and the average loss rate of each link is calculated based on the total number of packets N and the number of lost packets P; The traffic distribution module is used to calculate the weight value of each link according to the average delay time, maximum bandwidth utilization and average packet loss rate of all data packets in each link, and reset the traffic distribution ratio of each link according to the weight value; The long-term monitoring module is used to set the monitoring period of each link according to the weight value of each link, and continuously monitor the average delay time, maximum bandwidth utilization, and average packet loss rate of all data packets in each link. When any of the indicators is detected to be greater than the preset threshold, the distribution ratio of the traffic on each link is adjusted again.

2. According to claim 1, an intelligent communication data processing and analysis system based on edge computing is characterized in that: In the traffic distribution module, the weight value μ of each link is j The calculation formula is: Among them, w1, w2 and w3 are preset coefficients, and w1+w2+w3=1.

3. According to claim 1, an intelligent communication data processing and analysis system based on edge computing is characterized in that: In the data acquisition module, the preprocessing process is: Verify the data packet information collected from each node, supplement missing values ​​or outliers, convert date and time fields from strings to a unified format, detect and remove duplicate data records, remove outliers and invalid characters in the data, and unify the values ​​of different types of data.

4. According to claim 3, an intelligent communication data processing and analysis system based on edge computing is characterized in that: In the long-term monitoring module, the process of setting the monitoring period of each link is as follows: Get the weight value sequence μ1, μ2, ..., μ of all links m , m represents the number of links, and the weight value is normalized, the formula is: In the formula, γ is the correction value, and γ < 1, then the weight of the link with the largest weight is 1, the monitoring frequency of the link with the largest weight is set to f, and the monitoring frequency of each link is fZ j , then the monitoring period of each link is 1 / fZ j .

5. The intelligent communication data processing and analysis system based on edge computing according to claim 4 is characterized in that: In the long-term monitoring module, the process of adjusting the allocation ratio again is as follows: Collect performance data of each link in real time, including but not limited to average delay time, maximum bandwidth utilization and average packet loss rate; compare the collected performance data with the preset performance threshold; When it is detected that the performance indicator of a link exceeds the preset threshold, an alarm is triggered and the event is recorded; Calculate the new weight value μ based on the degree of exceeding the threshold and the historical performance trend j '; Use the new weight value μ j 'Update the traffic distribution ratio and monitoring period of the link 1 / fZ j '.

6. The intelligent communication data processing and analysis system based on edge computing according to claim 1, characterized in that: The data acquisition module also includes an API interface module for integrating and interacting with the data source, importing and exporting data, remotely calling functions, and accessing third-party services through the API interface. The communication protocols supported by the API interface module include HTTP / HTTPS, WebSocket, MQTT, and CoAP.

7. The intelligent communication data processing and analysis system based on edge computing according to claim 1, characterized in that: The long-term monitoring module stores the collected data packet information, processed alarm results and historical records of traffic distribution in the cloud service end.

Images