Communication network data processing method and system based on real-time data processing mechanism

Abstract

The invention discloses a communication network data processing method and system based on a real-time data processing mechanism, and belongs to the technical field of data processing. The technical problems that a current data processing system is long in data processing period, data storage and query are not matched with large-scale data, and an efficient and real-time data display and early warning mechanism is lacked are solved. Comprising the following steps: regularly collecting real-time data of network equipment; performing data preprocessing on the network data after the formatting operation through an ETL tool, and loading the structured data into a Kafka cluster in real time through the ETL tool; storing the structured data to a high-performance data storage system; real-time calculation and business logic processing are carried out on the structured data through an Flink cluster, and a processing result is cached through Redis; displaying the structured data and the processing result through a visual platform; and performing early warning analysis based on the processing result.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, specifically to a communication network data processing method and system based on a real-time data processing mechanism. Background Technology

[0002] With the increasing complexity of modern communication networks, traditional data processing systems (such as batch processing and long-latency data queries) can no longer meet the demands of real-time data. Traditional methods typically rely on periodic data capture and batch processing, which are unsuitable for rapidly responding to dynamically changing needs within the network. Therefore, building an efficient real-time data processing system is of great significance for improving network monitoring, fault prediction, and business decision-making.

[0003] Currently, the existing technology has the following shortcomings: 1. The data processing cycle is long, making it impossible to quickly reflect the network status; 2. Traditional data storage and retrieval methods do not support real-time processing of large-scale data; 3. Lack of efficient real-time data display and early warning mechanisms.

[0004] The current data processing systems suffer from problems such as long data processing cycles, inadequate data storage and retrieval for large-scale data, and a lack of efficient real-time data display and early warning mechanisms. These are technical issues that need to be addressed. Summary of the Invention

[0005] The technical objective of this invention is to address the above-mentioned shortcomings by providing a communication network data processing method and system based on a real-time data processing mechanism, thereby solving the technical problems of long data processing cycles, unsuitable data storage and retrieval for large-scale data, and lack of efficient real-time data display and early warning mechanisms in current data processing systems.

[0006] In a first aspect, the present invention provides a communication network data processing method based on a real-time data processing mechanism, comprising the following steps: Real-time data acquisition: Regularly collect real-time data from network devices as network data, and format the network data, which includes device status, network traffic, and fault information; Real-time data loading: The network data after formatting is preprocessed using ETL tools. The preprocessed data is cleaned and converted to obtain structured data, which is then loaded into the Kafka cluster in real time using ETL tools. Data storage and retrieval: Structured data is saved to a high-performance data storage system, and formatted network data is periodically stored as historical data in a historical data storage system. Both the high-performance data storage system and the historical data storage system support data retrieval and export. Data processing and computation: Real-time computation and business logic processing of structured data are performed through a Flink cluster. The structured data is summarized and analyzed from multiple dimensions to obtain data processing results. The processing results are cached through Redis. The multiple dimensions include time and network elements. Real-time monitoring and display: The visualization platform displays structured data and processing results, and supports users in querying structured data and processing results; Data output and alarms: Based on the processing results, early warning analysis is performed, and the processing results and early warning analysis results are pushed to network operation and maintenance personnel. The push methods include email and SMS notifications.

[0007] The process of selecting the best essays also includes the following steps: Intelligent auditing: Performs compliance checks on the collected network data and formats the network data that passes the compliance check.

[0008] The process of selecting the best essays also includes the following steps: Online service recommendation: Collect user behavior data, and based on the user behavior data and processing results, learn user preferences through a large AI model to generate personalized online service recommendation schemes, and optimize and adjust the large model according to real-time user feedback; The analysis based on user behavior data and processing results includes the following operations: The user behavior data is analyzed to identify the user's operation type, the network device or service identifier operated on, and the operation time sequence; The data processing results are analyzed to extract network device status indicators, network traffic characteristics, and fault warning information related to user operations; The parsed user behavior data is spatiotemporally correlated and pattern mined with the corresponding data processing results to analyze users’ operating habits, service demand preferences and potential service quality demands under different network conditions. The AI ​​big model includes a multi-source data fusion layer, a user-network joint feature engineering layer, a personalized recommendation generation layer, and an online learning and optimization layer; The multi-source data fusion layer takes parsed user behavior data, real-time data processing results, historical data processing results, and network topology information as inputs. It aligns, correlates, and vectorizes the input multi-source heterogeneous data to form a unified fusion feature vector and outputs a unified spatiotemporally correlated feature dataset. The user-network joint feature engineering layer takes the feature dataset as input and automatically learns and generates deep features that reflect the complex relationship between user behavior patterns, network status and service quality through the feature extraction network. It generates high-dimensional deep feature vectors that include temporal features, cross features and high-order abstract features. The personalized recommendation generation layer takes high-dimensional deep feature vectors as input and is used to model based on deep neural networks to predict the probability of a user's preference for different network services in a given network context. It generates personalized recommendation schemes that include specific service parameters, execution timing, and expected benefits, and obtains a list of personalized network service recommendation schemes and their confidence scores. The deep neural network is a Transformer or a deep cross network, and the different network services include bandwidth adjustment, routing optimization, security policy deployment, and maintenance window suggestions. The online learning and optimization layer takes the generated recommendation schemes, real-time user feedback on the recommendation schemes, and network state changes after feedback as input. Using online learning algorithms, it dynamically adjusts the parameters of the recommendation generation layer model based on user feedback and subsequent network performance, thereby achieving continuous optimization and personalized adaptation of the model and obtaining updated recommendation model parameters.

[0009] For essay selection, during scheduled data collection, real-time data is obtained from the EMS device via the SFTP protocol, and network traffic and status information are collected from the core network device via the SNMP protocol.

[0010] For the essay selection process, during data storage and retrieval, all structured data is saved to HBase, a high-performance data storage system, while key data within the structured data is saved to StarRocks for real-time querying. Formatted network data is stored as historical data in HBase, a historical data storage system.

[0011] In a second aspect, the present invention provides a communication network data processing system based on a real-time data processing mechanism, comprising a real-time data acquisition module, a real-time data loading module, a data storage and query module, a data processing and calculation module, a real-time monitoring and display module, and a data output and alarm module. The real-time data acquisition module is used to perform the following: periodically collect real-time data from network devices as network data, and format the network data, which includes device status, network traffic, and fault information; The real-time data loading module is used to perform the following: preprocess the formatted network data using ETL tools, clean and convert the data to obtain structured data, and load the structured data into the Kafka cluster in real time using ETL tools; The data storage and query module is used to perform the following: save structured data to a high-performance data storage system, and periodically store formatted network data as historical data in a historical data storage system. Both the high-performance data storage system and the historical data storage system support data query and export. The data processing and computation module is used to perform the following: real-time computation and business logic processing of structured data through the Flink cluster, summarizing and analyzing structured data from multiple dimensions to obtain data processing results, and caching the processing results through Redis. The multiple dimensions include time and network elements. The real-time monitoring and display module is used to perform the following: display structured data and processing results through a visualization platform, and support users to query structured data and processing results; The data output and alarm module is used to perform the following: perform early warning analysis based on the processing results, and push the processing results and early warning analysis results to network operation and maintenance personnel, including email and SMS notifications.

[0012] Preferably, the system further includes an intelligent audit module, which is used to perform the following: perform compliance checks on the collected network data, and perform formatting operations on the network data that passes the compliance check.

[0013] Preferably, the system also includes a network service recommendation module, which is used to perform the following: collect user behavior data, learn user preferences and generate personalized network service recommendation schemes based on user behavior data and processing results through an AI big model, and optimize and adjust the big model according to real-time user feedback; The analysis based on user behavior data and processing results includes the following operations: The user behavior data is analyzed to identify the user's operation type, the network device or service identifier operated on, and the operation time sequence; The data processing results are analyzed to extract network device status indicators, network traffic characteristics, and fault warning information related to user operations; The parsed user behavior data is spatiotemporally correlated and pattern mined with the corresponding data processing results to analyze users’ operating habits, service demand preferences and potential service quality demands under different network conditions. The AI ​​big model includes a multi-source data fusion layer, a user-network joint feature engineering layer, a personalized recommendation generation layer, and an online learning and optimization layer; The multi-source data fusion layer takes parsed user behavior data, real-time data processing results, historical data processing results, and network topology information as inputs. It aligns, correlates, and vectorizes the input multi-source heterogeneous data to form a unified fusion feature vector and outputs a unified spatiotemporally correlated feature dataset. The user-network joint feature engineering layer takes the feature dataset as input and automatically learns and generates deep features that reflect the complex relationship between user behavior patterns, network status and service quality through the feature extraction network. It generates high-dimensional deep feature vectors that include temporal features, cross features and high-order abstract features. The personalized recommendation generation layer takes high-dimensional deep feature vectors as input and is used to model based on deep neural networks to predict the probability of a user's preference for different network services in a given network context. It generates personalized recommendation schemes that include specific service parameters, execution timing, and expected benefits, and obtains a list of personalized network service recommendation schemes and their confidence scores. The deep neural network is a Transformer or a deep cross network, and the different network services include bandwidth adjustment, routing optimization, security policy deployment, and maintenance window suggestions. The online learning and optimization layer takes the generated recommendation schemes, real-time user feedback on the recommendation schemes, and network state changes after feedback as input. Using an online learning algorithm, it dynamically adjusts the parameters of the recommendation generation layer model based on user feedback and subsequent network performance, achieving continuous model optimization and personalized adaptation, resulting in updated recommendation model parameters.

[0014] Preferably, the timed data acquisition module is used to obtain real-time data from the EMS device via the SFTP protocol and to collect network traffic and status information from the core network device via the SNMP protocol.

[0015] Preferably, the data storage and query module is used to save all structured data to HBase, a high-performance data storage system, and save key data in the structured data to StarRocks for real-time querying, and store the formatted network data as historical data in HBase, a historical data storage system.

[0016] The communication network data processing method and system based on real-time data processing mechanism of the present invention have the following advantages: 1. Significantly improve data processing efficiency: By adopting streaming computing technologies such as Kafka and Flink, real-time data loading and processing significantly reduces data processing latency, shortening processing time to the second level. Compared with traditional batch processing methods, it can quickly respond to changes in network status, improve overall data processing efficiency, and ensure efficient processing and transmission of real-time data. 2. Optimize network management and fault prediction: By performing real-time computing and business logic processing based on Flink, it supports dynamic monitoring and fault diagnosis of network traffic. By acquiring network device status and traffic data in real time, it can promptly detect potential faults or performance bottlenecks, provide early warnings, and help network administrators make quick decisions and effectively optimize network resource allocation through multi-dimensional data display. 3. High-efficiency data storage and query: By storing real-time data in StarRocks, high-speed real-time query capabilities are provided, while storing historical data in HBase ensures efficient storage and query of massive amounts of data. This design greatly improves data access speed, especially when it is necessary to quickly query historical data and process large amounts of real-time data, which can significantly improve query response speed and storage efficiency. 4. Enhance the intelligence and scalability of network management: It has good system integration capabilities and can seamlessly connect with existing network equipment and management systems to achieve real-time data analysis and display. In addition, it has strong scalability, which can adapt to the ever-increasing data demand and business scale, support more complex network management needs in the future, and provide a comprehensive intelligent management solution for large-scale communication networks. 5. Improved User Experience: By providing network status monitoring and alarms based on real-time data, network administrators can grasp the network operation status in real time and respond quickly to potential problems. At the same time, the data visualization platform and real-time alarm function improve the convenience and efficiency of user operation. The efficient support for business operations helps to improve customer satisfaction, especially in high-frequency network maintenance and equipment monitoring scenarios, which can bring significant improvement in user experience. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] The invention will be further described below with reference to the accompanying drawings.

[0019] Figure 1 This is a flowchart of a communication network data processing method based on a real-time data processing mechanism, as described in Example 1. Detailed Implementation

[0020] The present invention will be further described below with reference to the accompanying drawings and specific embodiments, so that those skilled in the art can better understand and implement the present invention. However, the embodiments are not intended to limit the present invention. In the absence of conflict, the embodiments of the present invention and the technical features in the embodiments can be combined with each other.

[0021] This invention provides a communication network data processing method and system based on a real-time data processing mechanism to solve the technical problems of current data processing systems, such as long data processing cycles, unsuitable data storage and retrieval for large-scale data, and lack of efficient real-time data display and early warning mechanisms. Example

[0022] This invention discloses a communication network data processing method based on a real-time data processing mechanism, comprising six steps: real-time data acquisition, real-time data loading, data storage and querying, data processing and calculation, real-time monitoring and display, and data output and alarm.

[0023] Step S100 Real-time data acquisition: Periodically collect real-time data from network devices as network data, and format the network data, which includes device status, network traffic, and fault information.

[0024] As a specific implementation of timed data acquisition, real-time data is obtained from EMS devices via the SFTP protocol, and network traffic and status information are collected from core network devices via the SNMP protocol.

[0025] Step S200 Real-time data loading: The network data after formatting is preprocessed using an ETL tool. The preprocessed data is cleaned and converted to obtain structured data, which is then loaded into the Kafka cluster in real time using an ETL tool.

[0026] As a concrete implementation of real-time data loading, data is loaded into Kafka in real time using ETL tools (such as Flink) and processed in real time. Kafka is used to ensure the reliability and efficiency of the data stream, reducing the processing latency to the second level.

[0027] Step S300 Data Storage and Query: Save structured data to a high-performance data storage system. Periodically store formatted network data as historical data in a historical data storage system. Both the high-performance data storage system and the historical data storage system support data query and export.

[0028] As a specific implementation of data storage and querying, all structured data is saved to HBase, a high-performance data storage system, and key data in the structured data is saved to StarRocks for real-time querying. The formatted network data is stored as historical data in HBase, a historical data storage system.

[0029] Step S400 Data Processing and Computation: The structured data is processed in real time using a Flink cluster for business logic processing. The structured data is summarized and analyzed from multiple dimensions to obtain the data processing results. The processing results are cached using Redis. The multiple dimensions include time and network elements.

[0030] Step S500 Real-time Monitoring and Display: The structured data and processing results are displayed through a visualization platform, and users can query the structured data and processing results.

[0031] In this embodiment, the data processing results are output in multiple ways, including real-time reports, data visualization, and real-time alarm systems, to help network maintenance personnel detect anomalies and take timely measures.

[0032] Step S600 Data Output and Alarm: Based on the processing results, perform early warning analysis and push the processing results and early warning analysis results to network operation and maintenance personnel. The push methods include email and SMS notifications.

[0033] As an improvement to this embodiment, the method performs intelligent auditing during real-time data acquisition: it performs compliance checks on the acquired network data and formats the network data that passes the compliance check.

[0034] As an improvement to this embodiment, the method further includes step S700: network service recommendation: collecting user behavior data, learning user preferences and generating personalized network service recommendation schemes based on user behavior data and processing results through an AI big model, and optimizing and adjusting the big model according to real-time user feedback; The analysis based on user behavior data and processing results includes the following operations: The user behavior data is analyzed to identify the user's operation type, the network device or service identifier operated on, and the operation time sequence; The data processing results are analyzed to extract network device status indicators, network traffic characteristics, and fault warning information related to user operations; The parsed user behavior data is spatiotemporally correlated and pattern mined with the corresponding data processing results to analyze users’ operating habits, service demand preferences and potential service quality demands under different network conditions. The AI ​​big model includes a multi-source data fusion layer, a user-network joint feature engineering layer, a personalized recommendation generation layer, and an online learning and optimization layer; The multi-source data fusion layer takes parsed user behavior data, real-time data processing results, historical data processing results, and network topology information as inputs. It aligns, correlates, and vectorizes the input multi-source heterogeneous data to form a unified fusion feature vector and outputs a unified spatiotemporally correlated feature dataset. The user-network joint feature engineering layer takes the feature dataset as input and automatically learns and generates deep features that reflect the complex relationship between user behavior patterns, network status and service quality through the feature extraction network. It generates high-dimensional deep feature vectors that include temporal features, cross features and high-order abstract features. The personalized recommendation generation layer takes high-dimensional deep feature vectors as input and is used to model based on deep neural networks to predict the probability of a user's preference for different network services in a given network context. It generates personalized recommendation schemes that include specific service parameters, execution timing, and expected benefits, and obtains a list of personalized network service recommendation schemes and their confidence scores. The deep neural network is a Transformer or a deep cross network, and the different network services include bandwidth adjustment, routing optimization, security policy deployment, and maintenance window suggestions. The online learning and optimization layer takes the generated recommendation schemes, real-time user feedback on the recommendation schemes, and network state changes after feedback as input. Using online learning algorithms, it dynamically adjusts the parameters of the recommendation generation layer model based on user feedback and subsequent network performance, thereby achieving continuous optimization and personalized adaptation of the model and obtaining updated recommendation model parameters.

[0035] In this large AI model, the multi-source data fusion layer receives user behavior data from the real-time monitoring and display module, as well as processing results from the data processing and computation module, to complete data fusion and preliminary association. Its output feature dataset is input to the user-network joint feature engineering layer for deep feature extraction. The extracted deep feature vectors are passed to the personalized recommendation generation layer to calculate and generate recommendation schemes. The generated recommendation schemes are displayed to users through a visualization platform, and real-time user feedback is collected. This feedback, along with relevant network state data, is used as input to the online learning and optimization layer to evaluate the recommendation effect and update the model parameters, completing the iterative optimization of the recommendation generation layer model and forming a closed-loop intelligent recommendation system.

[0036] The method in this embodiment integrates big data processing technologies such as Kafka, Flink, Redis, and StarRocks to automate the entire process from data acquisition and processing to storage and querying. By using streaming computing technology to process and analyze network data in real time, it significantly improves data processing efficiency and optimizes network management and business operations. Example

[0037] This invention discloses a communication network data processing system based on a real-time data processing mechanism, comprising a real-time data acquisition module, a real-time data loading module, a data storage and query module, a data processing and calculation module, a real-time monitoring and display module, and a data output and alarm module.

[0038] The real-time data acquisition module is used to perform the following: periodically collect real-time data from network devices as network data, and format the network data, which includes device status, network traffic, and fault information.

[0039] As a specific implementation of the timed data acquisition module, this module is used to obtain real-time data from EMS devices via the SFTP protocol and to collect network traffic and status information from core network devices via the SNMP protocol.

[0040] The real-time data loading module performs the following: it preprocesses the formatted network data using an ETL tool, cleans and converts the data to obtain structured data, and then loads the structured data into the Kafka cluster in real time using an ETL tool.

[0041] As a specific implementation of the real-time data loading module, this module loads data into Kafka in real time using ETL tools (such as Flink) and processes it in real time. Kafka is used to ensure the reliability and efficiency of the data stream, reducing the processing latency to the second level.

[0042] The data storage and query module is used to perform the following: save structured data to a high-performance data storage system, and periodically store formatted network data as historical data in a historical data storage system. Both the high-performance data storage system and the historical data storage system support data querying and exporting.

[0043] As a specific implementation of the data storage and query module, this module saves all structured data to HBase, a high-performance data storage system, and saves key data in the structured data to StarRocks for real-time querying. It also stores the formatted network data as historical data in HBase, a historical data storage system.

[0044] The data processing and computation module is used to perform the following: real-time computation and business logic processing of structured data through the Flink cluster, summarizing and analyzing structured data from multiple dimensions to obtain data processing results, and caching the processing results through Redis. The multiple dimensions include time and network elements.

[0045] The real-time monitoring and display module is used to perform the following: display structured data and processing results through a visualization platform, and support users to query structured data and processing results.

[0046] In this embodiment, the data processing results are output in multiple ways, including real-time reports, data visualization, and real-time alarm systems, to help network maintenance personnel detect anomalies and take timely measures.

[0047] The data output and alarm module is used to perform the following: perform early warning analysis based on the processing results, and push the processing results and early warning analysis results to network operation and maintenance personnel, including email and SMS notifications.

[0048] As an improvement to this embodiment, the system also includes an intelligent audit module. The real-time data acquisition module calls the intelligent audit module to perform the following: perform compliance verification on the acquired network data, and perform formatting operation on the network data that passes the compliance verification.

[0049] As an improvement to this embodiment, the system also includes a network service recommendation module, which performs the following operations: collecting user behavior data, learning user preferences and generating personalized network service recommendation schemes based on user behavior data and processing results through an AI big model, and optimizing and adjusting the big model based on real-time user feedback; The analysis based on user behavior data and processing results includes the following operations: The user behavior data is analyzed to identify the user's operation type, the network device or service identifier operated on, and the operation time sequence; The data processing results are analyzed to extract network device status indicators, network traffic characteristics, and fault warning information related to user operations; The parsed user behavior data is spatiotemporally correlated and pattern mined with the corresponding data processing results to analyze users’ operating habits, service demand preferences and potential service quality demands under different network conditions. The AI ​​big model includes a multi-source data fusion layer, a user-network joint feature engineering layer, a personalized recommendation generation layer, and an online learning and optimization layer; The multi-source data fusion layer takes parsed user behavior data, real-time data processing results, historical data processing results, and network topology information as inputs. It aligns, correlates, and vectorizes the input multi-source heterogeneous data to form a unified fusion feature vector and outputs a unified spatiotemporally correlated feature dataset. The user-network joint feature engineering layer takes the feature dataset as input and automatically learns and generates deep features that reflect the complex relationship between user behavior patterns, network status and service quality through the feature extraction network. It generates high-dimensional deep feature vectors that include temporal features, cross features and high-order abstract features. The personalized recommendation generation layer takes high-dimensional deep feature vectors as input and is used to model based on deep neural networks to predict the probability of a user's preference for different network services in a given network context. It generates personalized recommendation schemes that include specific service parameters, execution timing, and expected benefits, and obtains a list of personalized network service recommendation schemes and their confidence scores. The deep neural network is a Transformer or a deep cross network, and the different network services include bandwidth adjustment, routing optimization, security policy deployment, and maintenance window suggestions. The online learning and optimization layer takes the generated recommendation schemes, real-time user feedback on the recommendation schemes, and network state changes after feedback as input. Using online learning algorithms, it dynamically adjusts the parameters of the recommendation generation layer model based on user feedback and subsequent network performance, thereby achieving continuous optimization and personalized adaptation of the model and obtaining updated recommendation model parameters.

[0050] In this large AI model, the multi-source data fusion layer receives user behavior data from the real-time monitoring and display module, as well as processing results from the data processing and computation module, to complete data fusion and preliminary association. Its output feature dataset is input to the user-network joint feature engineering layer for deep feature extraction. The extracted deep feature vectors are passed to the personalized recommendation generation layer to calculate and generate recommendation schemes. The generated recommendation schemes are displayed to users through a visualization platform, and real-time user feedback is collected. This feedback, along with relevant network state data, is used as input to the online learning and optimization layer to evaluate the recommendation effect and update the model parameters, completing the iterative optimization of the recommendation generation layer model and forming a closed-loop intelligent recommendation system.

[0051] The system in this embodiment can execute the method disclosed in Embodiment 1 to perform communication network data processing.

[0052] The communication network data processing method and system based on real-time data processing mechanism provided by the present invention have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only for the purpose of helping to understand the method and core ideas of the present invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A communication network data processing method based on a real-time data processing mechanism, characterized in that, Includes the following steps: Real-time data acquisition: Regularly collect real-time data from network devices as network data, and format the network data, which includes device status, network traffic, and fault information; Real-time data loading: The network data after formatting is preprocessed using ETL tools. The preprocessed data is cleaned and converted to obtain structured data, which is then loaded into the Kafka cluster in real time using ETL tools. Data storage and retrieval: Structured data is saved to a high-performance data storage system, and formatted network data is periodically stored as historical data in a historical data storage system. Both the high-performance data storage system and the historical data storage system support data retrieval and export. Data processing and computation: Real-time computation and business logic processing of structured data are performed through a Flink cluster. The structured data is summarized and analyzed from multiple dimensions to obtain data processing results. The processing results are cached through Redis. The multiple dimensions include time and network elements. Real-time monitoring and display: The visualization platform displays structured data and processing results, and supports users in querying structured data and processing results; Data output and alarms: Based on the processing results, early warning analysis is performed, and the processing results and early warning analysis results are pushed to network operation and maintenance personnel. The push methods include email and SMS notifications.

2. The communication network data processing method based on a real-time data processing mechanism according to claim 1, characterized in that, It also includes the following steps: Intelligent auditing: Performs compliance checks on the collected network data and formats the network data that passes the compliance check.

3. The communication network data processing method based on a real-time data processing mechanism according to claim 1, characterized in that, It also includes the following steps: Online service recommendation: Collect user behavior data, and based on the user behavior data and processing results, learn user preferences through a large AI model to generate personalized online service recommendation schemes, and optimize and adjust the large model according to real-time user feedback; The analysis based on user behavior data and processing results includes the following operations: The user behavior data is analyzed to identify the user's operation type, the network device or service identifier operated on, and the operation time sequence; The data processing results are analyzed to extract network device status indicators, network traffic characteristics, and fault warning information related to user operations; The parsed user behavior data is spatiotemporally correlated and pattern mined with the corresponding data processing results to analyze users’ operating habits, service demand preferences and potential service quality demands under different network conditions. The AI ​​big model includes a multi-source data fusion layer, a user-network joint feature engineering layer, a personalized recommendation generation layer, and an online learning and optimization layer; The multi-source data fusion layer takes parsed user behavior data, real-time data processing results, historical data processing results, and network topology information as inputs. It aligns, correlates, and vectorizes the input multi-source heterogeneous data to form a unified fusion feature vector and outputs a unified spatiotemporally correlated feature dataset. The user-network joint feature engineering layer takes the feature dataset as input and automatically learns and generates deep features that reflect the complex relationship between user behavior patterns, network status and service quality through the feature extraction network. It generates high-dimensional deep feature vectors that include temporal features, cross features and high-order abstract features. The personalized recommendation generation layer takes high-dimensional deep feature vectors as input and is used to model based on deep neural networks to predict the probability of a user's preference for different network services in a given network context. It generates personalized recommendation schemes that include specific service parameters, execution timing, and expected benefits, and obtains a list of personalized network service recommendation schemes and their confidence scores. The deep neural network is a Transformer or a deep cross network, and the different network services include bandwidth adjustment, routing optimization, security policy deployment, and maintenance window suggestions. The online learning and optimization layer takes the generated recommendation schemes, real-time user feedback on the recommendation schemes, and network state changes after feedback as inputs. Using online learning algorithms, it dynamically adjusts the parameters of the recommendation generation layer model based on user feedback and subsequent network performance, thereby achieving continuous optimization and personalized adaptation of the model and obtaining updated recommendation model parameters.

4. The communication network data processing method based on a real-time data processing mechanism according to claim 1, characterized in that, During scheduled data collection, real-time data is obtained from the EMS device via the SFTP protocol, and network traffic and status information are collected from the core network device via the SNMP protocol.

5. The communication network data processing method based on a real-time data processing mechanism according to claim 1, characterized in that, During data storage and retrieval, all structured data is saved to HBase, a high-performance data storage system, and key data in the structured data is saved to StarRocks for real-time querying. Formatted network data is stored as historical data in HBase, a historical data storage system.

6. A communication network data processing system based on a real-time data processing mechanism, characterized in that, It includes a real-time data acquisition module, a real-time data loading module, a data storage and query module, a data processing and calculation module, a real-time monitoring and display module, and a data output and alarm module; The real-time data acquisition module is used to perform the following: periodically collect real-time data from network devices as network data, and format the network data, which includes device status, network traffic, and fault information; The real-time data loading module is used to perform the following: preprocess the formatted network data using ETL tools, clean and convert the data to obtain structured data, and load the structured data into the Kafka cluster in real time using ETL tools; The data storage and query module is used to perform the following: save structured data to a high-performance data storage system, and periodically store formatted network data as historical data in a historical data storage system. Both the high-performance data storage system and the historical data storage system support data query and export. The data processing and computation module is used to perform the following: real-time computation and business logic processing of structured data through the Flink cluster, summarizing and analyzing structured data from multiple dimensions to obtain data processing results, and caching the processing results through Redis. The multiple dimensions include time and network elements. The real-time monitoring and display module is used to perform the following: display structured data and processing results through a visualization platform, and support users to query structured data and processing results; The data output and alarm module is used to perform the following: perform early warning analysis based on the processing results, and push the processing results and early warning analysis results to network operation and maintenance personnel, including email and SMS notifications.

7. The communication network data processing system based on a real-time data processing mechanism according to claim 6, characterized in that, The system also includes an intelligent audit module, which performs the following actions: performs compliance checks on the collected network data and formats the network data that passes the compliance check.

8. The communication network data processing system based on a real-time data processing mechanism according to claim 6, characterized in that, The system also includes a network service recommendation module, which is used to perform the following: collect user behavior data, learn user preferences and generate personalized network service recommendation schemes based on user behavior data and processing results through an AI big model, and optimize and adjust the big model according to real-time user feedback; The analysis based on user behavior data and processing results includes the following operations: The user behavior data is analyzed to identify the user's operation type, the network device or service identifier operated on, and the operation time sequence; The data processing results are analyzed to extract network device status indicators, network traffic characteristics, and fault warning information related to user operations; The parsed user behavior data is spatiotemporally correlated and pattern mined with the corresponding data processing results to analyze users’ operating habits, service demand preferences and potential service quality demands under different network conditions. The AI ​​big model includes a multi-source data fusion layer, a user-network joint feature engineering layer, a personalized recommendation generation layer, and an online learning and optimization layer; The multi-source data fusion layer takes parsed user behavior data, real-time data processing results, historical data processing results, and network topology information as inputs. It aligns, correlates, and vectorizes the input multi-source heterogeneous data to form a unified fusion feature vector and outputs a unified spatiotemporally correlated feature dataset. The user-network joint feature engineering layer takes the feature dataset as input and automatically learns and generates deep features that reflect the complex relationship between user behavior patterns, network status and service quality through the feature extraction network. It generates high-dimensional deep feature vectors that include temporal features, cross features and high-order abstract features. The personalized recommendation generation layer takes high-dimensional deep feature vectors as input and is used to model based on deep neural networks to predict the probability of a user's preference for different network services in a given network context. It generates personalized recommendation schemes that include specific service parameters, execution timing, and expected benefits, and obtains a list of personalized network service recommendation schemes and their confidence scores. The deep neural network is a Transformer or a deep cross network, and the different network services include bandwidth adjustment, routing optimization, security policy deployment, and maintenance window suggestions. The online learning and optimization layer takes the generated recommendation schemes, real-time user feedback on the recommendation schemes, and network state changes after feedback as inputs. Using online learning algorithms, it dynamically adjusts the parameters of the recommendation generation layer model based on user feedback and subsequent network performance, thereby achieving continuous optimization and personalized adaptation of the model and obtaining updated recommendation model parameters.

9. The communication network data processing system based on a real-time data processing mechanism according to claim 6, characterized in that, The timed data acquisition module is used to obtain real-time data from EMS devices via the SFTP protocol and to collect network traffic and status information from core network devices via the SNMP protocol.

10. The communication network data processing system based on a real-time data processing mechanism according to claim 6, characterized in that, The data storage and query module is used to save all structured data to HBase, a high-performance data storage system, and save key data in the structured data to StarRocks for real-time querying. It also stores formatted network data as historical data in HBase, a historical data storage system.

Images