Abnormal signal intelligent identification method based on big data

Abstract

The invention discloses an abnormal signal intelligent identification method based on big data, and the method achieves the precise identification and quick recovery of a network packet leakage condition through the construction of a network behavior identification model in combination with a data packet caching and retransmission mechanism. According to the method, a time domain state space is fused with a deep learning algorithm, a model is dynamically trained to adapt to network environment changes, and the method has strong adaptive ability and intelligent level. Data loss and transmission delay are effectively reduced by capturing transmission information in real time, detecting abnormal behaviors, isolating abnormal network nodes and re-planning a transmission path. And meanwhile, by iteratively updating the model, the detection accuracy and efficiency are continuously improved, and the integrity and real-time performance of data transmission are ensured.

Description

Technical Field

[0001] The present invention relates to the field of network communication technology, and in particular to a method for intelligently identifying abnormal signals based on big data. Background Art

[0002] With the rapid development of the Internet, the amount of network data transmission has increased dramatically, and the complexity and instability of the network environment have also increased. During the network transmission process, due to various factors such as network congestion, equipment failure, and link instability, data packet loss (i.e., packet leakage) often occurs, which not only affects the integrity and real-time performance of data transmission, but may also cause significant losses to business systems that rely on data transmission.

[0003] Most of the existing network transmission monitoring technologies focus on statistical analysis and anomaly detection of network traffic. Although they can identify anomalies in the network to a certain extent, they still have many deficiencies in packet leakage detection. On the one hand, traditional methods often rely on simple packet counting or timestamp comparison. These methods are often difficult to accurately distinguish between normal packet loss and abnormal packet leakage when facing a high-concurrency, dynamically changing network environment, resulting in high false alarm and missed alarm rates. On the other hand, even if packet leakage can be detected, most of the existing technical solutions lack an effective recovery mechanism and can only adopt a simple retransmission strategy, which not only increases the network burden, but may also aggravate the packet leakage problem due to improper retransmission path selection.

[0004] In addition, with the rise of emerging technologies such as cloud computing, big data, and the Internet of Things, the diversity and complexity of network transmission have further increased, placing higher demands on packet leakage detection and recovery technologies. Traditional rule-based detection methods are difficult to adapt to such changes, and more intelligent and adaptive technical means need to be introduced to cope with them.

[0005] Therefore, how to provide an intelligent abnormal signal recognition method based on big data that can adaptively recognize and process packet leakage is a technical problem that needs to be solved urgently by those skilled in the art. Summary of the invention

[0006] In view of the above research status and existing problems, the present invention provides an intelligent recognition method for abnormal signals based on big data. By building a network behavior recognition model based on machine learning and combining the packet cache and retransmission mechanism, the system can achieve accurate recognition and rapid recovery of packet leakage. At the same time, the system also has strong adaptive capabilities and intelligent level, and can automatically adjust the detection strategy and recovery plan according to changes in the network environment to ensure the integrity and real-time performance of data transmission.

[0007] The present invention provides a method for intelligently identifying abnormal signals based on big data, comprising the following steps:

[0008] S1: dynamically obtain a first data packet and a first transmission path between a transmitting end and a receiving end; extract a first transmission information set;

[0009] S2: uploading the first data packet to the cache server in real time and setting a cache period;

[0010] S3: constructing a network behavior recognition model based on a time-domain state-space fusion deep learning algorithm, which is used to determine whether packet leakage occurs according to the data transmitted by the sending end and the receiving end, and dynamically training the network behavior recognition model with the first transmission information set as input and the network behavior judgment result of each network node as output;

[0011] S4: acquiring in real time a second data packet and a second transmission path transmitted between the transmitting end and the receiving end, extracting a second transmission information set; uploading the second data packet to a cache server in real time, and setting a cache period;

[0012] S5: calling the dynamically trained network behavior recognition model to identify abnormal signals on the second transmission information set and outputting a judgment result; if the result contains abnormal behavior, isolating the abnormal network node, regenerating the third transmission path, and extracting the third transmission information set based on the second data packet and the third transmission path;

[0013] S6: Retrieving the second data packet from the cache server and resending it to the receiving end, and inputting the third transmission information set into the network behavior recognition model again for judgment, until a normal judgment result is output;

[0014] S7: Iteratively update the network behavior identification model in real time based on the normal judgment result and the corresponding transmission information set; the first transmission information set, the second transmission information set and the third transmission information set all include: data packet capacity information, sending time node, and data packet arrival time of each network node in the transmission path.

[0015] Preferably, the step of extracting the first transmission information set includes:

[0016] Get the sender IP and the receiver IP;

[0017] Obtain the network architecture according to the IP, calculate its transmission path and initialize the delay weight of each network node on the transmission path;

[0018] Obtaining the sending time node of the first data packet, and calculating the data packet arrival time of each network node on the transmission path based on the transmission path and the delay weight;

[0019] The delay weight is iteratively updated during the real-time iterative update process of the network behavior recognition model.

[0020] Preferably, the step of uploading the data packet to the cache server in real time includes:

[0021] Assign a unique data packet ID to the data packet, obtain the sender IP and the sending time node, generate a data packet header, and store it in correspondence with the data packet;

[0022] The data packet includes a first data packet or a second data packet; the transmission information set includes a first transmission information set or a second transmission information set.

[0023] Preferably, the step of retrieving the second data packet from the cache server in S6 includes:

[0024] Acquire the sender IP and the sending time node according to the second transmission information set;

[0025] Traverse and search the data packet message header in the cache server, and retrieve the second data packet corresponding to the current sending IP and sending time node.

[0026] Preferably, the S6 further includes:

[0027] While retrieving the second data packet from the cache server and resending it to the receiving end, intercepting the data packet sent by the receiving end at the next time node after receiving the second data packet;

[0028] According to the receiving time node of the second data packet corresponding to the output normal judgment result, the data packet sending operation of the receiving end at the next time node after the receiving time node is restored.

[0029] Preferably, the step of constructing a network behavior recognition model based on the time-domain state-space fusion deep learning algorithm in S3 includes:

[0030] constructing an input layer, and inputting the first transmission information set into the input layer;

[0031] Construct a feature extraction layer and use the CNN layer to perform preliminary feature extraction on the input time series data to capture local features;

[0032] A time series modeling layer is constructed, and stacked RNN layers are used to process the time series data after feature extraction to capture the transmission status and evolution law of the dynamically acquired first data packet in the network; each layer of the stacked RNN layer receives the output of the previous layer as input.

[0033] Construct a state space fusion layer to integrate state information at different time steps;

[0034] Construct the output layer: Use the fully connected layer and activation function to convert the fused features into the final judgment results.

[0035] Preferably, the feature extraction layer extracts key features including the rate of change of data packet size, transmission delay and jitter characteristics.

[0036] Preferably, the step of isolating abnormal network nodes in S5 includes:

[0037] According to the network node containing abnormal behavior output by the network behavior recognition model, combined with the network topology structure and the transmission path record, locate the network node;

[0038] An isolation mechanism is initiated through a security device in the networking architecture to isolate the network node from the networking architecture.

[0039] The abnormal signal intelligent recognition method based on big data proposed in the present invention has the following beneficial effects compared with the prior art:

[0040] Accurately identify packet leakage: The network behavior recognition model constructed by the time-domain state-space fusion deep learning algorithm can accurately identify packet leakage caused by factors such as network congestion, equipment failure, and link instability, greatly reducing false alarm and missed alarm rates.

[0041] Quick recovery mechanism: The present invention combines data packet caching and retransmission mechanisms. Once a packet leak is detected, the data packet is immediately retrieved from the cache server and resent. Intelligent path planning is used to avoid improper retransmission path selection, effectively reducing data loss and transmission delays.

[0042] Adaptive adjustment capability: The present invention can automatically adjust the detection strategy and recovery scheme according to the changes in the network environment, ensuring that efficient packet leakage detection and recovery capabilities can be maintained under different network conditions.

[0043] High level of intelligence: By continuously iterating and updating the network behavior recognition model, the system can continuously learn and adapt to the dynamic changes of the network environment, thereby improving the accuracy and efficiency of detection.

[0044] Reducing network burden: The present invention reduces unnecessary retransmission operations through an intelligent packet loss detection and recovery mechanism, thereby reducing network burden and improving overall network performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the prior art descriptions are briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention, and for ordinary technicians in this field, other drawings can be obtained based on the provided drawings without creative work.

[0046] Figure 1This is a flow chart of an abnormal signal intelligent identification method based on big data provided by an embodiment of the present invention;

[0047] Figure 2 is a flow chart of the steps of extracting the first transmission information set provided by an embodiment of the present invention;

[0048] Figure 3 is a flowchart of steps for retrieving a second data packet from a cache server provided by an embodiment of the present invention;

[0049] Figure 4 It is a network behavior identification model architecture diagram provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0050] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0051] like Figure 1-4 As shown, an embodiment of the present invention provides an abnormal signal intelligent identification method based on big data, and the specific execution steps are as follows:

[0052] Step S1: Dynamically obtain transmission information:

[0053] A first data packet and a first transmission path between a sending end and a receiving end are captured in real time.

[0054] The first transmission information set is extracted, including data packet capacity information, sending time node, data packet arrival time of each network node in the transmission path, etc.

[0055] Get the sender IP and the receiver IP, obtain the network architecture based on the IP, calculate and initialize the delay weight of each network node on the transmission path.

[0056] Step S2: Packet Buffering:

[0057] The first data packet is uploaded to the cache server, and a unique data packet ID is assigned, and a data packet message header is generated and stored corresponding to the data packet.

[0058] Set a reasonable cache period to facilitate possible subsequent retransmission operations.

[0059] Step S3: Build and train a network behavior recognition model:

[0060] A network behavior recognition model is constructed using a time-domain state-space fusion deep learning algorithm.

[0061] The input layer receives a first transmission information set.

[0062] The feature extraction layer uses the CNN layer to extract key features such as the rate of change of packet size, transmission delay, and jitter.

[0063] The timing modeling layer uses stacked RNN layers to capture the transmission status and evolution rules of data packets.

[0064] The state space fusion layer integrates the state information at different time steps.

[0065] The output layer uses a fully connected layer and an activation function to output the network behavior judgment results.

[0066] The network behavior recognition model is dynamically trained using the first transmission information set.

[0067] Steps S4 to S7: Real-time detection and recovery

[0068] S4: Capture the second data packet and the second transmission path in real time, and repeat steps S1 and S2.

[0069] S5: Use the trained network behavior recognition model to identify abnormal signals in the second transmission information set. If abnormal behavior is found, isolate the abnormal network node (activate the isolation mechanism through the security device in the networking architecture), and regenerate the third transmission path. The isolation operation can be implemented through a network management system (NMS) or (such as a firewall).

[0070] S6: retrieve the second data packet from the cache server, resend it to the receiving end, and extract the third transmission information set based on the third transmission path. Repeat the detection process until a normal judgment result is output.

[0071] S7: Iteratively update the network behavior recognition model in real time based on the normal judgment result and the corresponding transmission information set, including updating parameters such as delay weight.

[0072] In one embodiment, the step of extracting the first transmission information set includes:

[0073] Obtain the sender IP and the receiver IP; obtain the networking architecture based on the IP, calculate its transmission path and initialize the delay weight of each network node on the transmission path; obtain the sending time node of the first data packet, and calculate the data packet arrival time of each network node on the transmission path based on the transmission path and the delay weight; iteratively update the delay weight during the real-time iterative update of the network behavior recognition model.

[0074] In one embodiment, while retrieving the second data packet from the cache server and resending it to the receiving end, the data packet sent by the receiving end at the next time node after receiving the data packet is intercepted to avoid data conflict. According to the receiving time node of the second data packet corresponding to the output normal judgment result, the data packet sending operation of the receiving end at the next time node after the receiving time node is restored.

[0075] In one embodiment, the specific steps of regenerating the third transmission path include:

[0076] Step 1: Create a shortest path tree to store the vertices to be processed and the current shortest distances to the source. Initialize the distance from the source to itself to 0, and the distance from the source to all other vertices to infinity (or a large number). Add the source to the shortest path tree.

[0077] Step 2: Take the vertex u with the shortest distance from the shortest path tree (i.e., the unprocessed vertex closest to the source point). Traverse all the adjacent points v of vertex u, and for each adjacent point v, calculate the path length from the source point to v through u (i.e., the distance from u to the source point plus the weight of the edge from u to v). If the calculated path length is less than the currently recorded distance from v to the source point, update the distance from v to the source point and add v to the priority queue (if v is not already in the queue).

[0078] Step 3: Repeat step 2, continuously taking out the vertex with the smallest distance from the shortest path tree, and updating the distance of its adjacent points. The shortest distance from each vertex to the source point has been calculated and stored in the corresponding data structure.

[0079] Step 4: Output the shortest path from any vertex to the source point as needed.

[0080] Through the above implementation, the present invention realizes accurate identification and rapid recovery of packet leakage, while improving the adaptive ability and intelligence level of the system, ensuring the integrity and real-time performance of data transmission.

[0081] The above is a detailed introduction to the abnormal signal intelligent identification method based on big data provided by the present invention. In this article, specific examples are used to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea; at the same time, for general technical personnel in this field, according to the idea of ​​the present invention, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.

[0082] In this article, relational terms such as first and second, etc. are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "comprise", "include" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the statement "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

Claims

1. A method for intelligent identification of abnormal signals based on big data, characterized in that: The steps include: S1: dynamically obtain a first data packet and a first transmission path between a transmitting end and a receiving end; extract a first transmission information set; S2: uploading the first data packet to the cache server in real time and setting a cache period; S3: constructing a network behavior recognition model based on a time-domain state-space fusion deep learning algorithm, which is used to determine whether packet leakage occurs according to the data transmitted by the sending end and the receiving end, and dynamically training the network behavior recognition model with the first transmission information set as input and the network behavior judgment result of each network node as output; S4: acquiring in real time a second data packet and a second transmission path transmitted between the transmitting end and the receiving end, and extracting a second transmission information set; uploading the second data packet to the cache server in real time and setting a cache period; S5: calling the dynamically trained network behavior recognition model to perform abnormal signal recognition on the second transmission information set, and outputting a judgment result; If the result includes abnormal behavior, isolating the abnormal network node, regenerating a third transmission path, and extracting a third transmission information set based on the second data packet and the third transmission path; S6: Retrieving the second data packet from the cache server and resending it to the receiving end, and inputting the third transmission information set into the network behavior recognition model again for judgment, until a normal judgment result is output; S7: Iteratively updating the network behavior recognition model in real time based on the normal judgment result and the corresponding transmission information set; The first transmission information set, the second transmission information set and the third transmission information set all include: data packet capacity information, sending time node, and data packet arrival time of each network node in the transmission path.

2. According to the method of intelligent recognition of abnormal signals based on big data in claim 1, it is characterized in that: The step of extracting the first transmission information set comprises: Get the sender IP and the receiver IP; Obtain the network architecture according to the IP, calculate its transmission path and initialize the delay weight of each network node on the transmission path; Obtaining the sending time node of the first data packet, and calculating the data packet arrival time of each network node on the transmission path based on the transmission path and the delay weight; The delay weight is iteratively updated during the real-time iterative update process of the network behavior recognition model.

3. The method for intelligently identifying abnormal signals based on big data according to claim 1 is characterized in that: The steps of uploading data packets to the cache server in real time include: Assign a unique data packet ID to the data packet, obtain the sender IP and the sending time node, generate a data packet header, and store it in correspondence with the data packet; The data packet includes a first data packet or a second data packet; the transmission information set includes a first transmission information set or a second transmission information set.

4. The method for intelligently identifying abnormal signals based on big data according to claim 3 is characterized in that: The step of retrieving the second data packet from the cache server in S6 includes: Acquire the sender IP and the sending time node according to the second transmission information set; Traverse and search the data packet message header in the cache server, and retrieve the second data packet corresponding to the current sending IP and sending time node.

5. The method for intelligently identifying abnormal signals based on big data according to claim 1 is characterized in that: The S6 further includes: While retrieving the second data packet from the cache server and resending it to the receiving end, intercepting the data packet sent by the receiving end at the next time node after receiving the second data packet; According to the receiving time node of the second data packet corresponding to the output normal judgment result, the data packet sending operation of the receiving end at the next time node after the receiving time node is restored.

6. The method for intelligently identifying abnormal signals based on big data according to claim 1 is characterized in that: The step of constructing a network behavior recognition model based on the time-domain state-space fusion deep learning algorithm in S3 includes: constructing an input layer, and inputting the first transmission information set into the input layer; Construct a feature extraction layer and use the CNN layer to perform preliminary feature extraction on the input time series data to capture local features; Constructing a time series modeling layer, using a stacked RNN layer to process the time series data after feature extraction, capturing the transmission status and evolution law of the dynamically acquired first data packet in the network; each layer of the stacked RNN layer receives the output of the previous layer as input; Construct a state space fusion layer to integrate state information at different time steps; Construct the output layer: Use the fully connected layer and activation function to convert the fused features into the final judgment results.

7. The method for intelligently identifying abnormal signals based on big data according to claim 6 is characterized in that: The feature extraction layer extracts key features including the rate of change of data packet size, transmission delay and jitter features.

8. The method for intelligently identifying abnormal signals based on big data according to claim 1 is characterized in that: The step of isolating abnormal network nodes in S5 includes: According to the network node containing abnormal behavior output by the network behavior recognition model, combined with the network topology structure and the transmission path record, locate the network node; An isolation mechanism is initiated through a security device in the networking architecture to isolate the network node from the networking architecture.

Images