Intelligent recognition method of abnormal signals based on big data

Abstract

This invention discloses a method for intelligently identifying abnormal signals based on big data. By constructing a network behavior recognition model and combining it with a packet caching and retransmission mechanism, it accurately identifies and quickly recovers from network packet leakage. This method utilizes a time-domain state-space fusion deep learning algorithm to dynamically train the model to adapt to changes in the network environment, and possesses strong adaptability and intelligence. By capturing transmission information in real time, detecting abnormal behavior, isolating abnormal network nodes, and replanning transmission paths, data loss and transmission delays are effectively reduced. At the same time, by iteratively updating the model, detection accuracy and efficiency are continuously improved, ensuring the integrity and real-time nature of data transmission.

Description

Technical Field

[0001] The present invention relates to the field of network communication technology, and in particular to a method for intelligently identifying abnormal signals based on big data. Background Art

[0002] With the rapid development of the internet, the volume of network data transmission has increased dramatically, exacerbating the complexity and instability of the network environment. During network transmission, packet loss (i.e., packet leakage) often occurs due to various factors, such as network congestion, equipment failure, and link instability. This not only affects the integrity and real-time nature of data transmission, but can also cause significant losses to business systems that rely on data transmission.

[0003] Existing network transmission monitoring technologies mostly focus on statistical analysis and anomaly detection of network traffic. While these technologies can identify anomalies in the network to a certain extent, they still have many shortcomings in detecting packet loss. On the one hand, traditional methods often rely on simple packet counting or timestamp comparison. These methods often have difficulty accurately distinguishing normal packet loss from abnormal packet loss in highly concurrent and dynamically changing network environments, resulting in high rates of false positives and false negatives. On the other hand, even if packet loss can be detected, existing technical solutions often lack effective recovery mechanisms and often rely on simple retransmission strategies. This not only increases the network burden but can also exacerbate the packet loss problem due to improper retransmission path selection.

[0004] Furthermore, with the rise of emerging technologies such as cloud computing, big data, and the Internet of Things, the diversity and complexity of network transmission have further increased, placing higher demands on packet loss detection and recovery technologies. Traditional rule-based detection methods are unable to adapt to these changes, requiring the introduction of more intelligent and adaptive technologies.

[0005] Therefore, how to provide an intelligent abnormal signal recognition method based on big data that can adaptively identify and process packet leakage is a technical problem that needs to be solved urgently by those skilled in the art. Summary of the Invention

[0006] In response to the aforementioned research status and existing challenges, this paper provides a big data-based intelligent abnormal signal identification method. By constructing a network behavior recognition model based on machine learning and combining it with a packet caching and retransmission mechanism, this method achieves accurate identification and rapid recovery from packet loss. Furthermore, the system possesses powerful adaptive capabilities and intelligence, automatically adjusting detection strategies and recovery plans based on changes in the network environment, ensuring the integrity and real-time nature of data transmission.

[0007] The present invention provides a method for intelligently identifying abnormal signals based on big data, comprising the following steps:

[0008] S1: Dynamically obtain a first data packet and a first transmission path between a sending end and a receiving end; extract a first transmission information set;

[0009] S2: Upload the first data packet to the cache server in real time and set a cache period;

[0010] S3: Build a network behavior recognition model based on a time-domain state-space fusion deep learning algorithm, and use it to determine whether packet loss occurs based on the data transmitted by the transmitter and receiver. The network behavior recognition model is dynamically trained using the first transmission information set as input and the network behavior judgment results of each network node as output.

[0011] S4: Acquire a second data packet and a second transmission path transmitted between the sending end and the receiving end in real time, extract a second transmission information set; upload the second data packet to a cache server in real time, and set a cache period;

[0012] S5: Calling the dynamically trained network behavior recognition model to perform abnormal signal recognition on the second transmission information set and outputting a judgment result; if the result contains abnormal behavior, isolating the abnormal network node, regenerating a third transmission path, and extracting a third transmission information set based on the second data packet and the third transmission path;

[0013] S6: Retrieving the second data packet from the cache server and resending it to the receiving end, and inputting the third transmission information set into the network behavior recognition model again for judgment, until a normal judgment result is output;

[0014] S7: The network behavior identification model is updated in real time based on the normal judgment result and the corresponding transmission information set; the first transmission information set, the second transmission information set and the third transmission information set all include: data packet capacity information, sending time node, and data packet arrival time of each network node in the transmission path.

[0015] Preferably, the step of extracting the first transmission information set includes:

[0016] Get the sender IP and the receiver IP;

[0017] Obtain the network architecture based on the IP address, calculate its transmission path, and initialize the delay weight of each network node on the transmission path;

[0018] Obtaining the sending time node of the first data packet, and calculating the data packet arrival time of each network node on the transmission path based on the transmission path and delay weight;

[0019] The delay weight is iteratively updated during the real-time iterative update process of the network behavior recognition model.

[0020] Preferably, the step of uploading the data packet to the cache server in real time includes:

[0021] Assign a unique data packet ID to the data packet, obtain the sender IP and sending time node, generate the data packet header, and store it in correspondence with the data packet;

[0022] The data packet includes a first data packet or a second data packet; the transmission information set includes a first transmission information set or a second transmission information set.

[0023] Preferably, the step of retrieving the second data packet from the cache server in S6 includes:

[0024] Acquire the sender IP and sending time node according to the second transmission information set;

[0025] Traverse and search the data packet message header in the cache server to retrieve the second data packet corresponding to the current sending end IP and sending time node.

[0026] Preferably, the S6 further includes:

[0027] Retrieving the second data packet from the cache server and resending it to the receiving end, while intercepting the data packet sent by the receiving end at the next time point after receiving the second data packet;

[0028] According to the receiving time node of the second data packet corresponding to the output normal judgment result, the data packet sending operation of the receiving end at the next time node after the receiving time node is restored.

[0029] Preferably, the step of constructing a network behavior recognition model based on the time-domain state-space fusion deep learning algorithm in S3 includes:

[0030] constructing an input layer, and inputting the first transmission information set into the input layer;

[0031] Construct a feature extraction layer and use the CNN layer to perform preliminary feature extraction on the input time series data to capture local features;

[0032] A time series modeling layer is constructed, and stacked RNN layers are used to process the time series data after feature extraction to capture the transmission status and evolution law of the dynamically acquired first data packet in the network; each layer of the stacked RNN layer receives the output of the previous layer as input.

[0033] Construct a state space fusion layer to integrate state information at different time steps;

[0034] Construct the output layer: Use the fully connected layer and activation function to convert the fused features into the final judgment results.

[0035] Preferably, the key features extracted by the feature extraction layer include the rate of change of data packet size, transmission delay and jitter characteristics.

[0036] Preferably, the step of isolating abnormal network nodes in S5 includes:

[0037] Locating the network node containing abnormal behavior according to the output of the network behavior recognition model in combination with the network topology and transmission path records;

[0038] An isolation mechanism is initiated by a security device in the networking architecture to isolate the network node from the networking architecture.

[0039] The intelligent recognition method of abnormal signals based on big data proposed in this invention has the following beneficial effects compared with the existing technology:

[0040] Accurately identify packet leakage: The network behavior recognition model constructed by the present invention adopts the time-domain state-space fusion deep learning algorithm, which can accurately identify packet leakage caused by factors such as network congestion, equipment failure, and link instability, and greatly reduce the false alarm and missed alarm rates.

[0041] Fast recovery mechanism: The present invention combines data packet caching and retransmission mechanisms. Once a packet loss is detected, the data packet is immediately retrieved from the cache server and resent. Intelligent path planning is used to avoid improper retransmission path selection, effectively reducing data loss and transmission delays.

[0042] Adaptive adjustment capability: The present invention can automatically adjust the detection strategy and recovery scheme according to the changes in the network environment, ensuring that efficient packet leakage detection and recovery capabilities can be maintained under different network conditions.

[0043] High level of intelligence: By continuously iteratively updating the network behavior recognition model, the present invention enables the system to continuously learn and adapt to dynamic changes in the network environment, thereby improving the accuracy and efficiency of detection.

[0044] Reduce network burden: The present invention reduces unnecessary retransmission operations through an intelligent packet loss detection and recovery mechanism, thereby reducing network burden and improving overall network performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] To more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only embodiments of the present invention. Those skilled in the art can also derive other drawings based on the provided drawings without inventive effort.

[0046] Figure 1This is a flow chart of a method for intelligently identifying abnormal signals based on big data provided by an embodiment of the present invention;

[0047] Figure 2 is a flow chart of the steps for extracting the first transmission information set provided by an embodiment of the present invention;

[0048] Figure 3 is a flow chart of the steps of retrieving a second data packet from a cache server provided by an embodiment of the present invention;

[0049] Figure 4 This is a diagram of the network behavior recognition model architecture provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0050] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0051] like Figure 1-4 As shown, an embodiment of the present invention provides an abnormal signal intelligent identification method based on big data, and the specific execution steps are as follows:

[0052] Step S1: Dynamically obtain transmission information:

[0053] A first data packet and a first transmission path between a sending end and a receiving end are captured in real time.

[0054] Extract the first transmission information set, including data packet capacity information, sending time node, data packet arrival time of each network node in the transmission path, etc.

[0055] Obtain the sender IP and receiver IP, obtain the network architecture based on the IP, and calculate and initialize the delay weight of each network node on the transmission path.

[0056] Step S2: Data packet buffering:

[0057] The first data packet is uploaded to the cache server and a unique data packet ID is assigned. At the same time, a data packet message header is generated and stored corresponding to the data packet.

[0058] Set a reasonable cache period to facilitate possible subsequent retransmission operations.

[0059] Step S3: Build and train a network behavior recognition model:

[0060] A network behavior recognition model is constructed using a time-domain state-space fusion deep learning algorithm.

[0061] The input layer receives a first transmission information set.

[0062] The feature extraction layer uses the CNN layer to extract key features such as the rate of change of packet size, transmission delay, and jitter.

[0063] The timing modeling layer uses stacked RNN layers to capture the transmission status and evolution of data packets.

[0064] The state space fusion layer integrates the state information of different time steps.

[0065] The output layer uses a fully connected layer and activation function to output the network behavior judgment results.

[0066] The network behavior recognition model is dynamically trained using the first transmission information set.

[0067] Steps S4 to S7: Real-time detection and recovery

[0068] S4: Capture the second data packet and the second transmission path in real time, and repeat steps S1 and S2.

[0069] S5: Use the trained network behavior recognition model to identify abnormal signals in the second transmission information set. If abnormal behavior is detected, the abnormal network node is isolated (by initiating an isolation mechanism using security devices in the networking architecture) and a new third transmission path is generated. This isolation operation can be implemented through a network management system (NMS) or a firewall.

[0070] S6: Retrieve the second data packet from the cache server, resend it to the receiving end, and extract the third transmission information set based on the third transmission path. Repeat the detection process until a normal judgment result is output.

[0071] S7: Based on the normal judgment results and the corresponding transmission information set, the network behavior recognition model is updated in real time, including updating parameters such as delay weight.

[0072] In one embodiment, the step of extracting the first transmission information set includes:

[0073] Obtain the sending IP and receiving IP; obtain the network architecture based on the IP, calculate its transmission path and initialize the delay weight of each network node on the transmission path; obtain the sending time node of the first data packet, and calculate the data packet arrival time of each network node on the transmission path based on the transmission path and delay weight; iteratively update the delay weight during the real-time iterative update of the network behavior recognition model.

[0074] In one embodiment, while retrieving the second data packet from the cache server and retransmitting it to the receiving end, the receiving end intercepts the data packet sent by the receiving end at the next time point after receiving the data packet to avoid data conflicts. Based on the time point of receiving the second data packet corresponding to the output normal determination result, the receiving end resumes sending the data packet at the next time point after the reception time point.

[0075] In one embodiment, the specific steps of regenerating the third transmission path include:

[0076] Step 1: Create a shortest path tree to store the vertices to be processed and the current shortest distances to the source vertex. Initialize the distance from the source vertex to itself to 0 and the distances from the source vertex to all other vertices to infinity (or some large number). Add the source vertex to the shortest path tree.

[0077] Step 2: From the shortest path tree, extract the vertex u with the shortest distance (i.e., the unprocessed vertex closest to the source). Traverse all of u's neighboring vertices v. For each neighbor v, calculate the length of the path from the source to v through u (i.e., the distance from u to the source plus the weight of the edge from u to v). If the calculated path length is less than the currently recorded distance from v to the source, update the distance from v to the source and add v to the priority queue (if v is not already in the queue).

[0078] Step 3: Repeat step 2, continuously extract the vertex with the smallest distance from the shortest path tree, and update the distance of its adjacent points. The shortest distance from each vertex to the source point has been calculated and stored in the corresponding data structure.

[0079] Step 4: Output the shortest path from any vertex to the source point as needed.

[0080] Through the above-mentioned implementation manner, the present invention realizes the accurate identification and rapid recovery of packet leakage, while improving the adaptive ability and intelligence level of the system, and ensuring the integrity and real-time performance of data transmission.

[0081] The above is a detailed introduction to the method for intelligent identification of abnormal signals based on big data provided by the present invention. Specific examples are used in this article to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea. At the same time, for those skilled in the art, according to the ideas of the present invention, there will be changes in the specific implementation methods and application scopes. In summary, the content of this specification should not be understood as limiting the present invention.

[0082] In this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or apparatus. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of additional identical elements in the process, method, article, or apparatus comprising the element.

Claims

1. A method for intelligent identification of abnormal signals based on big data, characterized in that: The steps include: S1: Dynamically obtain a first data packet and a first transmission path between a sending end and a receiving end; extract a first transmission information set; S2: Upload the first data packet to the cache server in real time and set a cache period; S3: Building a network behavior recognition model based on a time-domain state-space fusion deep learning algorithm, which is used to determine whether packet leakage occurs based on the data transmitted by the sending end and the receiving end. The network behavior recognition model is dynamically trained with the first transmission information set as input and the network behavior judgment results of each network node as output, including: constructing an input layer, and inputting the first transmission information set into the input layer; Construct a feature extraction layer and use the CNN layer to perform preliminary feature extraction on the input time series data to capture local features; Constructing a time series modeling layer, using stacked RNN layers to process the feature-extracted time series data, capturing the transmission status and evolution of the dynamically acquired first data packet in the network; each layer of the stacked RNN layer receives the output of the previous layer as input; Construct a state space fusion layer to integrate state information at different time steps; Construct the output layer: Use the fully connected layer and activation function to convert the fused features into the final judgment result; S4: Acquire a second data packet and a second transmission path transmitted between the sending end and the receiving end in real time, extract a second transmission information set; upload the second data packet to a cache server in real time, and set a cache period; S5: Calling the dynamically trained network behavior recognition model to perform abnormal signal recognition on the second transmission information set and outputting a judgment result; if the result contains abnormal behavior, isolating the abnormal network node, regenerating a third transmission path, and extracting a third transmission information set based on the second data packet and the third transmission path; S6: Retrieving the second data packet from the cache server and resending it to the receiving end, and inputting the third transmission information set into the network behavior recognition model again for judgment, until a normal judgment result is output; S7: The network behavior identification model is updated in real time based on the normal judgment result and the corresponding transmission information set; the first transmission information set, the second transmission information set and the third transmission information set all include: data packet capacity information, sending time node, and data packet arrival time of each network node in the transmission path.

2. The method for intelligently identifying abnormal signals based on big data according to claim 1, characterized in that: The step of extracting the first transmission information set includes: Get the sender IP and the receiver IP; Obtain the network architecture based on the IP address, calculate its transmission path, and initialize the delay weight of each network node on the transmission path; Obtaining the sending time node of the first data packet, and calculating the data packet arrival time of each network node on the transmission path based on the transmission path and delay weight; The delay weight is iteratively updated during the real-time iterative update process of the network behavior recognition model.

3. The method for intelligently identifying abnormal signals based on big data according to claim 1, characterized in that: The steps for uploading data packets to the cache server in real time include: Assign a unique data packet ID to the data packet, obtain the sender IP and sending time node, generate the data packet header, and store it in correspondence with the data packet; The data packet includes a first data packet or a second data packet; the transmission information set includes a first transmission information set or a second transmission information set.

4. The method for intelligently identifying abnormal signals based on big data according to claim 3, characterized in that: The step of retrieving the second data packet from the cache server in S6 includes: Acquire the sender IP and sending time node according to the second transmission information set; Traverse and search the data packet message header in the cache server to retrieve the second data packet corresponding to the current sending end IP and sending time node.

5. The method for intelligently identifying abnormal signals based on big data according to claim 1, characterized in that: The S6 further includes: While retrieving the second data packet from the cache server and resending it to the receiving end, the data packet sent by the receiving end at the next time node after receiving the second data packet is intercepted; based on the receiving time node of the second data packet corresponding to the output normal judgment result, the data packet sending operation of the receiving end at the next time node after the receiving time node is restored.

6. The method for intelligently identifying abnormal signals based on big data according to claim 1, characterized in that: The feature extraction layer extracts key features including the rate of change of data packet size, transmission delay and jitter features.

7. The method for intelligently identifying abnormal signals based on big data according to claim 1, characterized in that: The step of isolating abnormal network nodes in S5 includes: Locating the network node containing abnormal behavior according to the output of the network behavior recognition model in combination with the network topology and transmission path records; An isolation mechanism is initiated by a security device in the networking architecture to isolate the network node from the networking architecture.

Images