Message processing method and device, electronic equipment and medium product

Abstract

The invention provides a message processing method and device, electronic equipment and a medium, and the method comprises the steps: obtaining message data; performing first clustering processing on the message data by using a pre-training model to obtain a class cluster message; determining a classification result of the message data by utilizing a frequent pattern mining algorithm in the pre-training model based on the class cluster message; and based on the classification result, sending the message data to corresponding processing equipment for processing by utilizing a preset mapping relationship between the classification result and the processing equipment. According to the method disclosed by the invention, the clustering algorithm and the frequent pattern mining algorithm are utilized to identify the message type, so that different types of messages can be distributed to different processors for rapid processing, and the message processing efficiency is improved.

Description

Technical Field

[0001] The present disclosure relates to the field of cloud computing, and in particular to a message processing method, device, electronic equipment, and medium product. Background Art

[0002] With the introduction of concepts such as cloud-network integration, the requirements for data throughput and network speed have increased dramatically. Users are looking for a low-latency, high-bandwidth network switching device. However, current network devices basically rely on a single thread for message processing. In order to increase network bandwidth and throughput, higher-performance chips are usually sought and embedded in network switches. However, this will increase the cost of the switch and cannot fundamentally solve the problem of increasing network speed. Summary of the invention

[0003] The present disclosure provides a message processing method, device, electronic device and medium product to solve the problems in related technologies and improve the message processing efficiency.

[0004] The first aspect of the present disclosure proposes a message processing method, which includes: obtaining message data; clustering the message data using a pre-trained model to obtain cluster messages; based on the cluster messages, using a frequent pattern mining algorithm in the pre-trained model to determine the classification results of the message data; based on the classification results, using a preset mapping relationship between the classification results and the processing device, sending the message data to the corresponding processing device for processing.

[0005] In some embodiments of the present disclosure, clustering processing is performed on message data using a pre-trained model to obtain cluster messages, including: determining a message feature vector based on the message data; determining median information and message distance of the message data based on the message feature vector; and clustering processing is performed on the message data based on the median information and the message distance to obtain cluster messages.

[0006] In some embodiments of the present disclosure, clustering is performed on message data based on median information and message distance to obtain cluster messages, including: determining a first search radius of a pre-trained model based on median information; performing initial clustering on message data based on median information, the first search radius and message distance; iterating the first search radius based on the clustering result of the initial clustering; stopping iteration when the number of iterations reaches a preset maximum value, or when the accuracy of the clustering result after iteration meets preset requirements, to obtain a target search radius; determining a cluster message based on the target search radius.

[0007] In some embodiments of the present disclosure, based on cluster messages, a frequent pattern mining algorithm in a pre-trained model is used to determine the classification result of message data, including: based on the message feature vector of the cluster message, determining the first byte of the cluster message; based on the position of the first byte, determining the offset position of the second byte of the cluster message relative to the first byte, the first byte is different from the second byte; based on the position and offset position of the first byte, utilizing the mapping relationship between frequent byte items and byte positions in a preset model to determine the classification result of the message data.

[0008] In some embodiments of the present disclosure, based on the classification result and the preset mapping relationship between the classification result and the processing device, the message data is processed using the processing device corresponding to the classification result, including: when the classification result is a control class message, the message data is processed using an X86 processing device; when the classification result is a stateless message, the message data is forwarded using a high-speed port; when the classification result is a stateful message, the message data is processed using an NP network processor, and / or, when the classification result is a stateful message, the message data is processed using a SOC FPGA processor.

[0009] In some embodiments of the present disclosure, when the classification result is a stateful message, using the NP network processor to perform message processing on the message data includes: when the classification result is a stateful message, using the NP network processor to query and access the session connection library, and the session connection library is communicatively connected to the NP network processor; when the result of the query access is whether there is a historical session connection of the message data in the session connection library, using the historical session connection to forward the message data.

[0010] In some embodiments of the present disclosure, when the classification result is a stateful message, using the SOC FPGA processor to perform message processing on the message data includes: when the classification result is a stateful message, using the SOC FPGA processor to perform security analysis on the message data; when the security analysis is a secure message, using the SOC FPGA processor to forward the message data; when the security analysis is an illegal message, using the SOC FPGA processor to discard the message data; using the SOC FPGA processor to count the number of repeated occurrences of the message data; when the number of repeated occurrences is greater than a preset value, using the SOC FPGA processor to send the forwarding processing information of the message data to the NP network processor, so as to use the NP network processor to add the session connection of the message data to the session connection library.

[0011] The second aspect embodiment of the present disclosure proposes a message processing device, which includes: an acquisition unit, used to acquire message data; a clustering unit, used to cluster the message data using a pre-trained model to obtain cluster messages; a classification unit, used to determine the classification results of the message data based on the cluster messages and using a frequent pattern mining algorithm in the pre-trained model; and a processing unit, used to send the message data to a corresponding processing device for processing based on the classification results and using a preset mapping relationship between the classification results and the processing device.

[0012] The third aspect embodiment of the present disclosure proposes an electronic device, comprising: a processor and a memory for storing a computer program that can be run on the processor, wherein the processor, when used to run the computer program, executes the method described in the first aspect embodiment of the present disclosure.

[0013] The fourth aspect embodiment of the present disclosure proposes a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to enable a computer to execute the method described in the first aspect embodiment of the present disclosure.

[0014] In summary, according to a message processing method proposed in the present disclosure, the method includes: obtaining message data; performing a first clustering process on the message data using a pre-trained model to obtain a cluster message; based on the cluster message, using a frequent pattern mining algorithm in the pre-trained model to determine a classification result of the message data; based on the classification result, using a preset mapping relationship between the classification result and the processing device, the message data is sent to a corresponding processing device for processing. The method of the present disclosure utilizes a clustering algorithm and a frequent pattern mining algorithm to realize the identification of message types, so that different types of messages can be distributed to different processors for rapid processing, thereby improving the efficiency of message processing.

[0015] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The drawings herein are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the description are used to explain the principles of the present disclosure, and do not constitute improper limitations on the present disclosure.

[0017] Figure 1 A flowchart of a message processing method provided by an embodiment of the present disclosure;

[0018] Figure 2 A flowchart of another message processing method provided in an embodiment of the present disclosure;

[0019] Figure 3A flowchart of a method for obtaining a pre-training model provided in an embodiment of the present disclosure;

[0020] Figure 4 An architecture diagram of a message processing system provided in an embodiment of the present disclosure;

[0021] Figure 5 A schematic diagram of the structure of a message processing device provided in an embodiment of the present disclosure;

[0022] Figure 6 A schematic diagram of the structure of an electronic device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0023] Embodiments of the present disclosure are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present disclosure, and should not be construed as limiting the present disclosure.

[0024] With the introduction of concepts such as cloud-network integration, the requirements for data throughput and network speed have increased dramatically. Users are looking for a low-latency, high-bandwidth network switching device. However, current network devices basically rely on a single thread for message processing. In order to increase network bandwidth and throughput, higher-performance chips are usually sought and embedded in network switches. However, this will undoubtedly increase the cost of the switch and cannot fundamentally solve the problem of increasing network speed.

[0025] The present disclosure is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0026] Figure 1 A flowchart of a message processing method provided by an embodiment of the present disclosure. Figure 1 As shown, steps 101-105 are included.

[0027] Step 101, obtaining message data.

[0028] In some embodiments, the method may be executed by a P4 interaction chip, but is not limited thereto and may also be executed by other chips similar to the P4 interaction chip.

[0029] In some embodiments, message data that needs to be processed can be received through a high-speed port, but this is not limited to this, and the present disclosure does not limit the method for obtaining message data.

[0030] Step 102: cluster the message data using the pre-trained model to obtain cluster messages.

[0031] In some embodiments, by clustering the message data to cluster messages with similar characteristics into messages of the same cluster, preliminary classification of the message data is achieved to facilitate subsequent identification of message types and improve message identification efficiency.

[0032] In some embodiments, the acquisition of the pre-trained model can refer to Figure 3 The embodiments shown will not be described in detail here.

[0033] In some optional embodiments, feature extraction may be performed on message data to obtain feature vectors of the message data, and then clustering may be performed on the message data according to the feature vectors of the message data to obtain cluster messages.

[0034] In other words, after clustering, the messages contained in a cluster are a cluster message.

[0035] Step 103, based on the clustered messages, a frequent pattern mining algorithm in the pre-trained model is used to determine the classification result of the message data.

[0036] In some embodiments, the class clusters may be used as units and input into the pre-trained model to determine the classification results of the message data using the pre-trained model, thereby improving the efficiency of message classification.

[0037] In some optional embodiments, the first byte of the cluster message can be determined based on the message feature vector of the cluster message; then, based on the position of the first byte, the positions of all bytes in the cluster message can be determined; and the values ​​corresponding to each byte can be determined, thereby determining the positions of frequent bytes in each message; and finally, the pre-trained model determines the classification results of each message based on the positions of frequent bytes.

[0038] In some optional embodiments, the messages are classified into: control messages, stateless messages and stateful messages. However, this is not limited to this and the present disclosure does not limit this.

[0039] Step 104: Based on the classification result, the message data is sent to the corresponding processing device for processing by using the preset mapping relationship between the classification result and the processing device.

[0040] In some embodiments, the preset mapping relationship between the classification result and the processing device can be: when the classification result is a control class message, the message data is processed using an X86 processing device; when the classification result is a stateless message, the message data is forwarded using a high-speed port; when the classification result is a stateful message, the message data is processed using an NP network processor, and / or, when the classification result is a stateful message, the message data is processed using a SOC FPGA processor.

[0041] In some embodiments, different types of messages may be sent to different processors for message processing according to the classification results, thereby achieving parallel processing of message data to improve the processing efficiency of the message data.

[0042] In summary, according to the message processing method proposed in the present disclosure, the method includes: obtaining message data; performing a first clustering process on the message data using a pre-trained model to obtain a cluster message; based on the cluster message, using the frequent pattern mining algorithm in the pre-trained model to determine the classification result of the message data; based on the classification result, using the preset mapping relationship between the classification result and the processing device, the message data is sent to the corresponding processing device for processing. The method of the present disclosure uses a clustering algorithm and a frequent pattern mining algorithm to realize the identification of message types, so that different types of messages can be distributed to different processors for rapid processing, thereby improving the efficiency of message processing.

[0043] Figure 2 A flowchart of a message processing method proposed in the present disclosure is further shown. Figure 1 The embodiment shown, Figure 2 The following steps are included.

[0044] Step 201, obtaining message data.

[0045] In some embodiments, the principle of step 201 is the same as that of the embodiment shown in step 101, and reference may be made to the above related description, which will not be repeated here.

[0046] Step 202: Determine a message feature vector based on the message data.

[0047] In some embodiments, the pre-trained model can extract message feature vectors based on valid information in the message data, for example, based on the location of the information in the message (such as header information, message entity, etc.), to determine the feature vector of the message data, but is not limited to this. The present disclosure does not limit the method of extracting message feature vectors by the pre-trained model.

[0048] In some optional embodiments, the message feature vector may include at least one of the following: a protocol field feature vector, a message length feature vector, a message header information feature vector, and a message load feature vector.

[0049] Step 203: Determine the median information and message distance of the message data based on the message feature vector.

[0050] In some embodiments, the median information of the message data refers to the median of the message byte size.

[0051] In some embodiments, since the message feature vector includes feature vectors for indicating message size, message length and message position, the median information of the message data can be determined based on the message feature vector related to the message size, and the message distance can be determined based on the message feature vector related to the message length and message position.

[0052] For example, if the message byte size is 10 bytes, then the median of the message data is 5 bytes.

[0053] For example, the message distance can be determined by the following formula:

[0054]

[0055] Among them, d(m i , m j ) indicates the message distance, m i and m j Indicates two different messages, len is the message length.

[0056] Step 204: cluster the message data based on the median information and the message distance to obtain cluster messages.

[0057] Specifically, the first search radius of the pre-trained model can be determined based on the median information; the message data can be initially clustered based on the median information, the first search radius and the message distance; the first search radius can be iterated based on the clustering result of the initial clustering processing; when the number of iterations reaches a preset maximum value, or when the accuracy of the clustering result after the iteration meets the preset requirements, the iteration is stopped to obtain the target search radius; based on the target search radius, the cluster message is determined.

[0058] For example, 1 / 4 of the median of the message data is determined as the initial search radius ra; the byte size of the message is used as the identification feature, the median of all message sizes is calculated (for example: the message byte size is 10 bytes, then the median is 5 bytes), and then 1 / 4 of the median is used as the initial search radius ra (that is, the first search radius mentioned above), and clustering processing is performed based on the message distance between the message data.

[0059] Then, the first search radius is iterated. For example, the search radius of each iteration is 1.5 times of the previous search radius, that is, rb=1.5×ra, where rb is the search radius after iteration, but not limited thereto. The present disclosure does not limit the radius size when iterating the first search radius.

[0060] During the process of iterating the first search radius, when the clustering processing result after a certain iteration meets the preset accuracy requirement, the corresponding search radius is determined as the target search radius; or, when the iteration reaches the preset maximum number of times here, the corresponding search radius is determined as the target search radius, and the target search radius is used to process the message data to determine the cluster message.

[0061] By iterating the first search radius, each message can be utilized to the maximum extent, while reducing the influence of overfitting in the clustering process.

[0062] Step 205: Determine the first byte of the cluster message based on the message feature vector of the cluster message.

[0063] In some embodiments, there is a vector indicating the header information of the message data in the message feature vector. Therefore, the first byte of each message data in the cluster message can be determined based on the message feature vector of the cluster message, so as to subsequently determine the positions of the remaining bytes in the message data based on the position of the first byte.

[0064] In some embodiments, the first byte may be the first byte of the message, but this is not limited. The remaining byte may also be determined as the first byte based on information characteristics of the message.

[0065] Step 206: Based on the position of the first byte, determine the offset position of the second byte of the cluster message relative to the first byte.

[0066] In some embodiments, the second byte is different from the first byte, that is, the second byte is all the bytes in the message except the first byte.

[0067] In some embodiments, the position of the first byte may be used as a reference to traverse each byte in the message data and calculate the offset position of each byte relative to the first byte, thereby determining the position of the second byte.

[0068] Step 207: Based on the position and offset position of the first byte, a classification result of the message data is determined using a mapping relationship between frequent byte items and byte positions in a preset model.

[0069] In some embodiments, since the value frequencies corresponding to each position of different types of messages are different, the pre-trained model can determine the classification result of the message based on the value of each byte in the message data and the mapping relationship between frequent byte items and byte positions stored in itself.

[0070] The frequent byte item refers to the byte value with the highest frequency at the same byte position of a certain type of message.

[0071] For example, during the model training process, there are 8 training messages of the same type, where the values ​​of the first bytes of these 8 messages are 1, 1, 5, 4, 2, 2, 1, 1 respectively. The pre-trained model can determine the frequent byte item of the first byte of this type of message as 1.

[0072] For example, in the pre-trained model, the mapping relationship between the frequent byte items and byte positions of the control class messages is that the frequent byte item of the first byte is determined to be 1, and the frequent byte item of the second byte is 3. Then, when the first byte value of message #4 in the class cluster message is 1 and the value of the second byte is 3, message #4 can be determined as a control class message.

[0073] Step 208: Based on the classification result, the message data is sent to the corresponding processing device for processing by using the preset mapping relationship between the classification result and the processing device.

[0074] In some embodiments, when the classification result is a control class message, the message data is processed using an X86 processing device; when the classification result is a stateless message, the message data is forwarded using a high-speed port; when the classification result is a stateful message, the message data is processed using an NP network processor, and / or, when the classification result is a stateful message, the message data is processed using a SOC FPGA processor.

[0075] Specifically, in some embodiments, when the classification result is a stateful message, the NP network processor can be used to query and access the session connection library, and the session connection library is communicatively connected to the NP network processor; if the result of the query access is whether there is a historical session connection of the message data in the session connection library, the historical session connection is used to forward the message data, thereby improving the message processing efficiency.

[0076] Specifically, in some embodiments, when the classification result is a stateful message, the SOC FPGA processor is used to perform security analysis on the message data; when the security analysis is a secure message, the SOC FPGA processor is used to forward the message data; when the security analysis is an illegal message, the SOC FPGA processor is used to discard the message data, thereby preventing the device from being attacked by illegal messages.

[0077] Based on the above embodiment, the SOC FPGA processor can also count the number of repeated occurrences of message data; when the number of repeated occurrences is greater than a preset value, the SOC FPGA processor is used to send the forwarding processing information of the message data to the NP network processor, so that the NP network processor can add the session connection of the message data to the session connection library, so that when the repeated message appears again, the SOC FPGA processor can send the message to the NP processor, so that the NP processor calls the connection in the session connection library to process the message, thereby improving the processing efficiency of the message.

[0078] Example 1: If the network message is a control message, it is handed over to the X86 processor for processing; the X86 processor can process the control message by virtue of its own stability advantage, thereby ensuring the stability of the network switching device.

[0079] Example embodiment 2, if the network message is a stateless message, it is directly forwarded by the interactive chip through the high-speed port;

[0080] For example, if the values ​​of the frequent byte items of a message at different positions are relatively fixed, it may indicate that the message is stateful because it has a certain regularity. On the contrary, if the values ​​of the frequent byte items of a message at different positions vary greatly, it may indicate that the message is stateless because it does not have obvious regularity.

[0081] For stateless messages, that is, stateless services, the processing of a single request does not depend on other requests. In other words, all the information required to process a request is either included in the request or can be obtained from the outside (such as a database). The chip or server itself does not store any information. The stateless message is handed over to the P4 programmable chip for processing, and the fast processing speed can be used to achieve fast processing of stateless messages. At the same time, since stateless messages have no memory requirements, the defect of low memory of the P4 programmable chip can be overcome.

[0082] Example embodiment 3, if the network message is a stateful message, the stateful message is respectively handed over to the NP network processor and the SOC FPGA processor for processing:

[0083] For stateful messages, a handshake is used to connect and the stateful messages are handed over to the NP network processor for processing. By establishing a session pool (i.e., the above-mentioned session connection pool), the connected session is called to reduce the time spent on repeatedly establishing connections. The NP network processor data plane executes session search and creation functions, and designs an efficient ultra-large session storage architecture, which can effectively solve the problem of insufficient session table entries.

[0084] Example 3 may include the following two situations:

[0085] In case 1, the NP network processor performs a quick query to check whether there is a historical session connection for the current network message in the session connection library. If there is, the previous session connection is directly called to complete network data forwarding.

[0086] In case 2, the SOC FPGA processor performs security checks on the same batch of stateful messages, performs security level analysis on the received data messages, sends the legal messages back to the switching chip, performs internal switching and then forwards them outward (network requests corresponding to the whitelist), while some illegal messages similar to message attacks are directly discarded (network requests corresponding to the blacklist), and the number of times the message appears is counted, and the network connection with the repeated number of times is reported to the NP network processor to establish a session connection for it.

[0087] Furthermore, in case 1, the NP network connection processor calculates the information entropy of the session connection. If the network connection is more orderly and stable, the information entropy is lower. Information entropy lower than the preset threshold is handed over to the X86 processor for processing, and information entropy greater than or equal to the preset threshold is processed by the NP network connection processor for stateful message data.

[0088] Since X86 processors are usually used for general computing tasks, NP network connection processors are specially designed to process network packets. Handing ordered packets to X86 processors for processing can make full use of the general computing power of X86 processors, while handing out-of-order packets to NP network connection processors for processing can give full play to their specially optimized network packet processing capabilities. This division of labor can make more efficient use of system resources and improve performance. Ordered packets tend to come from reliable connections or are sorted, so they may be easier to process and pass to applications quickly. This helps reduce processing delays and improve user experience, especially for applications that require low latency, such as real-time audio and video communications or online games. By handing out-of-order packets to NP network connection processors for processing, mixed traffic from different network connections can be better handled. This helps to better manage network traffic, reduce congestion and packet loss, and improve the overall reliability of the network. By classifying packets according to information entropy, network security can be improved. Ordered packets may be easier to perform deep packet inspection and traffic analysis in order to detect potential threats or attacks. Out-of-order packets can be sent to dedicated network connection processors for higher-level analysis and processing to further improve network security. At the same time, this setup has good scalability. If you need to increase processing power, you can easily add more X86 processors or NP network connection processors to accommodate larger network traffic and more complex processing requirements.

[0089] Among them, the information entropy can be calculated by the following formula:

[0090]

[0091] Where: H(X) represents information entropy; X is the set of IP addresses corresponding to a certain IP address, xi is one of the IP addresses; p(xi) represents the ratio of the number of IP addresses to the number of all IP addresses; the frequency of occurrence of each different IP address is recorded as f1, f2...fn, where M is the number of different IP addresses.

[0092] In summary, the message processing method proposed in the present disclosure includes: obtaining message data; determining a message feature vector based on the message data; determining the median information and message distance of the message data based on the message feature vector; clustering the message data based on the median information and the message distance to obtain a cluster message; determining the first byte of the cluster message based on the message feature vector of the cluster message; determining the offset position of the second byte of the cluster message relative to the first byte based on the position of the first byte, the first byte is different from the second byte; determining the classification result of the message data based on the position and offset position of the first byte using the mapping relationship between the frequent byte items and the byte positions in the preset model; based on the classification result, using the preset mapping relationship between the classification result and the processing device, sending the message data to the corresponding processing device for processing. The method disclosed in the present disclosure realizes the recognition of message types by using a clustering algorithm and a frequent pattern mining algorithm, which can improve the accuracy of message classification, and can distribute different types of messages to different processors for rapid processing, thereby improving the efficiency of message processing.

[0093] Therefore, this solution has the following beneficial effects:

[0094] 1. The method disclosed in the present invention realizes the recognition of message types by utilizing clustering algorithm and frequent pattern mining algorithm, which can improve the accuracy of message recognition.

[0095] 2. By allocating different types of messages to different processors for processing, the efficiency of message processing is improved.

[0096] The following is an exemplary description of this scenario:

[0097] like Figure 3 As shown, the acquisition of the pre-trained model in this solution may include the following steps:

[0098] 301, prepare training data for the model to be trained.

[0099] Connect the device that needs to perform message recognition to the network environment where it needs to be used, and collect and prepare message data sets for training and testing within a preset period (for example, 1 day). The data set should contain different types of message samples, and each sample should have a corresponding label to indicate its message type.

[0100] 302. Perform feature extraction on the training data to obtain a feature vector;

[0101] Useful features are extracted from the message data and all the features are formed into a feature vector, which includes the protocol field, message length, message header information, and message load characteristics.

[0102] 303. Create label encodings for training data.

[0103] Convert the message type label to a numerical form using one-hot encoding or other methods so that the algorithm can understand and process it. For example, the label encoding of control messages is [1,0,0], stateless messages is [0,1,0], and stateful messages is [0,0,1]. Create label encodings for training data based on the above label setting rules.

[0104] 304. Divide the training data into a training set and a test set.

[0105] The training set is used to train the algorithm, and the test set is used to evaluate the performance of the algorithm.

[0106] 305. Train the model to be trained to obtain a pre-trained model.

[0107] The model to be trained includes a clustering algorithm model using an adaptive minimum number of messages and a search radius, wherein the training process includes the following steps:

[0108] 1. Get the median information of the message, and determine 1 / 4 of the message median as the initial search radius ra; use the byte size of the message as the identification feature, calculate the median of all message sizes (for example: if the message byte size is 10 bytes, then the median is 5 bytes), and then use 1 / 4 of the median as the initial search radius ra (to determine the clustering radius), and perform clustering based on the distance between messages.

[0109] 2. The new search radius after each iteration is rb = 1.5 × ra

[0110] Repeat step 2 until the accuracy meets the requirements, or the number of iterations exceeds the preset number. The distance between messages is calculated using the following formula:

[0111]

[0112] Among them, len is the message length. In order to maximize the use of each message, while reducing the impact of overfitting in the clustering process.

[0113] Based on message clustering, the extraction of protocol keywords from each cluster of messages is mainly done by frequent pattern mining algorithms, which mainly include:

[0114] 1) Get the first byte from the message in the message data and use it as a reference point.

[0115] 2) Traverse each byte in the message and calculate the offset position of each byte relative to the first byte.

[0116] 3) Count the frequency of the byte values ​​at each position, and select the bytes with high frequency as the fixed-value frequent byte items.

[0117] 4) Associate each frequent byte item with its corresponding position to form a mapping relationship between byte value and byte position, thereby obtaining a pre-trained model.

[0118] By this method, the fixed-value bytes that frequently appear at different positions can be mined and associated with the positions. The results obtained by the above algorithm are used for data for message type identification, which can be more objective and accurate.

[0119] 306. Pre-training model performance evaluation;

[0120] Use the test set to evaluate the trained pre-trained model and calculate the model's accuracy, precision, recall and other indicators.

[0121] 307. Pre-training model performance optimization

[0122] Optimize the pre-trained model based on the evaluation results, adjust the model parameters, and select more appropriate features.

[0123] Figure 4 A system architecture for applying this method is shown in Figure 4 As shown, the system architecture includes:

[0124] 1. High-speed port: To meet the fast and high throughput requirements of switching equipment, the capacity of the high-speed port is at least 500G

[0125] 2. P4 interactive chip: used to determine the message type. Messages are divided into stateless messages, control messages and stateful messages. Stateless messages are directly forwarded by the P4 interactive chip, control messages are processed by X86, and stateful messages are handed over to the SOC FPGA processor and NP network processor.

[0126] 3. X86 processor runs the operating system: providing complete configuration distribution and system management functions.

[0127] 4. SOC FPGA processor: SoC FPGA chip resources are mainly divided into two parts: PL and PS. PL is the programmable logic part, which refers to the FPGA part. PS is the programmable system part, which refers to the embedded processor part. PS and PL can communicate through the bus. Stateful messages include: management messages, layer 2 / 3 protocol messages, and encrypted messages. The management messages and layer 2 / 3 protocol messages are transmitted to the PS programmable system part of the SoC FPGA. This part of the function will unpack and process the received data messages according to the relevant protocols, control and manage the relevant equipment for the management messages, and process the layer 2 / 3 protocol messages for the layer 2 / 3 protocol. The processed data messages are repacked and sent to the switching chip, which is then transmitted to the network interface through the switching chip to interact with external devices. The security and encrypted messages are transmitted to the PL programmable logic part of the SoC FPGA, and the programmable logic part uses programmable features to set security rules (i.e. whitelist and / or blacklist) according to users or specific environments.

[0128] 5. According to the predefined message header, the NP network processor performs session search and creation functions on the data plane, designs an efficient ultra-large session storage architecture, and has functions such as message-triggered session timeout reset and session timeout aging. It also provides complete monitoring and management capabilities on the X86 processor side, with session custom display, session timeout management, table item usage statistics, forwarding and packet loss statistics, and chip health status functions. Session connection is a historical connection. Through the Session connection pool in the NP network processor, the previous connection can be quickly called, thereby achieving the effect of improving the network.

[0129] Figure 5 FIG. 5 is a schematic diagram of a message processing device 500 provided in an embodiment of the present disclosure. Figure 5 As shown, the message processing device includes:

[0130] An acquisition unit 510, configured to acquire message data;

[0131] A clustering unit 520, configured to perform clustering processing on the message data using a pre-trained model to obtain cluster messages;

[0132] A classification unit 530, configured to determine a classification result of the message data based on the clustered messages using a frequent pattern mining algorithm in a pre-trained model;

[0133] The processing unit 540 is used to send the message data to the corresponding processing device for processing based on the classification result and using the preset mapping relationship between the classification result and the processing device.

[0134] In some embodiments of the present disclosure, the clustering unit 520 is also used to determine a message feature vector based on the message data; determine the median information and message distance of the message data based on the message feature vector; and cluster the message data based on the median information and the message distance to obtain a cluster message.

[0135] In some embodiments of the present disclosure, the clustering unit 520 is also used to determine the first search radius of the pre-trained model based on the median information; perform initial clustering processing on the message data based on the median information, the first search radius and the message distance; iterate the first search radius based on the clustering result of the initial clustering processing; when the number of iterations reaches a preset maximum value, or when the accuracy of the clustering result after the iteration meets the preset requirements, stop the iteration to obtain the target search radius; determine the cluster message based on the target search radius.

[0136] In some embodiments of the present disclosure, the classification unit 530 is also used to determine the first byte of the cluster message based on the message feature vector of the cluster message; determine the offset position of the second byte of the cluster message relative to the first byte based on the position of the first byte, and the first byte is different from the second byte; based on the position and offset position of the first byte, determine the classification result of the message data by utilizing the mapping relationship between frequent byte items and byte positions in a preset model.

[0137] In some embodiments of the present disclosure, the processing unit 540 is also used to perform message processing on message data using an X86 processing device when the classification result is a control class message; to forward the message data using a high-speed port when the classification result is a stateless message; to perform message processing on the message data using an NP network processor when the classification result is a stateful message, and / or to perform message processing on the message data using a SOC FPGA processor when the classification result is a stateful message.

[0138] In some embodiments of the present disclosure, the processing unit 540 is also used to, when the classification result is a stateful message, use the NP network processor to query and access the session connection library, and the session connection library is communicatively connected to the NP network processor; when the result of the query access is whether there is a historical session connection of the message data in the session connection library, use the historical session connection to forward the message data.

[0139] In some embodiments of the present disclosure, the processing unit 540 is also used to perform security analysis on the message data using the SOC FPGA processor when the classification result is a stateful message; when the security analysis is a secure message, forward the message data using the SOC FPGA processor; when the security analysis is an illegal message, discard the message data using the SOC FPGA processor; count the number of repeated occurrences of the message data using the SOC FPGA processor; when the number of repeated occurrences is greater than a preset value, send the forwarding processing information of the message data to the NP network processor using the SOC FPGA processor, so as to add the session connection of the message data to the session connection library using the NP network processor.

[0140] In summary, the message processing device proposed in the present disclosure includes: an acquisition unit for acquiring message data; a clustering unit for clustering the message data using a pre-trained model to obtain cluster messages; a classification unit for determining the classification result of the message data based on the cluster messages using a frequent pattern mining algorithm in the pre-trained model; and a processing unit for sending the message data to a corresponding processing device for processing based on the classification result and using a preset mapping relationship between the classification result and the processing device. The device of the present disclosure uses a clustering algorithm and a frequent pattern mining algorithm to realize the identification of message types, so that different types of messages can be distributed to different processors for rapid processing, thereby improving the efficiency of message processing.

[0141] It should be noted that: when the message processing device provided in the above embodiment performs message processing, it only uses the division of the above-mentioned program modules as an example. In actual applications, the above-mentioned processing can be assigned to different program modules as needed, that is, the internal structure of the message processing device is divided into different program modules to complete all or part of the processing described above.

[0142] Since the device provided in the embodiment of the present disclosure corresponds to the methods provided in the above-mentioned embodiments, the implementation of the method is also applicable to the device provided in the embodiment and will not be described in detail in this embodiment.

[0143] In the embodiments provided in the present application, the methods and devices provided in the embodiments of the present application are introduced. In order to implement the functions in the methods provided in the embodiments of the present application, the electronic device may include a hardware structure and a software module, and implement the functions in the form of a hardware structure, a software module, or a hardware structure plus a software module. A function of the functions may be executed in the form of a hardware structure, a software module, or a hardware structure plus a software module.

[0144] Figure 6 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present disclosure is shown in FIG. Figure 6 As shown, the electronic device 600 includes at least one processor 602; and a memory 601 communicatively connected to the at least one processor 602; wherein the memory 601 stores instructions executable by the at least one processor 602, and the instructions are executed by the at least one processor 602 to implement the steps of the message processing method described in the embodiment of the present disclosure; or, the instructions are executed by the at least one processor 602 to implement the steps of the message processing method described in the embodiment of the present disclosure.

[0145] It is understood that the electronic device also includes a communication interface. The various components in the electronic device are coupled together through a bus system. It is understood that the bus system is used to achieve connection and communication between these components. In addition to the data bus, the bus system also includes a power bus, a control bus and a status signal bus.

[0146] It can be understood that the memory 601 can be a volatile memory or a non-volatile memory, and can also include both volatile and non-volatile memories. Among them, the non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a magnetic random access memory (FRAM), a flash memory, a magnetic surface memory, an optical disk, or a compact disc read-only memory (CDROM); the magnetic surface memory can be a disk memory or a tape memory. The volatile memory can be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static random access memory (SRAM), synchronous static random access memory (SSRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM, SyncLink Dynamic Random Access Memory), and direct RAM bus random access memory (DRRAM, Direct Rambus Random Access Memory).The memory 601 described in this embodiment of the present solution is intended to include, but is not limited to, these and any other suitable types of memory.

[0147] The method disclosed in the above embodiment of the present disclosure may be applied to the processor 602, or implemented by the processor 602. The processor 602 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method may be completed by an integrated logic circuit of hardware in the processor 602 or an instruction in the form of software.

[0148] The processor 602 described above may be a general-purpose processor, a DSP, or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The processor 602 may implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this solution. A general-purpose processor may be a microprocessor or any conventional processor, etc. In combination with the steps of the method disclosed in the embodiments of this solution, it may be directly embodied as being executed by a hardware decoding processor, or executed by a combination of hardware and software modules in a decoding processor. The software module may be located in a storage medium, which is located in the memory 601, and the processor 602 reads the information in the memory 601 and completes the steps of the aforementioned method in combination with its hardware.

[0149] In an exemplary embodiment, the electronic device may be implemented by one or more application specific integrated circuits (ASIC), DSP, programmable logic device (PLD), complex programmable logic device (CPLD), FPGA, general purpose processor, controller, MCU, microprocessor, or other electronic components to execute the aforementioned method.

[0150] The embodiment of the present disclosure also provides a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to implement the steps of the message processing method described in the embodiment of the present solution when the computer instructions are executed.

[0151] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0152] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "illustrative embodiments", "examples", "specific examples" or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiments or examples are included in at least one embodiment or example of the present solution. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0153] Any process or method description in the flowchart or otherwise described herein may be understood to represent a module, fragment or portion of code that includes one or more executable instructions for implementing the steps of a specific logical function or process, and the scope of the preferred embodiment of the present scheme includes alternative implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in the reverse order depending on the functions involved, which should be understood by technicians in the technical field to which the embodiments of the present scheme belong.

[0154] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processing module, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in combination with these instruction execution systems, devices or apparatuses. For the purposes of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in combination with these instruction execution systems, devices or apparatuses. More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or more wires (control method), a portable computer disk box (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), a fiber optic device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium and then editing, interpreting or otherwise processing in a suitable manner if necessary, and then stored in a computer memory.

[0155] It should be understood that the various parts of the implementation of the present solution can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned implementation, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another implementation, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0156] A person skilled in the art may understand that all or part of the steps in the above-mentioned embodiment method may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiment.

[0157] In addition, each functional unit in each embodiment of the present solution can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into one module. The above-mentioned integrated module can be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. The above-mentioned storage medium can be a read-only memory, a disk or an optical disk, etc.

[0158] Although the implementation methods of the present scheme have been shown and described above, it can be understood that the above implementation methods are exemplary and cannot be understood as limitations on the present scheme. Ordinary technicians in this field can change, modify, replace and modify the above implementation methods within the scope of the present scheme.

Claims

1. A message processing method, characterized in that: The method comprises: Get message data; Using a pre-trained model to cluster the message data to obtain cluster messages; Based on the clustered messages, a frequent pattern mining algorithm in the pre-trained model is used to determine a classification result of the message data; Based on the classification result, the message data is sent to the corresponding processing device for processing by utilizing the preset mapping relationship between the classification result and the processing device.

2. The method according to claim 1, characterized in that The clustering of the message data using the pre-trained model to obtain cluster messages includes: Based on the message data, determining a message feature vector; Based on the message feature vector, determining the median information and the message distance of the message data; Based on the median information and the message distance, clustering processing is performed on the message data to obtain cluster messages.

3. The method according to claim 2, characterized in that The clustering process of the message data based on the median information and the message distance to obtain cluster messages includes: Based on the median information, determining a first search radius of the pre-trained model; Based on the median information, the first search radius and the message distance, performing initial clustering processing on the message data; Iterating the first search radius based on the clustering result of the initial clustering process; When the number of iterations reaches a preset maximum value, or when the accuracy of the clustering result after iteration meets the preset requirement, the iteration is stopped to obtain the target search radius; Based on the target search radius, the cluster message is determined.

4. The method according to claim 1, characterized in that: The determining of the classification result of the message data based on the cluster message and using the frequent pattern mining algorithm in the pre-trained model includes: Determine the first byte of the cluster message based on the message feature vector of the cluster message; Determine, based on the position of the first byte, an offset position of a second byte of the class cluster message relative to the first byte, the first byte being different from the second byte; Based on the position of the first byte and the offset position, a classification result of the message data is determined by utilizing a mapping relationship between frequent byte items and byte positions in a preset model.

5. The method according to claim 1, characterized in that The processing of the message data by using the processing device corresponding to the classification result based on the classification result and the preset mapping relationship between the classification result and the processing device includes: When the classification result is a control message, performing message processing on the message data by using an X86 processing device; When the classification result is a stateless message, forwarding the message data using a high-speed port; When the classification result is a stateful message, the message data is processed by using an NP network processor, and / or, when the classification result is a stateful message, the message data is processed by using an SOC FPGA processor.

6. The method according to claim 5, characterized in that When the classification result is a stateful message, using the NP network processor to perform message processing on the message data includes: When the classification result is a stateful message, the NP network processor is used to query and access the session connection library, and the session connection library is communicatively connected with the NP network processor; When the query access result is whether there is a historical session connection of the message data in the session connection library, the message data is forwarded using the historical session connection.

7. The method according to claim 5, characterized in that When the classification result is a stateful message, performing message processing on the message data by using a SOC FPGA processor includes: When the classification result is a stateful message, using the SOC FPGA processor to perform security analysis on the message data; When the security analysis is a secure message, forwarding the message data using the SOC FPGA processor; When the security analysis shows that the message is illegal, the SOC FPGA processor is used to discard the message data; Utilizing the SOC FPGA processor to count the number of repetitions of the message data; When the number of repetitions is greater than a preset value, the SOC FPGA processor is used to send forwarding processing information of the message data to the NP network processor, so that the NP network processor is used to add the session connection of the message data to the session connection library.

8. A message processing device, characterized in that: The device comprises: An acquisition unit, used for acquiring message data; A clustering unit, used for clustering the message data using a pre-trained model to obtain cluster messages; A classification unit, configured to determine a classification result of the message data based on the cluster message and using a frequent pattern mining algorithm in the pre-trained model; The processing unit is used to send the message data to the corresponding processing device for processing based on the classification result and using the preset mapping relationship between the classification result and the processing device.

9. An electronic device, characterized in that: include: a processor and a memory for storing a computer program capable of being executed on the processor, Wherein, when the processor is used to run the computer program, it executes the method described in any one of claims 1-7.

10. A non-transitory computer-readable storage medium storing computer instructions, characterized in that: The computer instructions are used to cause the computer to execute the method according to any one of claims 1-7.

Images