Hydrological monitoring data communication protocol intelligent analysis method based on similarity measurement

Abstract

The invention provides an intelligent analysis method for a hydrological monitoring data communication protocol based on similarity measurement, which comprises the following steps of: processing message data of the hydrological monitoring data communication protocol into text data in combination with a text classification thought, and performing template matching on the message data to be analyzed and a generated protocol template by adopting a similarity measurement scheme so as to obtain an intelligent analysis result of the hydrological monitoring data communication protocol. And the analysis of the message data is realized. The method comprises the following steps: carrying out word segmentation processing on message data according to semantic information of a protocol; adding position information of different segments into the data after word segmentation, and binding protocol semantics with the data to form a word segmentation effect in text classification; mining the protocol message by using word frequency-inverse document frequency (TF-IDF) to obtain a weight matrix of a high-frequency field as a feature matrix of the message; generating a protocol type template and a protocol field template by using the feature matrix; and finally, realizing accurate matching of the message and the template according to a similarity measurement rule, and realizing intelligent analysis of the protocol message according to a matching result of the template.

Description

Technical Field

[0001] The invention relates to the technical field of communication protocol analysis of a water conservancy monitoring system, and in particular to an intelligent analysis method for hydrological monitoring data communication protocol based on similarity measurement. Background Art

[0002] With the development of water conservancy informatization, the hydrological automatic monitoring system has developed from the automatic monitoring of single water level and rainfall data to the dynamic automatic monitoring of multiple parameters covering digital information such as temperature, humidity, wind speed, wind direction, rainfall, water quality, water flow rate, water volume, video images or pictures, etc. Information transmission is an important basic work of water conservancy informatization. In the hydrological monitoring system, communication protocols play a vital role in the safe and reliable operation of hydrological monitoring. At the same time, hydrological data acquisition and transmission technology has also developed rapidly. Different manufacturers and equipment use different data transmission protocols, which makes it very difficult for equipment to be compatible and share resources between systems. This is a great challenge for system developers.

[0003] Existing hydrological monitoring systems all use hard-coded methods to perform protocol parsing. This traditional protocol parsing method requires writing a corresponding parsing program for each protocol. The parsing program is complex and has poor scalability. It will cause a lot of redundancy in the development process and increase the workload of developers. This non-universal and poorly scalable protocol parsing method may require rewriting the parsing code once the underlying device or communication protocol changes, and the entire application needs to be recompiled, thereby increasing the system maintenance cost and the burden on developers. Summary of the invention

[0004] Purpose of the invention: Aiming at the communication protocol parsing problem in the hydrological monitoring system, the present invention combines the idea of ​​text classification and proposes an intelligent parsing method for hydrological monitoring data communication protocols based on similarity measurement, which can effectively parse different communication protocols in the hydrological monitoring system to solve the technical problems of poor flexibility, maintainability and scalability of existing water conservancy communication protocol processing methods.

[0005] Technical solution: To achieve the above-mentioned invention object, the present invention adopts the following technical solution:

[0006] A method for intelligent analysis of hydrological monitoring data communication protocols based on similarity measurement, the method comprising the following processes:

[0007] (1) Collect the uplink message data between the telemetry station and the central station in the hydrological monitoring system, and process the message data into hexadecimal text format.

[0008] (2) The semantics of the protocol is used to implement the word segmentation operation of the protocol text data. The protocol format is used as the basis for word segmentation. Due to the different lengths of the semantics of the hydrological communication protocol, word segmentations of different lengths are formed to increase the number of feature values ​​of the word segmentation.

[0009] (3) The position information of different semantic segments of the protocol is used as supplementary information to supplement the protocol data information after word segmentation. The semantics of the communication protocol is generated by the position of the protocol. The same field is placed at different positions in the protocol message to represent different semantics. The protocol semantics and data are bound to form a word segmentation effect in text classification.

[0010] (4) Word vectorization: Use term frequency-inverse document frequency (TF-IDF) to mine the protocol message, and calculate the weight of the word segmentation through TF-IDF to indicate the importance of each segment in the entire message text as a message feature. Calculate the frequency of each high-frequency field and obtain the weight matrix of the high-frequency field as the feature matrix of the message.

[0011] (5) Classify the protocols according to the protocol labels, calculate the feature matrix of the message as the feature sample set, and mine the representative messages of the protocol as the protocol type template. Use the clustering algorithm to automatically determine the number of clusters by calculating the silhouette coefficient, then calculate the center point in each cluster as a representative sample, and use the feature matrix corresponding to the sample as the protocol field template for the cluster. Generate a field dictionary based on the position of the field in the template.

[0012] (6) According to the message text similarity measurement rules, the message and the template are accurately matched, and the protocol message is parsed according to the template matching results.

[0013] Among them, the semantic segmentation according to the protocol in step (2) means that, taking the hydrological monitoring data communication protocol as an example, the message data contains a frame start symbol (1 byte), a central station address (2 bytes), a telemetry station address (10 bytes), a password (4 bytes), a function code (2 bytes), a message uplink and downlink identifier and length (4 bytes), a message length (n bytes), etc. According to the meaning of different segments of the protocol, the data is decomposed into multiple parts of different lengths to increase the number of characteristic values ​​of the protocol.

[0014] Among them, step (4) includes calculating the frequency of each field and obtaining the weight of the high-frequency field as the feature matrix of the message. The calculation formula is as follows:

[0015]

[0016] TF-IDF i，j =TF i，j *IDF i

[0017] Among them, n i，jRepresents frequent field t i In message d j The number of times it appears in the corpus, |D| represents the total number of messages in the corpus, |{j:t i ∈d j}| means including t i IDF stands for inverse document frequency.

[0018] In step (5), the definition of representative message is: in the same cluster, the message with the most similar features to the messages in other clusters is the representative message, and the protocol type template is constructed based on this. The solution formula is:

[0019]

[0020] Among them, p represents the protocol type of the message, N p ={x (1) , x (2) ,...x (m)} represents the feature set of the specification p, and m represents the number of message samples. The improved K-means clustering algorithm is used to determine the specification field template. The messages are clustered according to the Euclidean distance between the sample points in the feature set, and then the silhouette coefficient is used as an evaluation index to adjust the number of clusters. Finally, the center point of each cluster is used as a representative sample, and the feature matrix corresponding to the sample is used as the specification field template of the cluster.

[0021] In step (6), template matching is a pattern recognition method based on the message text similarity measurement rule. According to the message text similarity measurement formula, the similarity between the message and the template is calculated to determine the protocol type of the message. Among them, it includes the continuous consistency measurement of the message sequence, which is defined as: assuming that sequence C is a subsequence of sequences A and B, and the elements in C are adjacent in A and B, then sequence C is a continuous consistent subsequence of A and B. The sum of the lengths of all subsequences that meet the above conditions and do not overlap is called the continuous consistency index (CCI). The formula is:

[0022]

[0023] Where D is the matrix used to derive common substrings, S i It is the i-th element in the common string set S, A and B represent the two messages to be compared, and m represents the size of the substring set S.

[0024] Preferably, the similarity metric in step (5) is obtained by adding a continuous consistency index based on the longest common subsequence and Lewenstein distance. The calculation method is:

[0025]

[0026] Among them, A and B represent the protocol message text sequence, m and n represent the length of message A and B respectively, r is the difference coefficient used to limit the repetitiveness between LCS and CCI, L LCS Indicates the length of the longest common subsequence of A and B, L LD represents the Lewenstein distance between A and B.

[0027] Beneficial effects: Compared with the prior art, the present invention combines the idea of ​​text classification, processes the message data into text data, introduces protocol semantic information and position information during feature extraction, increases the number of feature values ​​of word segmentation, and reduces the interference of position information on the protocol format. A method based on similarity measurement is used to parse the hydrological monitoring data communication protocol. In view of the poor flexibility, scalability and maintainability of the existing parsing methods, a general parsing scheme is proposed, which effectively divides the communication protocols in the hydrological monitoring system into different categories and extracts templates for different protocols for matching and parsing, so as to achieve efficient and accurate parsing of different protocols, with obvious advantages and easy implementation. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 It is a flowchart of the specific implementation of the intelligent analysis method of hydrological monitoring data communication protocol based on similarity measurement.

[0029] Figure 2 It is a schematic diagram of reduced semantic segmentation.

[0030] Figure 3 It is a schematic diagram of the semantics of each field and the location information of the field.

[0031] Figure 4 It is a schematic diagram to supplement the semantic information of the specified position.

[0032] Figure 5 It is a comparison chart of the parsing accuracy of different types of specifications between the method of the present invention and the existing parsing method based on compilation technology. DETAILED DESCRIPTION

[0033] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0034] It should be understood that the reference to "embodiment" in this application means that the specific features, structures or characteristics described in conjunction with the embodiment may be included in at least one embodiment of the present application. The appearance of the phrase in various places in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those of ordinary skill in the art that the embodiments described in this application can be combined with other embodiments without conflict.

[0035] In order to solve the problem of poor flexibility and scalability of communication protocol parsing in hydrological monitoring systems, the present invention proposes the following Figure 1 The method for intelligently parsing hydrological monitoring data communication protocols based on similarity measurement is shown, comprising:

[0036] Step 1: Collect the uplink message data sent from the telemetry station at the central station of the hydrological monitoring system and process the message data into hexadecimal text format, such as Figure 2 As shown, the specified format is used as the basis for word segmentation to form word segmentations of different lengths and obtain the word segmentation results.

[0037] Step 2: Supplement the obtained word segmentation results with positional semantic information. The reduced positional semantic information is as follows: Figure 3 As shown in the figure, the position information of different segments is added to the word segmentation result. The word segmentation result after addition is as follows Figure 4 As shown in Figure 2, word segmentation without semantics will reduce the accuracy of protocol identification when converted into vector representation.

[0038] Step 3: Use TF-IDF to mine the protocol message based on the obtained word segmentation results, and count the frequency of each segment in the message to indicate the importance of each segment in the entire protocol text as a message feature. Calculate the frequency of each message segment, obtain the weight matrix of the message segment, and use the weight matrix as the feature matrix of the message.

[0039] Step 4: Generate the corresponding message template based on the obtained message feature matrix. First, classify the message according to the label, calculate the message feature matrix as the feature sample set, and obtain the most representative message data in the protocol message as the protocol type template according to the definition of the representative message. Use the K-Means clustering algorithm to cluster the messages according to the Euclidean distance between the feature sets of the protocol, and then dynamically adjust the number of classes based on the silhouette coefficient as the evaluation index. Finally, calculate the center point of each cluster as a representative sample, and use the feature matrix corresponding to the sample as the protocol field template for clustering.

[0040] Step 5: According to the generated protocol field template, the present invention uses template matching to implement the parsing of the protocol, and realizes the accurate matching of the message and the template according to a similarity measurement strategy. First, the continuous consistency index of the message to be parsed and the template is calculated, and then the continuous consistency index based on the longest common substring is introduced on the basis of the longest common subsequence and Lewenstein distance of the classical similarity measurement scheme to realize a more effective text similarity measurement scheme. In the template matching process, the similarity between the message to be parsed and each representative sample under the same protocol in the field template is first calculated. The field specification format corresponding to the representative sample with the highest similarity is the specification format of the current message to be parsed. By calculating the values ​​of the longest common subsequence, Lewenstein distance and continuous consistency index between the message to be parsed and each message in the template set, and obtaining similarity based on these values ​​to obtain a matching result, the field specification format set corresponding to the template is traversed to realize intelligent parsing of different types of protocol messages.

[0041] Figure 5 The bar graph is a comparison of the parsing accuracy results of 4 different types of hydrological monitoring communication protocols by the example method of the present invention and the existing parsing method based on compilation technology. In the experiment, 500 message data were selected for each of the 4 different types of communication protocols, and the message data were parsed using two methods respectively, and the parsing accuracy of each type of protocol was calculated. As can be seen from the figure, the accuracy of the protocol parsing method proposed in the present invention is greatly improved compared with the existing parsing method based on compilation technology, and it has good applicability for different types of protocol parsing. The experimental results prove that the hydrological monitoring data communication protocol parsing method based on similarity measurement realizes efficient and accurate parsing of different protocols, and can effectively solve the technical problems of poor flexibility, maintainability and scalability of the existing water conservancy communication protocol processing methods.

Claims

1. A method for intelligent analysis of hydrological monitoring data communication protocols based on similarity measurement, characterized in that: The central station collects the uplink message data from the telemetry station, processes the message data into text, extracts the characteristic data of messages of different protocol types to generate templates, and matches the message to be parsed with the generated template according to the template matching rules to realize message parsing; the specific steps are as follows: Step 1: Collect the uplink message data between the telemetry station and the central station in the hydrological monitoring system, and process the message data into a text format, that is, into a hexadecimal text format; Step 2: Use the semantics of the protocol to implement the word segmentation operation of the protocol text data, and use the protocol format as the basis for word segmentation. Due to the different lengths of the semantics of the hydrological communication protocol, different lengths of word segmentation are formed, which increases the number of feature values ​​of the word segmentation; Step 3: Use the location information of different semantic segments of the protocol as supplementary information to supplement the protocol data information after word segmentation. The semantics of the communication protocol is generated by the location of the protocol. The same field is placed at different locations in the protocol message to represent different semantics. Bind the protocol semantics and data to form the word segmentation effect in text classification. Step 4: Word vectorization. Use term frequency-inverse document frequency (TF-IDF) to mine the protocol message. Calculate the weight of the word segmentation through TF-IDF to indicate the importance of each segment in the entire message text as a message feature. Calculate the frequency of each high-frequency field and obtain the weight matrix of the high-frequency field as the feature matrix of the message. Step 5: Classify the protocols according to the protocol labels, calculate the feature matrix of the message as the feature sample set, and mine the representative messages of the protocol as the protocol type template; Use the clustering algorithm to automatically determine the number of clusters by calculating the silhouette coefficient, then calculate the center point in each cluster as a representative sample, and use the feature matrix corresponding to the sample as the clustering specification field template; generate a field dictionary based on the position of the field in the template; Step 6: According to the message text similarity measurement rules, the message and the template are accurately matched, and the protocol message is intelligently parsed according to the template matching result.

2. According to claim 1, a method for intelligent analysis of hydrological monitoring data communication protocol based on similarity measurement, characterized in that: The definition of representative message in step 5 is: in the same cluster, the message with the most similar features to messages in other clusters is the representative message, and the protocol type template is constructed based on this.

3. According to claim 1, a method for intelligent analysis of hydrological monitoring data communication protocol based on similarity measurement, characterized in that: The generation of the reduced field template in step 5 adopts a K-means-based clustering algorithm, clusters the message according to the Euclidean distance between the sample points of the feature set, and then uses the silhouette coefficient as an evaluation index to adjust the number of clusters. Finally, the center point of each cluster is used as a representative sample, and the feature matrix corresponding to the sample is used as the reduced field template of the cluster.

Images