A Classification Method for Intelligent Communication Behaviors Based on Markov Decision Processes

Abstract

This invention provides an intelligent communication behavior classification method based on Markov decision processes, belonging to the field of machine learning. The invention includes: constructing a sampled dataset based on "state-action-reward" transition pairs, compressing state vector samples, determining the number of clusters, and completing the intelligent communication behavior classification. Based on Markov decision processes, this invention introduces multi-dimensional communication behavior information, including communication behavior transition patterns, communication interference actions, and reward information based on normalized interference effect evaluation. It compresses the communication behavior data using principal component analysis and employs the k-Means classification machine learning method, transforming traditional classification methods oriented towards static communication behavior information into classification methods oriented towards dynamic communication behavior information, effectively classifying unlabeled communication behavior data.

Description

Technical Field

[0001] This invention relates to an intelligent communication behavior classification method based on Markov decision processes, belonging to the field of machine learning. Background Technology

[0002] Unsupervised learning can learn the categories, transformations, or probabilities of data from unlabeled data. Essentially, it aims to obtain the underlying statistical rules and structures within the data, and is currently mostly applied to static data. However, in the context of communication warfare engineering applications, communication behavior data often possesses dynamic attributes. For example, in fields such as reinforcement learning, data modeling has potential transition outcomes, thus exhibiting Markov decision transition characteristics. In such cases, directly applying unsupervised learning methods for static data will fail to accurately uncover the statistical regularities and structures contained within the data.

[0003] "Automating DBSCAN via Deep Reinforcement Learning" utilizes reinforcement learning to train machine learning classification methods. However, the density-based spatial clustering of applications with noise (DBSCAN) algorithm struggles to achieve unsupervised learning clustering without prior knowledge to adjust parameters.

[0004] The paper "An improved DBSCAN algorithm using local parameter" proposes an improved DBSCAN method that avoids adjusting the Eps radius and MinPts threshold simultaneously, but it requires the cutoff distance parameter to be determined in advance and has high computational complexity.

[0005] The patent application CN202211332360.2, entitled "Hazard Source Early Warning Method, Device, Equipment and Medium Based on Unsupervised Learning", is also based on the DBSCAN method and requires prior knowledge to support parameter settings.

[0006] The patent application CN202211245210.8, entitled "An Example-Based Unsupervised Face Image Translation Method", achieves high-quality unsupervised learning based on deep neural network learning, but still requires a small number of samples for learning and fitting. Summary of the Invention

[0007] In view of this, the present invention provides an intelligent communication behavior classification method based on Markov decision processes. This method can effectively improve the classification performance of unlabeled communication behavior data.

[0008] The technical solution adopted in this invention is as follows:

[0009] A method for classifying intelligent communication behaviors based on Markov decision processes includes the following steps:

[0010] Step 1: Construct a sampled dataset based on the "state-action-reward" transition pairs;

[0011] Step 2: Compress the state vector samples;

[0012] Step 3: Determine the number of clusters;

[0013] Step 4: Complete the intelligent communication behavior classification.

[0014] Furthermore, the specific method of step 1 is as follows:

[0015] Step 1-1: Collect observation dataset

[0016] In the communication countermeasures experiment scenario, an initial communication behavior observation vector o is randomly selected. The observation vector consists of communication behavior parameters, and each communication behavior parameter is normalized according to its maximum value. Within the interference parameter range dimension N, all interference actions a are selected. o,n For n∈N, obtain the communication behavior observation data transferred to the target. and rewards based on normalized interference effect evaluation

[0017] The initial observation vector o is transferred to the observation data. and the rewards received Composition of state vector sample I:

[0018]

[0019] The labels y of the observation data provided in advance in the experimental scenario are used as the labels of the corresponding samples for classification effect verification; Step 1-2: Construct a state dataset based on feature engineering.

[0020] Repeat steps 1-1 for a total of T times to obtain T state vector samples I and observation data labels y, constructing a sample dataset X and a label dataset Y; the sample dataset X is:

[0021]

[0022] The labeled dataset Y is:

[0023] Y = [y1…y T ] T .

[0024] Furthermore, the specific method for step 2 is as follows:

[0025] The sample dataset X is compressed using the principal component analysis method in the sklearn open-source library; the variance percentage threshold is set to δ, and the fit_transform function is called to compress the sample dataset X to obtain the dimensionality-reduced low-dimensional dataset X′.

[0026] Furthermore, the specific method for step 3 is as follows:

[0027] Step 3-1: Preset the range of the number of clusters k, k∈[2,K);

[0028] Step 3-2: Use the k-Means machine learning method in the sklearn open-source library to classify the low-dimensional dataset X′; call the silhouette function to obtain the silhouette coefficients s for the current number of clusters k;

[0029] Step 3-3: Repeat step 3-2, incrementing the cluster number k by 1 in each iteration until the cluster number k reaches K, and save the silhouette coefficients obtained from all cluster numbers k as a silhouette coefficient vector S:

[0030] S = [s2, ..., s K-1 ]

[0031] Steps 3-4: Find the maximum value in the silhouette coefficient vector S, and use the corresponding index as the cluster number k:

[0032] k = argmaxS.

[0033] Furthermore, step 4 is specifically implemented as follows:

[0034] The k-Means machine learning method from the sklearn open-source library is used to classify a low-dimensional dataset X′ and obtain the corresponding classification labels Y′=[y1′…y′]. T ] T .

[0035] The beneficial effects of this invention are as follows:

[0036] 1. This invention combines the Markov decision transition characteristics in dynamic communication adversarial scenarios, adds transition relationships and reward information to the original observation data features, and constructs state data containing transition features through feature engineering. Finally, it uses machine learning methods to mine statistical patterns from unlabeled data and complete labeling and classification.

[0037] 2. This invention fully considers the Markov decision transition characteristics in dynamic communication adversarial scenarios, and can accurately mine the statistical regularities and structures contained in the data.

[0038] 3. The method of the present invention is simple and easy to implement, and can effectively improve the classification effect of unlabeled communication behavior data. Attached Figure Description

[0039] Figure 1 This is a flowchart of an intelligent communication behavior classification method based on Markov decision process in an embodiment of the present invention.

[0040] Figure 2 Profile coefficient plots for different numbers of clusters.

[0041] Figure 3 This is a classification effect diagram of an embodiment of the present invention.

[0042] Figure 4 This is a graph showing the classification performance of the typical machine learning method k-Means.

[0043] Figure 5 This is a comparison chart of classification accuracy. Detailed Implementation

[0044] The present invention will be further described below with reference to the accompanying drawings and specific embodiments.

[0045] A smart communication behavior classification method based on Markov decision processes is proposed. This method is based on Markov decision processes and introduces multi-dimensional communication behavior information, including communication behavior transfer rules, communication interference actions, and reward information based on normalized interference effect evaluation. Principal component analysis is used to compress communication behavior data, and k-Means classification machine learning method is used as the basis to transform the traditional classification method for static communication behavior information into a classification method for dynamic communication behavior information, which can effectively classify unlabeled communication behavior data.

[0046] like Figure 1 As shown, this method specifically includes the following steps:

[0047] Step 1: Construct a sampled dataset based on the "state-action-reward" transition pairs.

[0048] Step 1-1: Collect observation dataset

[0049] In the communication countermeasures experiment scenario, a random initial communication behavior observation vector o is selected. This vector consists of communication behavior parameters, namely, elevation angle, azimuth angle, communication modulation parameters, modulation order, operating mode, operating frequency range, information type, center frequency, data rate, and frequency hopping rate. To facilitate processing by artificial intelligence methods, feature engineering is performed, and each communication behavior parameter is normalized to its maximum value. Within the selectable interference patterns and the preset interference parameter range dimension N=16, all interference actions a are selected. o,nFor n∈N, obtain the communication behavior observation data transferred to the target. And obtain rewards based on normalized interference effect evaluation.

[0050] The initial observation vector o is transferred to the observation data. and the rewards received Composition of state vector sample I:

[0051]

[0052] The label y of the observation data provided in advance in the experimental scenario is used as the label of the corresponding sample for verification of classification effect.

[0053] Steps 1-2: Constructing a state dataset based on feature engineering

[0054] Repeat step 1-1 for a total of T = 1000 times, and construct sample dataset X and label dataset Y from T state vector samples I and observation data labels y.

[0055] The sample dataset X is:

[0056]

[0057] The labeled dataset Y is:

[0058] Y = [y1…y T ] T

[0059] Step 2: State vector sample compression.

[0060] Principal component analysis (PCA) from the sklearn open-source library is used to compress the sample dataset X. A variance percentage threshold δ = 0.95 is set, and the `fit_transform` function is called to compress the sample dataset X, resulting in a dimensionality-reduced low-dimensional dataset X′.

[0061] Step 3: Determine the number of clusters.

[0062] Step 3-1: Preset the range of the number of clusters k, k∈[2,20);

[0063] Step 3-2: Classify the low-dimensional dataset X′ using the k-Means machine learning method from the sklearn open-source library. Call the silhouette function to obtain the silhouette coefficients s for the current number of clusters k;

[0064] Step 3-3: Repeat step 3-2, incrementing the cluster number k by 1 in each iteration until the cluster number k reaches K. Save the silhouette coefficients obtained from all cluster numbers k as a silhouette coefficient vector S, as shown below. Figure 2As shown.

[0065] S = [s2, ..., s 19 ]

[0066] Steps 3-4: Find the maximum value in the silhouette coefficient vector S, and use the corresponding index as the number of clusters k.

[0067] k = arg maxS

[0068] In step 4: Implement intelligent communication behavior classification.

[0069] The k-Means machine learning method from the sklearn open-source library is used to classify a low-dimensional dataset X′ and obtain the corresponding classification labels Y′=[y1′…y′]. T ] T .

[0070] To further verify the effectiveness of the method, it is compared with the typical machine learning method k-Means classification. The number of experiments is preset to a range of m, m∈[1,100]. The classification accuracy is calculated. The classification labels Y′=[y1′…y′ are traversed. T ] T Different class labels are used to determine the mapping in the label dataset Y = [y1…y2]. T ] T Determine if the labels are of the same type. Count the number l of labels of the same type and calculate the classification accuracy p = l / k. Repeat the above process, incrementing the number of experiments m by 1 each time, until the number of experiments m reaches 100. Save the classification accuracy p obtained from all experiments m as a classification accuracy vector P:

[0071] P = [p1, ..., p] 100 ]

[0072] Calculate the mean of the classification accuracy vector P, and use it as the evaluation value of classification performance.

[0073] Figure 3 This is a classification effect diagram of this method. Figure 4 This image shows the performance of the k-Means classification method, a typical machine learning approach. (Comparison is needed.) Figure 3 and Figure 4 It is clear that the typical machine learning method k-Means produces chaotic classification results, while our method produces orderly classification results.

[0074] Figure 5 This is a comparison chart of classification accuracy. The comparison shows that the classification performance of this method is significantly better than that of the typical machine learning method k-Means.

[0075] In summary, this invention combines the Markov decision transition properties with feature engineering methods, reconstructs data features using communication behavior transition relationships and reward information, and, based on machine learning methods, achieves statistical pattern mining and labeling classification from unlabeled data without prior knowledge, effectively improving the classification performance of unlabeled communication behavior data.

Claims

1. A smart communication behavior classification method based on Markov decision processes, applied to a communication adversarial experiment scenario, characterized in that, Includes the following steps: Step 1: Construct a sampling dataset based on "state-action-reward" transition pairs; the specific method is as follows: Step 1-1: Collect observation dataset In a communication adversarial experiment scenario, an initial communication behavior observation vector is randomly selected. The observation vector consists of communication behavior parameters, including elevation angle, azimuth angle, communication modulation parameters, modulation order, operating mode, operating frequency range, information type, center frequency, data rate, and frequency hopping rate. Each communication behavior parameter is normalized to its maximum value. Within the interference parameter range dimension N, all interference actions are selected. , The interference action refers to the interference applied to communication behavior, and the communication behavior observation data transferred after the interference is applied is acquired. and rewards based on normalized interference effect evaluation ; Initial observation vector , transferred to observation data and the rewards received , forming state vector samples : The labels of the observation data provided in advance in the experimental scenario The labels for the corresponding samples are used to verify the classification performance. Steps 1-2: Constructing a state dataset based on feature engineering Repeat step 1-1 a total of T times to obtain T state vector samples. and observation data labels Build a sample dataset and label dataset ; Sample dataset for: Labeled dataset for: Step 2: Compress the state vector samples; Step 3: Determine the number of clusters; Step 4: Complete the intelligent communication behavior classification to achieve effective classification of dynamic, unlabeled communication adversarial data.

2. The intelligent communication behavior classification method based on Markov decision process according to claim 1, characterized in that, The specific method for step 2 is as follows: Principal component analysis (PCA) was used on the sample dataset using the sklearn open-source library. Perform compression; set the variance percentage threshold to [value missing]. Call the fit_transform function on the sample dataset Compression is performed to obtain a low-dimensional dataset. .

3. The intelligent communication behavior classification method based on Markov decision process according to claim 2, characterized in that, The specific method for step 3 is as follows: Step 3-1: Preset the range of the number of clusters k. ; Step 3-2: Use the k-Means machine learning method from the sklearn open-source library to process low-dimensional datasets. Perform classification; call the silhouette function to obtain the silhouette coefficient s for the current number of clusters k; Step 3-3: Repeat step 3-2, incrementing the cluster number k by 1 in each iteration until the cluster number k reaches K, and save the silhouette coefficients obtained from all cluster numbers k as a silhouette coefficient vector S: Steps 3-4: Find the maximum value in the silhouette coefficient vector S, and use the corresponding index as the cluster number k: 。 4. The intelligent communication behavior classification method based on Markov decision process according to claim 1, characterized in that, The specific method for step 4 is as follows: Using the k-Means machine learning method from the sklearn open-source library for low-dimensional datasets Classify the data and obtain the corresponding category labels. .

Images