Intelligent electromagnetic situation classification method based on markov decision process

Abstract

This invention provides an intelligent electromagnetic situation classification method based on Markov decision processes, belonging to the field of machine learning technology. The invention includes: sampling and constructing a dataset based on Markov decision processes, compressing state vector samples based on autoencoders, unsupervised automatic classification based on k-Means, deep learning generalization based on fully connected neural networks, and evaluating the classification performance. This invention provides an intelligent electromagnetic situation classification method based on Markov decision processes, addressing the need for classification of dynamic electromagnetic situation information based on Markov decision processes. It integrates information such as electromagnetic behavior transfer patterns into features, combines machine learning and deep learning methods to extract electromagnetic information features from high-dimensional data, and achieves accurate classification of unlabeled electromagnetic situation data in dynamic electromagnetic domain scenarios.

Description

Technical Field

[0001] This invention relates to an intelligent electromagnetic situation classification method based on Markov decision processes, belonging to the field of machine learning. Background Technology

[0002] Unsupervised learning has the ability to learn predictive models from unlabeled data. Most current unsupervised machine learning methods are designed for classification and prediction of static data. However, in the practical engineering environment of electromagnetic situation classification, dynamic electromagnetic situation data is often obtained after interactions in the electromagnetic domain, and its data structure exhibits Markov properties.

[0003] The improved method proposed in "DSets-DBSCAN: A parameter-free clustering algorithm" is an effective machine learning approach capable of classifying and identifying unlabeled data. However, this method has not yet been improved for dynamic data characteristics, and it requires prior knowledge, including information such as the maximum distance threshold for different data types.

[0004] The principal component analysis method proposed in the paper "Multiview PCA: A Methodology of Feature Extraction and Dimension Reduction for High-Order Data" is a high-performance unsupervised learning method that achieves dimensionality reduction and compression based on the features of the data itself, but it is difficult to determine the current category.

[0005] In the patent application CN202310010073.8 entitled "A method for visualizing satellite beam coverage area, server and storage medium", k-Means, as a typical machine learning method, has the characteristics of requiring less prior knowledge and being more robust compared to BECAN-type methods and principal component analysis-type methods. However, it is difficult to reveal the dynamic transfer rules behind the data by directly using analysis methods for static data in dynamic data structures.

[0006] The patent application CN202211339317.9, entitled "A Semi-Supervised Learning Method for Intelligent Recognition of Deep Valleys Based on Remote Sensing Imagery," pertains to semi-supervised learning. This method requires first labeling a small number of samples and implementing supervised learning, then using the generalization ability of deep neural networks to perform heuristic classification of the unlabeled data. Because this method involves deep neural networks, the absolute amount of training data is relatively large compared to the small overall sample size, in order to ensure stable convergence of the network parameters. Summary of the Invention

[0007] In view of this, the present invention provides an intelligent electromagnetic situation classification method based on Markov decision processes. Addressing the need for dynamic electromagnetic situation information classification based on Markov decision processes, this invention integrates information such as electromagnetic behavior transfer patterns into features, and combines machine learning and deep learning methods to extract electromagnetic information features from high-dimensional data, achieving accurate classification of unlabeled electromagnetic situation data in dynamic electromagnetic domain scenarios.

[0008] The technical solution adopted in this invention is as follows:

[0009] A smart electromagnetic situation classification method based on Markov decision processes includes the following steps:

[0010] Step 1: Sampling and constructing a dataset based on Markov decision processes;

[0011] Step 2: Compress the state vector samples based on the autoencoder;

[0012] Step 3: Implement unsupervised automatic classification based on k-Means;

[0013] Step 4: Implement intelligent electromagnetic situation classification based on fully connected neural networks.

[0014] Furthermore, the specific method of step 1 is as follows:

[0015] Step 1-1: Collect observation dataset

[0016] In the electromagnetic situation dynamic experiment scenario, an initial electromagnetic situation observation vector o is randomly selected, with the dimension of o_dim. All interference patterns and preset parameters a are selected. o,n n∈N, where N is the dimension of the disturbance action range, and new electromagnetic situation observation data are obtained through feedback. and rewards based on normalized interference effect evaluation

[0017] The initial electromagnetic situation observation vector o, and the new observed electromagnetic situation data. and rewards The state vector sample I is composed of (N+1)o_dim+N dimensions:

[0018]

[0019] The label y of the observation data provided in advance in the experimental scenario is used as the label of the corresponding sample;

[0020] Steps 1-2: Constructing a state dataset based on feature engineering

[0021] Repeat step 1-1 a total of T times to obtain T observation vectors o, state vector samples I, and observation data labels y, forming the observation dataset O, the state dataset X, and the label dataset Y; wherein, the observation dataset O is:

[0022]

[0023] The state dataset X is:

[0024]

[0025] The labeled dataset Y is:

[0026] Y = [y1…y T ] T .

[0027] Furthermore, the specific method for step 2 is as follows:

[0028] Step 2-1: Randomly select state vector samples I from the state dataset X to form batch_num_1 batch state datasets X′. b Each batch state dataset X′ b It contains batch_size_1 sample vectors;

[0029] Step 2-2: Construct an encoder and a decoder based on a fully connected neural network; the input dimension of the encoder is (N+1)o_dim+N, where c is the compression factor. To round down; the input dimension of the decoder is... The output dimension is (N+1)o_dim+N;

[0030] Steps 2-3: Randomly select an X′ from the batch_num batch state datasets. b After being processed by the encoder and decoder, the output X″ of the decoder is obtained. b :

[0031] X″ b =Decoder[Encoder(X′)] b )]

[0032] Steps 2-4: Calculate X′ b and X″ b The root mean square error loss1 is obtained and used for backpropagation and gradient optimization.

[0033] loss1 = mean[(X″ b -X′ b )2 ]

[0034] Where mean is used to calculate the average value;

[0035] Step 2-5: Set the loss threshold η1, and determine the relationship between the root mean square error loss1 and the loss threshold η1. If loss1>η1, then repeat steps 2-3 and 2-4; if loss1≤η1, save the encoder model.

[0036] Steps 2-6: Input the state dataset X into the encoder to obtain the compressed state dataset X′ with compressed vector dimensions.

[0037] Furthermore, the specific method for step 3 is as follows:

[0038] Step 3-1: Preset the range of the number of clusters k, k∈[2,K);

[0039] Step 3-2: Use the k-Means machine learning method in the sklearn open-source library to classify the compressed state dataset X′; call the silhouette function to obtain the silhouette coefficients s for the current number of clusters k;

[0040] Step 3-3: Repeat step 3-2, incrementing the cluster number k by 1 in each iteration until the cluster number k reaches K, and save the silhouette coefficients obtained from all cluster numbers k as a silhouette coefficient vector S:

[0041] S = [s2, ..., s K-1 ]

[0042] Steps 3-4: Find the maximum value in the silhouette coefficient vector S, and use the corresponding index as the cluster number k:

[0043] k = arg max S

[0044] Steps 3-5: Use the k-Means machine learning method from the sklearn open-source library to classify the compressed state dataset X′ and obtain the corresponding classification label set Y′=[y′1…y′] T ] T .

[0045] Furthermore, step 4 is specifically implemented as follows:

[0046] Step 4-1: Randomly select batch_num_2 batch state datasets O′ from the observation dataset O and the classification label dataset Y′. b and batch classification label set Y′ b Each contains batch_size_2 observation vectors and batch_size_2 classification labels;

[0047] Step 4-2: Build a fully connected neural network NET with an input dimension of o_dim and an output dimension of N;

[0048] Step 4-3: Randomly select a batch state dataset O′ from the batch_num_2 batch state datasets. b and batch classification label set Y′ b After processing through a fully connected neural network NET, the output Y″ is obtained. b :

[0049] Y″ b =NET(O′ b )

[0050] Step 4-4: Calculate Y′ b and Y″ b The cross-entropy error loss2 is calculated and used for backpropagation and gradient optimization.

[0051] loss2 = Cross_Entropy(Y′) b ,Y″ b )

[0052] Cross_Entropy is used to calculate the cross-entropy;

[0053] Step 4-5: Set the loss threshold η2, and determine the relationship between the root mean square error loss2 and the loss threshold η2. If loss2 > η2, then repeat steps 4-3 and 4-4; if loss2 ≤ η2, save the fully connected neural network NET model.

[0054] Steps 4-6: Input the observation dataset O into the fully connected neural network NET to obtain the classification label set Y″=[y″1…y″ T ] T .

[0055] The beneficial effects of this invention are as follows:

[0056] 1. This invention addresses the objective needs of electromagnetic situation classification tasks in engineering practice. Based on Markov decision processes, it integrates data transformation characteristics and reward information in electromagnetic behavior interaction processes to reconstruct the characteristics of observation data. It also utilizes an autoencoder to significantly compress the data, reducing the computational load for subsequent learning.

[0057] 2. This invention uses the k-Means machine learning method to perform zero-label classification of observation data features. It uses a deep neural network to learn the relationship between labels and observation data, and achieves generalized mapping, thereby realizing a logically clear intelligent electromagnetic situation classification. Attached Figure Description

[0058] Figure 1 This is a flowchart of the present invention.

[0059] Figure 2 This is the root mean square error convergence curve.

[0060] Figure 3 This is a graph showing the relationship between the number of clusters and the silhouette coefficient.

[0061] Figure 4 This is a curve showing the convergence of the cross-entropy error.

[0062] Figure 5 This is a classification effect diagram of the present invention.

[0063] Figure 6 This is a classification result of the classic machine learning method k-Means.

[0064] Figure 7 This is a comparison chart of classification accuracy. Detailed Implementation

[0065] The present invention will be further described below with reference to the accompanying drawings and specific embodiments.

[0066] A smart electromagnetic situation classification method based on Markov decision process, the overall process of which is as follows: Figure 1 As shown.

[0067] Specifically, the steps include the following:

[0068] Step 1: Sampling and constructing a dataset based on Markov decision processes.

[0069] Step 1-1: Collect observation dataset

[0070] In the electromagnetic situation dynamic experiment scenario, within the permissible range of the scenario, an initial electromagnetic situation observation vector o is randomly selected. The observation vector consists of electromagnetic parameters, namely pulse width, pulse arrival time, pulse repetition frequency, modulation type of pulse interval, range of pulse interval variation, signal carrier frequency, bandwidth, frequency variation range, elevation angle, and azimuth angle. To facilitate data classification, normalization is performed according to different electromagnetic parameters. The electromagnetic situation observation vector dimension is o_dim = 10, and all interference patterns and preset parameters a are selected. o,n n∈N, N=9 is the selectable interference action range dimension, and new electromagnetic situation observation data are obtained through feedback. and rewards based on normalized interference effect evaluation

[0071] The initial electromagnetic situation observation vector o, and the new electromagnetic situation observation data and rewards based on normalized interference effect evaluation The state vector sample I is composed of (N+1)o_dim+N dimensions:

[0072]

[0073] The label y of the observation data provided in advance in the experimental scenario is used as the label of the corresponding sample.

[0074] Steps 1-2: Constructing a state dataset based on feature engineering

[0075] Repeat step 1-1 for a total of T = 1000 times. Combine the T observation vectors o, the state vector samples I, and the observation data labels y to form the observation dataset O, the state dataset X, and the label dataset Y.

[0076] The observation dataset O is:

[0077]

[0078] The state dataset X is:

[0079]

[0080] The labeled dataset Y is:

[0081] Y = [y1…y T ] T

[0082] Step 2: State vector sample compression based on autoencoder.

[0083] Step 2-1: Randomly select state vector samples I from the state dataset X to form batch state datasets X′ of size batch_num_1 = 64. b Each batch state dataset X′ b It contains batch_size_1 = 64 sample vectors;

[0084] Step 2-2: Construct an encoder and a decoder based on a fully connected neural network. The input dimension of the encoder is (N+1)o_dim+N, where c=4 is the compression factor. To round down. The input dimension of the decoder is... The output dimension is (N+1)o_dim+N;

[0085] Steps 2-3: Randomly select an X′ from the batch_num batch state datasets. b After being processed by the encoder and decoder, the output X″ of the decoder is obtained. b:

[0086] X″ b =Decoder[Encoder(X′)] b )]

[0087] Steps 2-4: Calculate X′ b and X″ b The root mean square error loss1 is obtained and used for backpropagation and gradient optimization.

[0088] loss1 = mean[(X″ b -X′ b ) 2 ]

[0089] Where mean is the mean value.

[0090] Step 2-5: Set the loss threshold η1 = 0.01 and determine the relationship between the root mean square error loss1 and the loss threshold η1. If loss1 > η1, repeat steps 2-3 and 2-4; if loss1 ≤ η1, save the encoder model. The root mean square error convergence curve is shown below. Figure 2 As shown;

[0091] Steps 2-6: Input the state dataset X into the encoder to obtain the compressed state dataset X′ with compressed vector dimensions.

[0092] Step 3: Unsupervised automatic classification based on k-Means.

[0093] Step 3-1: Preset the range of the number of clusters k, k∈[2,K), K=20;

[0094] Step 3-2: Classify the compressed state dataset X′ using the k-Means machine learning method from the sklearn open-source library. Call the silhouette function to obtain the silhouette coefficients s for the current number of clusters k;

[0095] Step 3-3: Repeat step 3-2, incrementing the cluster number k by 1 in each iteration until the cluster number k reaches K, and save the silhouette coefficients obtained from all cluster numbers k as a silhouette coefficient vector S:

[0096] S = [s2, ..., s K-1 ]

[0097] Steps 3-4: Find the maximum value in the silhouette coefficient vector S, and use the corresponding index as the cluster number k. The relationship between the cluster number and the silhouette coefficient is as follows: Figure 3 As shown.

[0098] k = arg max S

[0099] Steps 3-5: Use the k-Means machine learning method from the sklearn open-source library to classify the compressed state dataset X′ and obtain the corresponding classification label set Y′=[y′1…y′] T ] T ;

[0100] In step 4: Deep learning generalization based on fully connected neural networks.

[0101] Step 4-1: Randomly select 64 batch state datasets O′ from the observation dataset O and the classification label dataset Y′. b and batch classification label set Y′ b Each contains batch_size_2 = 64 observation vectors and batch_size_2 classification labels.

[0102] Step 4-2: Build a fully connected neural network NET. Its input dimension is o_dim, and its output dimension is N;

[0103] Step 4-3: Randomly select a batch state dataset O′ from the batch_num_2 batch state datasets. b and batch classification label set Y′ b After processing through a fully connected neural network NET, the output Y″ is obtained. b :

[0104] Y″ b =NET(O′ b )

[0105] Step 4-4: Calculate Y′ b and Y″ b The cross-entropy error loss2 is calculated and used for backpropagation and gradient optimization.

[0106] loss2 = Cross_Entropy(Y′) b ,Y″ b )

[0107] Cross_Entropy is used to calculate the cross-entropy.

[0108] Steps 4-5: Set the loss threshold η2 = 0.01 and determine the relationship between the root mean square error loss2 and the loss threshold η2. If loss2 > η2, repeat steps 4-3 and 4-4; if loss2 ≤ η2, save the fully connected neural network NET model. The cross-entropy error convergence curve is shown below. Figure 4 As shown;

[0109] Steps 4-6: Input the observation dataset O into the fully connected neural network NET to obtain the classification label set Y″=[y″1…y″ T ] T .

[0110] In step 5: the classification effect is evaluated.

[0111] Step 5-1: The number of experiments is m, m∈[1,M], M=1000;

[0112] Step 5-2: Calculate the classification accuracy. Extract the observation vector o corresponding to different labels Y″, and determine the system label Y = [y1…y″] to which the observation vector o of the same classification label Y″ belongs. T ] T Are they of the same type? Count the number of categories l of similar tags, and calculate the classification accuracy p = l / k;

[0113] Step 5-3: Repeat step 5-2, incrementing the number of experiments m by 1 each time, until the number of experiments m reaches M, and save the classification accuracy p obtained from all experiments m as a classification accuracy vector P:

[0114] P = [p1, ..., p] M ]

[0115] Step 5-4: Classification performance evaluation. Calculate the mean of the classification accuracy vector P, and use it as the evaluation value for classification performance.

[0116] Figure 5 This is a classification effect diagram of the present invention. Figure 6 This image shows the classification performance of the classic machine learning method k-Means. (Comparison) Figure 5 and Figure 6 It can be seen that the classification effect of the present invention is significantly more regular than the classification results of the classic machine learning method k-Means. Figure 7 The comparison chart shows that the classification accuracy of this invention is superior.

[0117] In summary, this invention provides an intelligent electromagnetic situation classification method based on Markov decision processes. Addressing the need for dynamic information classification based on Markov decision processes, it integrates information such as transition patterns into features and combines machine learning and deep learning methods to extract features from high-dimensional electromagnetic parameter data. Evaluation results based on Monte Carlo experiments demonstrate that this invention can achieve accurate intelligent classification of unlabeled electromagnetic situation data in dynamic electromagnetic domain scenarios.

Claims

1. A Markov decision process based intelligent electromagnetic posture classification method characterized in that, Comprising the following steps: Step 1: Sampling based on Markov decision process and constructing dataset; in the dynamic electromagnetic situation experiment scene, randomly selecting the initial electromagnetic situation observation vector , the observation vector dimension is , selecting all interference patterns and preset parameters , , N is the dimension of the interference action range, and the new electromagnetic situation observation data and the reward based on the normalized interference effect evaluation are obtained through feedback; the starting electromagnetic scenario observation vector , the new observed electromagnetic scenario data and the reward , to form a state vector sample of dimension : ； Step 2: compressing the state vector sample based on the autoencoder; the specific mode is: Step 2-1: Randomly sampling state vector samples from the state dataset Step 2-2: Assembling batch_num_1 batch state datasets Step 2-3: Each batch state dataset contains batch_size_1 sample vectors Step 2-4: Each batch state dataset contains batch_size_1 sample vectors Step 2-5: Each batch state dataset contains batch_size_1 sample vectors Step 2-2: Constructing the encoder Encoder and the decoder Decoder based on the full connection neural network; the input dimension of the encoder Encoder is , wherein c is the compression multiple, is the floor function; the input dimension of the decoder Decoder is , and the output dimension is ; Step 2-3: Randomly select one from the batch_num batch state data sets , the decoder Decoder output is obtained after operation by the encoder Encoder and the decoder Decoder : Step 2-4: Compute and the root mean square error loss1 of and use this error for backpropagation and gradient optimization; Wherein mean is to take the mean; Step 2-5: Set loss threshold , determine the relationship between the root mean square error loss1 and the loss threshold , if loss1> , repeat step 2-3 and step 2-4; if loss1 , save the encoder Encoder model; Step 2-6: Set the state data Input the encoder Encoder to obtain the compressed state data set with compressed vector dimension ; Step 3: Unsupervised automatic classification based on k-Means; using the k-Means machine learning method in the sklearn open source library to compress the state data set Classify and obtain the classification label set of the corresponding data ; Step 4: realizing intelligent electromagnetic situation classification based on a fully connected neural network; the specific mode is: Step 4-1: Randomly sampling batch_num_2 batch state data sets from the observation data set O and the classification label data set L and batch classification label sets , respectively containing batch_size_2 observation vectors and batch_size_2 classification labels , respectively containing batch_size_2 observation vectors and batch_size_2 classification labels Step 4-2: Build a fully connected neural network NET with input dimension and output dimension ; Step 4-3: Randomly select one batch state dataset from the batch_num_2 batch state datasets and the batch classification label set , and obtain the output : Step 4-4: Compute and cross-entropy error and backpropagate and gradient optimize using this error; Wherein Cross_Entropy is to take the cross-entropy; Step 4-5: setting a loss threshold , judging the relationship between the root mean square error and the loss threshold , if > , repeating step 4-3 and step 4-4; if , saving the fully connected neural network NET model; Step 4-6: input the observation dataset O into the fully connected neural network NET to obtain the classification label set .

2. The intelligent electromagnetic situation classification method based on Markov decision process according to claim 1, characterized in that, The specific mode of step 1 is: Step 1-1: collecting an observation data set In the electromagnetic situation dynamic experiment scene, a starting electromagnetic situation observation vector is randomly selected , the observation vector dimension is , all interference patterns and preset parameters , are selected, N is the dimension of the interference action range, the feedback obtains new electromagnetic situation observation data and the reward based on the normalized interference effect evaluation ; the starting electromagnetic scenario observation vector , the new observed electromagnetic scenario data and the reward , to form a state vector sample , of dimension : providing a label to which observation data of an experimental scenario is provided in advance as a label of a corresponding sample ; Step 1-2: constructing a state data set based on feature engineering Repeat step 1-1 for T times to obtain T observation vectors , state vector samples and observation data labels , which constitute an observation data set O, a state data set and a label data set ; wherein the observation data set O is: State data set Is: Tag dataset For: 。 3.The intelligent electromagnetic situation classification method based on Markov decision process of claim 1, wherein, The specific mode of step 3 is: Step 3-1: preset the range of the number of clustering clusters k, ; Step 3-2: Using k-Means machine learning method in sklearn open source library to classify the compressed state data set ; call silhouette function to get silhouette coefficient s under current cluster number k Step 3-3: repeating step 3-2, and each time the clustering cluster number k is added by 1 until the clustering cluster number k reaches K, and the silhouette coefficients obtained by all clustering cluster numbers k are saved as a silhouette coefficient vector S: Step 3-4: finding the maximum value in the silhouette coefficient vector S, and taking the corresponding serial number as the clustering cluster number k: Step 3-5: Using k-Means machine learning method in sklearn open source library to classify the compressed state dataset and obtain the classification label set of the corresponding data .

Images