Small sample modulation identification method based on combination of multi-domain contrast learning and reinforcement learning

Abstract

A small sample modulation identification method based on combination of multi-domain contrast learning and reinforcement learning comprises the following steps: 1) obtaining a data set, dividing the data set into a base set, a support set and a verification set, converting the data set into a time domain, a frequency domain and a constellation diagram form, and obtaining a corresponding encoder and a common projection head; 2) inputting the base set data into an encoder for feature extraction, inputting the obtained features and accuracy as a state and an award into a decision body for reinforcement learning training to obtain a data enhancement action, inputting the original data and the enhancement data into the encoder and a projection head, and performing comparative learning training in a high-dimensional space to obtain a data enhancement action; the reinforcement learning training is repeated until the rewards are not increased any more; 3) loading and freezing encoder parameters, and training an attention module and a linear classifier by using a support set; and 4) loading and freezing all module parameters, and verifying the performance by using the verification set. According to the method, the signal model can have good performance in a small sample scene.

Description

Technical Field

[0001] The present invention belongs to the intersection of communication signal processing and artificial intelligence, and relates to signal processing and neural network models, and specifically to a contrastive learning combined with a reinforcement learning framework to solve the problem of small sample recognition of modulated signals. Background Art

[0002] With the rapid development and widespread application of wireless communication technology, the complexity of radio signals is increasing. Early radio signal recognition mainly relied on professional equipment and manual operation. However, with the increase of signal load, the expansion of spectrum range and the improvement of recognition accuracy requirements, the traditional signal recognition method that relies on manual and primary equipment is obviously unable to meet the needs of modern society. As an emerging technology, deep learning has made rapid progress in recent years and has achieved outstanding results in image classification, speech recognition, natural language processing and other fields. With its powerful computing power and pattern recognition capabilities, deep learning provides a new possibility for solving radio signal recognition problems. With the deepening of research, deep learning related algorithms have gradually been applied to key areas of radio signal systems (reference [1]: Tu Y, Lin Y, Wang J, et al. Semi-Supervised Learning with Generative Adversarial Networks on Digital Signal Modulation Classification [J]. Computers, Materials & Continua, 2018, 55 (2).), and have also achieved remarkable results in the field of debugging recognition. However, the application of deep learning usually relies on a large amount of labeled data, and the data collection and labeling process requires a lot of manpower and material resources, which to a certain extent limits the widespread application of deep learning technology in complex communication environments. Traditional supervised debugging recognition methods are only applicable to scenarios with a large amount of labeled data, but in practical applications, data is easy to collect but faces the common problem of high labeling costs. How to use limited labeled data to train an effective encoder is still a core challenge that needs to be solved in the field of debugging recognition.

[0003] To address the problem of scarce labels, many enhancement techniques and methods based on meta-learning and metric learning have emerged. These meta-learning and metric learning-based methods usually learn an efficient base model from a large dataset and adapt it to few-shot tasks using distance metrics or gradient optimization. However, in the field of radio signal processing, the availability of base classes is usually limited, and directly applying these methods may lead to insufficient generalization ability of the model. For the few-shot learning scenario of debugging recognition tasks, using unlabeled samples in a large amount of target datasets for contrastive learning is an effective solution. In recent years, as an emerging self-supervised learning technique (Reference [2]: Li K, Zhang Y, Li K, et al. Adversarial feature hallucination networks for few-shot learning [C] / / Proceedings of the IEEE / CVF conference on computer vision and pattern recognition. 2020: 13470-13479.), contrastive learning represents the features of unlabeled signals by exploring the latent supervision information in large-scale data. For example, Liu et al. first introduced contrastive learning into the modulation recognition task and only performed a single rotation transformation after enhancing the signal data (Reference [3]: Liu D, Wang P, Wang T, et al. Self-contrastive learning based semi-supervised radio modulation classification [C] / / MILCOM 2021-2021 IEEE Military Communications Conference (MILCOM). IEEE, 2021: 777-782.). Subsequently, Davaslioglu et al. proposed a method for learning the representation of unlabeled signal features using MoCo (Reference [4]: Davaslioglu K, S, Ertem M C, et al. Self-supervised RF signal representation learning for NextG signal classification with deep learning [J]. IEEE Wireless Communications Letters, 2022, 12(1): 65-69.).

[0004] Although the above methods have achieved certain results, there are still many challenges in meeting the demand for efficient feature representation of unlabeled modulation signals, especially the following technical problems: 1) Effective self-supervised contrast learning methods rely on exploring potential hidden features in unlabeled signals through data augmentation. However, various interferences in the actual electromagnetic environment can cause severe distortion and data distribution changes in the received signals, thus affecting the classification task of unlabeled signals based on contrast learning and limiting the full utilization of hidden features. Therefore, traditional data augmentation methods and parameters often need to be adjusted and optimized according to different environments. 2) A single data representation space often cannot enable the encoder to fully exploit diverse hidden features in unlabeled signals. This makes it impossible for some traditional modulation recognition methods to extract sufficient feature information from a single representation domain in small-sample scenarios, thereby affecting the performance of downstream classification tasks. Summary of the Invention

[0005] To overcome the deficiencies of the prior art, the present invention proposes a small-sample modulation recognition method based on multi-domain contrast learning combined with reinforcement learning. Specifically, the present invention uses reinforcement learning to replace manual selection of data augmentation methods and degrees, and combines three representation forms, namely IQ signals, frequency domain, and constellation diagrams, for unsupervised contrast learning. In the fine-tuning stage, a lightweight feature fusion module is used to extract important features from the three representation domains, and finally, they are fed into a classifier for classification.

[0006] To solve the above technical problems, the present invention provides the following technical solutions:

[0007] A small-sample modulation recognition method based on multi-domain contrast learning combined with reinforcement learning, comprising the following steps:

[0008] S1: Given a dataset and time-domain, frequency-domain, and constellation diagram encoders, divide the dataset into a base set, a support set, and a validation set. The base set is a large amount of unlabeled data for contrast learning, the support set is a small batch of labeled data for fine-tuning, and the validation set is used for performance evaluation. Then, all three are transformed into three representation forms: time domain, frequency domain, and constellation diagram;

[0009] S2: Use the base set for unsupervised pre-training. First, pass the base set data through the three encoders to obtain the original features. Then, use the features as the state, and the accuracy of the features evaluated by K-means as the reward is fed into the reinforcement learning decision body to obtain the selection and parameters of data augmentation as the action. Then, pass the original samples and the augmented samples through the three encoders to obtain their respective features, and perform contrast learning in the same high-dimensional space through the projection head. After the encoder contrast learning training is completed, repeat the reinforcement learning training loop to give the next set of actions until the reward no longer improves, then end the training;

[0010] S3: Use the support set for supervised fine-tuning. After the training in S2, obtain the parameters of the three encoders, freeze them, and then directly perform feature concatenation on the features obtained by the encoders using the labeled support set. The concatenated features are trained using the lightweight attention module and the linear classifier with the cross-entropy loss function;

[0011] S4: Use the validation set for performance evaluation. After the training in S3, obtain the parameters of the lightweight attention module and the linear classifier, freeze the encoder and these two modules, and then use the unused validation set to test the accuracy, and finally obtain its performance metrics;

[0012] Further, in step S1, the three encoders for feature extraction in the time domain (T), frequency domain (F), and constellation diagram (C) are respectively denoted as f θ (), f ξ () and f φ (), and the shared projection head used to project them into the same high-dimensional space is denoted as g(). Among them, θ, ξ, and φ are the parameters of the encoder. For a dataset, it is generally split into a training set and a test set, denoted as and where Then, 90% of the samples are removed of their labels as the base set, denoted as The remaining 10% of the are retained with their labels as the support set And the query set is the test set, where x represents the sample and y represents the label; Then, for a sample it is transformed into the time domain, frequency domain, and constellation diagram, denoted as and where i ∈ [1, B], B represents the batch size, and the same applies to the support set and the validation set. Additionally, the enhanced signals in the base set are respectively denoted as and

[0013] Furthermore, in step S2, the process of using the base set for unsupervised pre-training is as follows:

[0014] In the contrastive learning training loop, through the encoders f θ (), f ξ () and f φ () extract the feature of the original signal samples and as well as the augmented samples and in the base set, and then project them into the same feature space using the same projection head g(·). The time-domain, frequency-domain, and constellation diagram features are denoted as and and and and For the time domain, calculate the cosine similarity of these features, and the formula is as follows:

[0015]

[0016] where \(i,j\in[1,B]\), is a similarity matrix, where \(L\) represents the dimension of the features. Then the similarity matrices of the frequency domain and the constellation diagram are expressed as and Then the calculation formula of the contrastive learning loss function in the time domain based on cosine similarity is as follows:

[0017]

[0018] where \(\tau\) is the temperature coefficient, \(i\neq j\), \(i\in[1,B]\), \(j\in[1,B - 1]\). Among them, the numerator is the element on the diagonal of the similarity matrix, that is, the original sample and its corresponding augmented sample, and these two are positive sample pairs, which should be pulled closer to each other in the high-dimensional space. Each term in the denominator removes the diagonal element. The former term is the negative sample pair of the original sample and other original samples, and the latter term is the negative sample pair of the original sample and the augmented samples of other original samples. These two should be far away from each other in the high-dimensional space. Similarly, the intra-domain contrast loss of the frequency domain and the constellation diagram is and Then the formula of the total intra-domain contrast loss function is as follows:

[0019]

[0020] For multi-domain contrastive learning, only the intra-domain contrast loss function is far from enough. It cannot ensure that the features of each representation domain can establish connections with each other. Therefore, the inter-domain contrast loss function should also be added. For the time domain and the frequency domain, the formula is as follows:

[0021]

[0022] Its function is the same as that of the intra-domain one. Then there are 5 other inter-domain contrast loss functions on this basis, which are respectively and Then the formula of the total inter-domain contrast loss is as follows:

[0023]

[0024] Therefore, the formula of the total contrastive learning loss is as follows:

[0025] L total =\(\lambda L\)intra +(1 - λ)L inter (6)

[0026] Where λ is a hyperparameter used to control the proportion of intra - domain and inter - domain losses. After the contrastive learning training is completed, the features output by the encoder are merged as the state in the reinforcement learning loop. The state formula is as follows:

[0027]

[0028] Where concat is an operation that directly concatenates the features. After that, K - means is used for clustering to evaluate the accuracy acc. Therefore, the reward formula is as follows:

[0029] r t = acc t - acc t-1 (8)

[0030] Among them, the reward is set to the accuracy obtained from this round of contrastive learning minus the accuracy of the previous round. t represents a certain round of the cycle. After the reinforcement learning training is completed, the decision - making body gives an action, that is, the selection of 5 data augmentation methods and parameters. Therefore, the action formula is as follows:

[0031] a t =(a1, a2, a3, a4, a5) (9)

[0032] And the reinforcement learning training process is represented as a Markov decision process, and the formula is as follows:

[0033]

[0034] Where is the state space, is the action space, P(s t+1 |s t , a t ) represents the state transition probability, R(s t , a t ) is the reward function, γ is the coefficient that balances the current and future rewards. Then, the soft actor - critic algorithm (SAC) is used as the decision - making body, and the goal is to learn an optimal policy π(a t |s t ), and the formula is as follows:

[0035]

[0036] Where γ ∈ [0, 1), r t is the current reward, E rlIt is the reinforcement learning loop period. The optimization objective of SAC combines the expected reward and the entropy of the policy, and the formula is as follows:

[0037]

[0038] In the formula, Q π (s t ,a t ) is the state-action value function, which represents the expected cumulative reward after taking an action. is the decision entropy that encourages exploration. α is the entropy regularization coefficient, which balances exploration and reward maximization. In SAC, the Q function is updated by minimizing the following mean squared error objective, and the formula is as follows:

[0039]

[0040] In the formula, is the Q function calculated using the following formula:

[0041]

[0042] Finally, the update objective of the policy π is to maximize the following expectation:

[0043]

[0044] The optimal policy is obtained by gradient optimization update of the parameterized policy. After the reinforcement learning training is completed, three trained encoder parameters will be obtained and saved.

[0045] Furthermore, in step S3, the process of using the support set for supervised fine-tuning is as follows:

[0046] In S2, the three encoders obtained are f θ (), f ξ () and f φ (). Their weight parameters are loaded and frozen, and the support set is used to perform supervised fine-tuning on the lightweight attention module and the linear classifier. First, it is necessary to perform a concatenation operation on the features extracted by the encoder from the time domain, frequency domain, and constellation diagram. The formula is as follows:

[0047]

[0048] Among them, is a 3L-dimensional concatenated feature matrix, and then this feature matrix is input into the attention module. This module uses a small number of linear layers and activation layers to assign weights to the features of each representation domain, effectively preventing overfitting. Subsequently, the weighted features are passed to a linear classifier, which classifies the features into the desired categories, and finally obtains the probability of each category. The output result of the linear classifier is compared with the true label, and the cross-entropy loss function is used. The formula is as follows:

[0049]

[0050] In the formula, y i is the true label, and y i is the predicted label. This loss function improves the classification performance of the model by minimizing the difference between the predicted probability and the true label. After training, the parameters of the attention module and the linear classifier are saved.

[0051] In step S4, the process of using the validation set for performance evaluation is as follows:

[0052] After reloading and freezing the weight parameters of the three encoders, the attention module, and the linear classifier obtained in S3, the validation set is used for performance evaluation. By taking the highest probability among the probabilities of each class output by the linear classifier as the predicted label and comparing it with the true label, the accuracy rate is statistically calculated.

[0053] The beneficial effects of the present invention are as follows: Using multi-representation domain contrast learning enriches the features of the signal. Using reinforcement learning to replace the artificial method of selecting data augmentation and parameters explores the hidden features of the signal. These two enable the signal features of different categories to be effectively distinguished in the feature space. Subsequently, using a lightweight attention module and a linear classifier can effectively avoid the overfitting problem in the downstream fine-tuning stage for small-sample tasks. Finally, the present invention effectively solves the problems of insufficient richness and depth of signal features and overfitting of small samples, so that the signal model can have good performance in small-sample scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1 is a flowchart of the method provided by the present invention.

[0055] Figure 2 is a structural diagram of the attention module and the linear classifier. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0056] The following clearly describes the method system in the present invention with reference to the above-mentioned drawings by taking a typical signal classification data set as an example.

[0057] Referring to Figure 1 and Figure 2, A few-shot modulation recognition method based on multi-domain contrast learning combined with reinforcement learning, comprising the following steps:

[0058] S1: Given a dataset and time-domain, frequency-domain, and constellation encoders, divide the dataset into a base set, a support set, and a validation set. The base set is a large amount of unlabeled data for contrast learning, the support set is a small batch of labeled data for fine-tuning, and the validation set is for performance evaluation. Then, convert all three into three representation forms: time-domain, frequency-domain, and constellation.

[0059] In this embodiment, given the dataset RML2016.10a, the time-domain and frequency-domain encoders are determined to be ResNet1D, the constellation encoder is ResNet18, and the projection head consists of a simple normalization layer, a fully connected layer, and an activation layer.

[0060] The RML2016.10a dataset has 11 modulation classes, 8BPSK, QPSK, 8PSK, 16QAM, 64QAM, BFSK, CPFSK, and PAM4, as well as 3 analog modulations WB-FM, AM-SSB, and AM-DSB. Each sample is IQ dual-channel data, and the modulation classes are uniformly distributed in the range of -20dB to 18dB. The dataset is divided into a training set and a test set in a ratio of 4:1. The signal size of the training set is 176000, and the size of the test set is 44000, and they have the same number of modulation classes. The three encoders for feature extraction are respectively denoted as f θ (), f ξ (), and f φ (), and the common projection head used to project into the same high-dimensional space is denoted as g(). Among them, θ, ξ, and φ are the parameters of the encoder. For RML2016.10a, it is split into a training set and a test set, denoted as and where Then, for each signal-to-noise ratio in each category, randomly take 90% of the data and remove the labels. A total of 158400 samples are used as the base set, denoted as Leave the remaining 10% of the A total of 17600 samples retain the labels as the support set And the query set is the test set, a total of 44000 samples, where x represents the sample and y represents the label. Then, for a sample in terms of, convert it into time-domain, frequency-domain, and constellation, denoted as and where i ∈ [1, 110], 110 represents the batch size, and the same applies to the support set and the validation set. In addition, the enhanced signals in the base set are respectively denoted as and Finally obtain the reference Figure 1 on the left side in

[0061] S2: Use the base set for unsupervised pre-training. First, pass the base set data through three encoders to obtain the original features. Then, take the features as the state, and use the accuracy of the K-means evaluation of the features as the reward to be fed into the reinforcement learning decision body to obtain the selection and parameters of data augmentation as the actions. After that, pass the original samples and the augmented samples through the three encoders to obtain their respective features, and perform contrastive learning in the same high-dimensional space through the projection head. After the encoder contrastive learning training is completed, repeat the reinforcement learning training loop, give the next set of actions, and end the training until the reward no longer improves;

[0062] In this embodiment, during the contrastive learning training loop, through the encoders f θ (), f ξ (), and f φ (), the original signal samples of the base set and as well as the augmented samples and are subjected to feature extraction, and then projected into the same feature space using the same projection head g(·). Their time-domain, frequency-domain, and constellation diagram features can be expressed as and and and and Then perform the contrastive learning training loop to obtain the encoder parameters after one round of training.

[0063] The specific method for obtaining the contrastive learning training is as shown in formulas (1) to (6). Then, input the obtained state and reward into the reinforcement learning SAC decision body to perform the reinforcement learning training loop, and finally obtain the actions of data augmentation. Finally, continue with the contrastive learning training loop and the reinforcement learning training until the reward converges. The specific method for obtaining the reinforcement learning training is as shown in formulas (7) to (15). Finally obtain the reference Figure 1 the first part in. The parameters of the three encoders are obtained and saved.

[0064] S3: Use the support set for supervised fine-tuning. After the training in S2 is completed, obtain the parameters of the three encoders, freeze them, and then directly perform feature concatenation on the features obtained by passing the labeled support set through the encoders. The concatenated features are trained using the cross-entropy loss function through the lightweight attention module and the linear classifier;

[0065] In this embodiment, the three encoders obtained are f θ (), f ξ (), and f φ (), load and freeze their weight parameters, and use the support set to perform supervised fine-tuning on the lightweight attention module and the linear classifier. First, it is necessary to perform a concatenation operation on the features extracted by the encoder from the time domain, frequency domain, and constellation diagram. Among them, is a 384-dimensional concatenated feature matrix. Then, this feature matrix is input into the attention module, which uses a small number of linear layers and activation layers to assign weights to the features of each representation domain, effectively preventing overfitting. Subsequently, the weighted features are passed to the linear classifier, which classifies the features into the required categories and finally obtains the probability of each category. The output result of the linear classifier is compared with the true label, and the cross-entropy loss function is used.

[0066] The specific methods for obtaining the trained attention module and linear classifier are shown in formulas (16) and (17). Refer to Figure 1 the second part in Figure 2 . The parameters of the attention module and the linear classifier are obtained and saved.

[0067] S4: Use the validation set for performance evaluation. After training is completed in S3, the parameters of the lightweight attention module and the linear classifier are obtained, and the encoder and these two modules are frozen. Then, the accuracy is tested using an unused validation set, and finally its performance metrics are obtained;

[0068] The detection results of this embodiment are measured by the model detection accuracy. After reloading and freezing the weight parameters of the three encoders, the attention module, and the linear classifier obtained in S3, the validation set is used for performance evaluation. For the probability of each class output by the linear classifier, the highest probability is taken as the predicted label and compared with the true label, so as to count the accuracy.

[0069] Refer to Figure 1 the third part in

[0070] the few-shot modulation recognition method based on multi-domain contrast learning combined with reinforcement learning in this embodiment. Compared with the existing few-shot modulation recognition methods for signal models, using multi-representation domain contrast learning enriches the features of the signal, and using reinforcement learning instead of the artificial method of selecting data augmentation and parameters explores the hidden features of the signal. These two make the signal features of different classes can be effectively distinguished in the feature space. Then, using a lightweight attention module and a linear classifier can effectively avoid the overfitting problem in the few-shot task during the downstream fine-tuning stage. Thus, the signal model can have good performance in few-shot scenarios.

[0071] The content described in the embodiments of this specification is only an enumeration of the implementation forms of the inventive concept and is for illustrative purposes only. The protection scope of the present invention should not be regarded as limited to the specific forms stated in this embodiment, and the protection scope of the present invention also extends to equivalent technical means that can be conceived by those of ordinary skill in the art based on the inventive concept of the present invention.

Claims

1. A few-shot modulation recognition method based on multi-domain contrastive learning combined with reinforcement learning, characterized in that The method includes the following steps: S1: Given a dataset and time-domain, frequency-domain, and constellation diagram encoders, divide the dataset into a base set, a support set, and a validation set. The base set is a large amount of unlabeled data for contrastive learning, the support set is a small batch of labeled data for fine-tuning, and the validation set is used for performance evaluation. Then, convert all three into three representation forms: time domain, frequency domain, and constellation diagram. S2: Use the base set for unsupervised pre-training. First, pass the base set data through the three encoders to obtain the original features. Then, take the features as the state, and use K-means to evaluate the accuracy of the features as the reward, which is fed into the reinforcement learning decision body to obtain the selection and parameters of data augmentation as the action. Then, pass the original samples and the augmented samples through the three encoders to obtain their respective features, and perform contrastive learning in the same high-dimensional space through the projection head. After the encoder contrastive learning training is completed, repeat the reinforcement learning training loop to give the next set of actions until the reward no longer improves, then end the training. S3: Use the support set for supervised fine-tuning. After the training in S2 is completed, obtain the parameters of the three encoders and freeze them. Then, directly splice the features obtained by passing the labeled support set through the encoders. The spliced features are trained using the cross-entropy loss function through the lightweight attention module and the linear classifier. S4: Use the validation set for performance evaluation. After the training in S3 is completed, obtain the parameters of the lightweight attention module and the linear classifier, and freeze the encoders and these two modules. Then, use the unused validation set to test the accuracy, and finally obtain its performance metrics.

2. The few-shot modulation recognition method based on multi-domain contrastive learning combined with reinforcement learning according to claim 1, characterized in that, In step S1, the three encoders for feature extraction are respectively denoted as f θ (), f ξ () and f φ (), and the shared projection head for projecting into the same high-dimensional space is denoted as g(), where θ, ξ, and φ are the parameters of the encoders; for a dataset, it is split into a training set and a test set, denoted as and where Then, 90% of the samples are removed of their labels as the base set, denoted as The remaining 10% of the are kept with their labels as the support set while the query set is the test set, where x represents the sample and y represents the label; then, for a sample it is transformed into the time domain, frequency domain, and constellation diagram, denoted as and where i ∈ [1, B], B represents the batch size, and the same applies to the support set and the validation set. Additionally, the enhanced signals in the base set are respectively denoted as and 3. The few-shot modulation recognition method based on multi-domain contrastive learning combined with reinforcement learning according to claim 2, wherein, In step S2, the unsupervised pre-training process is as follows: In the contrastive learning training loop, the original signal samples of the base set are feature-extracted through the encoders f θ (), f ξ (), and f φ (), and the augmented samples and as well as and . Then, they are projected into the same feature space using the same projection head g(·), and their time-domain, frequency-domain, and constellation diagram features are denoted as and and and and For the time domain, the cosine similarity of these features is calculated, and the formula is as follows: where \(i,j\in[1,B]\), is a similarity matrix, then the similarity matrices of the frequency domain and the constellation diagram are expressed as and Then the calculation formula of the contrastive learning loss function in the time domain based on cosine similarity is as follows: Where τ is the temperature coefficient, i≠j, i∈[1,B], j∈[1,B-1]. Among them, the numerator is the element on the diagonal of the similarity matrix, that is, the original sample and its corresponding augmented sample, and these two are positive sample pairs, which should be pulled closer to each other in the high-dimensional space. Each term in the denominator removes the diagonal element. The former term is the negative sample pair of the original sample and other original samples, and the latter term is the negative sample pair of the original sample and the augmented samples of other original samples. These two should be far away from each other in the high-dimensional space. Similarly, the intra-domain contrast loss in the frequency domain and constellation diagram is obtained as follows. and Then the formula for the total intra-domain contrast loss function is as follows: For multi-domain contrastive learning, only the intra-domain contrast loss function is far from enough. It cannot ensure that the features in each representation domain can establish connections with each other. Therefore, the inter-domain contrast loss function should also be added. For the time domain and the frequency domain, the formula is as follows: Its function is the same as that within the domain. On this basis, there are five other inter-domain contrastive loss functions, namely and Then the total inter-domain contrastive loss formula is as follows: Therefore, the total contrastive learning loss formula is as follows: L total = λL intra +(1 - λ)L inter (6) In the formula, λ is a hyperparameter used to control the proportion of the intra-domain and inter-domain losses. After the contrastive learning training is completed, merge the features output by the encoders as the state in the reinforcement learning loop. The state formula is as follows: In the formula, concat is an operation that directly concatenates the features. Then, use K-means for clustering to evaluate the accuracy to obtain acc. Therefore, the reward formula is as follows: r t = acc t -acc t-1 (8) Among them, set the reward to the accuracy obtained in this round of contrastive learning minus the accuracy of the previous round. Then, after the reinforcement learning training is completed, the decision body gives the action, that is, the selection of 5 data augmentation methods and parameters. Therefore, the action formula is as follows: a t = (a1, a2, a3, a4, a5) (9) The reinforcement learning training process is represented as a Markov decision process, and the formula is as follows: wherein, is the state space, is the action space, P(s t+1 |s t , a t ) represents the state transition probability, R(s t , a t ) is the reward function, and γ is the coefficient for balancing current and future rewards; then, using the Soft Actor-Critic algorithm SAC as the decision-making body, the goal is to learn an optimal policy π(a t |s t ) that maximizes the expected cumulative reward, and the formula is as follows: where γ ∈ [0, 1), r t is the current reward, and E rl is the reinforcement learning cycle. The optimization objective of SAC combines the expected reward and the entropy of the policy, and the formula is as follows: Where, Q π (s t , a t ) is the state-action value function, representing the expected cumulative reward after taking an action, is the decision entropy that encourages exploration, and α is the entropy regularization coefficient that balances exploration and reward maximization. In SAC, the Q function is updated by minimizing the following mean squared error objective, and the formula is as follows: In the formula, is the Q function calculated using the following formula: Finally, the update goal of the policy π is to maximize the following expectation: By performing gradient optimization updates on the parameterized policy, obtain the optimal policy. After the reinforcement learning training is completed, three trained encoder parameters will be obtained and saved.

4. The few-shot modulation recognition method based on multi-domain contrastive learning combined with reinforcement learning according to claim 3, wherein, In step S3, the supervised fine-tuning process is as follows: In S2, the three obtained encoders are f θ (), f ξ >() and f φ >(). Their weight parameters are loaded and frozen, and the support set is used to perform supervised fine-tuning on the lightweight attention module and the linear classifier. First, it is necessary to splice the features extracted by the encoders from the time domain, frequency domain, and constellation diagram. The formula is as follows: Among them, is a 384-dimensional concatenated feature matrix. Then, this feature matrix is input into the attention module, which uses a small number of linear layers and activation layers to assign weights to the features of each representation domain, effectively preventing overfitting. Subsequently, the weighted features are passed to a linear classifier, which classifies the features into the desired categories and finally obtains the probabilities for each category. The output result of the linear classifier is compared with the true labels, and the cross-entropy loss function is used, as shown in the following formula: where y i is the true label, and y i is the predicted label. This loss function improves the classification performance of the model by minimizing the difference between the predicted probability and the true label. After training, the parameters of the attention module and the linear classifier are saved.

5. A few-shot modulation recognition method combining multi-domain contrast learning and reinforcement learning according to claim 4, characterized in that In step S4, the performance evaluation process is as follows: After reloading and freezing the weight parameters of the three encoders, attention modules, and linear classifiers obtained in S3, the validation set is used for performance evaluation. By taking the highest probability among the probabilities of each class output by the linear classifier as the predicted label and comparing it with the true label, the accuracy is statistically calculated.

Images