A signal type recognition method based on reinforcement learning

Abstract

The application provides a signal type recognition method based on reinforcement learning, and relates to the technical field of artificial intelligence.The method comprises the following steps: performing zero-mean normalization processing and slicing operation on IQ data of training signals to generate IQ sample data; constructing a data set with one-hot labels; constructing a deep learning strategy network and a deep learning target network; iteratively performing a training process: extracting a sample data from the data set as a current state; selecting an action according to an epsilon-greedy algorithm, executing the action, and obtaining a reward and a next state according to the corresponding relationship of the data set; updating the parameters of the deep learning strategy network by a back propagation method according to the loss value calculation result and an optimizer, and outputting a deep learning strategy network model; inputting a signal to be recognized into the trained deep learning strategy network, and outputting a final signal type by a voting algorithm.The method can realize efficient recognition of communication signal types, and has strong signal recognition accuracy and generalization ability.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and more specifically to a signal type recognition method based on reinforcement learning. Background Technology

[0002] Signal recognition technology is one of the key foundational technologies in modern communications, electronic warfare, spectrum management, and wireless monitoring. Its core task is to identify the specific signal type from received wireless signals. Meanwhile, with the development of deep learning technology, it has also achieved initial success in tasks such as signal type recognition, spectrum sensing, and interference detection.

[0003] While deep learning offers a new solution for signal recognition, it still faces many challenges in practical applications. Deep models typically require large amounts of labeled data for training, but in scenarios such as wireless monitoring and non-cooperative communication, the complex signal sources and significant lack of labels hinder the effective application of supervised learning models.

[0004] Therefore, how to effectively integrate deep learning with signal recognition and design a highly adaptable, data-driven model with online learning capabilities has become an urgent problem to be solved. Summary of the Invention

[0005] To address the technical problems in related technologies, this invention provides a signal type recognition method based on reinforcement learning.

[0006] To achieve the above objectives, the technical solution adopted by the present invention includes:

[0007] According to a first aspect of the present invention, a signal type recognition method based on reinforcement learning is provided, comprising the following steps:

[0008] Step S1: Perform zero-mean normalization and slicing on the IQ data of the training signal to generate IQ sample data;

[0009] Step S2: Split the IQ sample data into I-channel data and Q-channel data, and construct a dataset with one-hot labels;

[0010] Step S3: Construct a deep learning policy network and a deep learning target network with identical structures;

[0011] Step S4: Iteratively execute the following training process:

[0012] Step S4-1: Extract a sample data from the dataset in sequence as the current state;

[0013] Step S4-2: According to The greedy algorithm selects actions, and the actions are output based on probabilities from a deep learning policy network model or random actions;

[0014] Step S4-3: Execute the action, obtain the reward and next state according to the correspondence in the dataset. If the action is the same as the label value, the reward is 1; otherwise, it is 0.

[0015] Step S4-4: Store the state, action, reward, next state, and whether it has ended as a record in the experience replay cache;

[0016] Step S4-5: When the number of cached records is greater than or equal to the batch size, randomly sample batch records; otherwise, return to step S4-1.

[0017] Steps S4-6: Calculate the Q value of the current state using the deep learning policy network and the current state data in the selected batch records; calculate the Q value of the next state data using the deep learning target network and the next state data in the selected batch records.

[0018] Step S4-7: Calculate the expected Q-value of the reinforcement learning network based on the Q-value of the next state data and the reward results in the batch records. The formula for calculating the expected Q-value is: In the formula, For the expected Q value, The Q-value is the predicted next state of the target network. For the next state, For all possible actions in the next state, The immediate reward obtained in the current step. Discount factor;

[0019] Step S4-8: Calculate the loss value using the expected Q-value, the Q-value in the current state, and the loss function. The formula for the loss function is: In the formula, The predicted value output by the model. For the true label or target value, This is the error term, and its value is... ;

[0020] Step S4-9: Based on the loss value calculation results and the optimizer, update the parameters of the deep learning policy network through backpropagation, and assign the parameters in the deep learning policy network model to the deep learning target network at certain intervals;

[0021] Step S4-10: Return to step S4-1 and continue model training until all training data for all training cycles has been trained, then stop and output the deep learning policy network model.

[0022] Step S5: The signal to be identified is processed by steps S1 to S2 and then input into the trained deep learning policy network. The final signal type is output through a voting algorithm.

[0023] Optionally, in step S1, the formula for zero-mean normalization is:

[0024]

[0025]

[0026]

[0027] In the formula, For the normalized first 1000 sample values, with zero mean and unit standard deviation. For the original first Each sample value The average of all data. Let be the standard deviation of all data. This represents the number of samples.

[0028] Optionally, in step S1, the slice length W = 8192, and the generated sample size is 1×8192.

[0029] Optionally, in step S2, the I-path data and Q-path data are split as follows: the 8192-dimensional sample is split into two 4096-dimensional vectors.

[0030] Optionally, in step S3, the deep learning policy network and the deep learning target network adopt a convolutional neural network structure based on a self-attention mechanism, which includes a feature extraction module and an attention fusion module.

[0031] The feature extraction module first uses a convolution kernel to extract features from the input feature matrix, then superimposes the extracted results with the input data and performs standardization, and finally performs pooling.

[0032] The attention fusion module uses a multi-head attention module to extract features from the input feature matrix. The extracted features are superimposed on the input features, and dropout and regularization are used for generalization. The results are then processed twice with full connection and dropout, and then superimposed again for a second generalization using regularization.

[0033] Optionally, the convolutional neural network structure based on the self-attention mechanism is a dual-input single-output structure. The input data are IQ signal data of size 1*4096. The network performs feature extraction operations on the IQ signal data using 128 and 64 convolutional kernel modules respectively, resulting in two 64*1024 feature matrices. The feature matrices are then processed by the attention mechanism module and 1*3 convolution, and then superimposed to obtain a 256*1024 fusion matrix. The fusion matrix is ​​then subjected to high-order feature compression using 512, 512, 256, and 128 convolutional kernel modules respectively, resulting in a 128*64 compressed matrix. Global adaptive pooling is used to obtain a 1*128 feature vector. Finally, 512 and 8-dimensional fully connected layers are used for feature compression respectively.

[0034] Optionally, in step S4, the key training parameters are set as follows:

[0035] Training batch size: 128, training epochs: 50, learning rate: 0.0001, greedy probability decay rate: 20000, discount factor: [missing information]. The experience replay cache capacity is 10,000, and the target network parameter update interval is once every 10 policy network updates.

[0036] Optionally, in step S5, the voting algorithm is specifically as follows:

[0037] A majority vote is conducted on the prediction results of multiple slice samples of the same signal.

[0038] According to a second aspect of the present invention, a computer device is also provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, is capable of implementing the steps of the signal type recognition method based on reinforcement learning as described in any of the technical solutions of the first aspect of the present invention.

[0039] According to a third aspect of the present invention, a computer-readable storage medium is also provided, on which a computer program is stored, which, when executed by a processor, is capable of implementing the steps of the signal type recognition method based on reinforcement learning as described in any of the technical solutions of the first aspect of the present invention.

[0040] Beneficial effects:

[0041] 1. First, the method of this invention possesses high recognition capability and robustness. This invention introduces a reinforcement learning framework into signal type recognition. By constructing a state, action, and reward mechanism, the model not only learns the direct mapping relationship between features and labels but also actively explores effective recognition strategies during trial and error. Compared to traditional supervised learning methods that rely on training with all labels, this method utilizes reward signals for dynamic feedback, ensuring the agent receives reinforcement signals after each recognition attempt, thereby continuously optimizing its action selection strategy. The introduction of the ε-greedy strategy balances exploration and utilization, enabling the model to maintain broad search capabilities in the early stages and focus on accurate discrimination in the later stages, effectively improving the stability and convergence speed of recognition. Through this mechanism, the model has a stronger ability to handle uncertainty and boundary samples, thereby improving the overall recognition accuracy and robustness, and is particularly suitable for signal recognition tasks in real-world communication scenarios such as high noise and uneven distribution.

[0042] Second, this invention possesses a stable and efficient reinforcement learning training mechanism. The method employs an experience replay mechanism and a dual-network structure (policy network and target network) for training, effectively alleviating the training instability and convergence difficulties inherent in traditional Q-learning. Through the experience replay module, the system can extract samples from historical interactions, breaking the temporal correlation of state-action transitions, thereby improving sample utilization and data distribution diversity. The use of the target network to calculate the Q-value of the next state introduces a delayed update mechanism, effectively avoiding oscillations and divergence problems caused by frequent updates of the policy network. This design continuously adjusts the direction of Q-value estimation during training, enhancing the smoothness of gradient updates and ensuring the stability of the network training process. Furthermore, batch experience sampling and backpropagation jointly optimize the policy network parameters, enabling the model to possess good generalization ability and adapt to signal recognition tasks with different channel conditions, signal types, and noise levels.

[0043] 2. Other beneficial effects or advantages of the present invention will be described in detail in the specific embodiments. Attached Figure Description

[0044] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0045] in:

[0046] Figure 1 This is a schematic diagram of the overall process of a signal type recognition method based on reinforcement learning provided in an exemplary embodiment of the present invention;

[0047] Figure 2 This is a schematic diagram of a convolutional neural network based on a self-attention mechanism provided in an exemplary embodiment of the present invention;

[0048] Figure 3 This is a schematic diagram of the training process of a signal type recognition method based on reinforcement learning provided in an exemplary embodiment of the present invention;

[0049] Figure 4 This is a schematic diagram of the Q-value update process based on the experience replay mechanism in a deep learning network provided by an exemplary embodiment of the present invention. Detailed Implementation

[0050] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are some embodiments of the present invention, but not all embodiments.

[0051] Therefore, the following detailed description of the embodiments of the invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the invention without inventive effort are within the scope of protection of the invention.

[0052] Furthermore, the terms "comprising" and "having," and any variations thereof, used in the description of this invention are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or units is not limited to the listed steps or units, but may optionally include other steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or devices. It should also be noted that in embodiments of this invention, the words "exemplary" or "for example" are used to indicate examples, illustrations, or descriptions. Any embodiment or design described as "exemplary" or "for example" in embodiments of this invention should not be construed as preferred or advantageous over other embodiments or designs. Specifically, the use of the words "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.

[0053] The technical solution of the present invention will be described in detail below with reference to the accompanying drawings.

[0054] like Figure 1 As shown, according to a first aspect of the present invention, a signal type recognition method based on reinforcement learning is provided, comprising the following steps:

[0055] Step S1: Perform zero-mean normalization and slicing on the IQ data of the training signal to generate IQ sample data;

[0056] Step S2: Split the IQ sample data into I-channel data and Q-channel data, and construct a dataset with one-hot labels;

[0057] Step S3: Construct a deep learning policy network and a deep learning target network with identical structures;

[0058] Step S4: Iteratively execute the following training process:

[0059] Step S4-1: Extract a sample data from the dataset in sequence as the current state;

[0060] Step S4-2: According to The greedy algorithm selects actions, and the actions are output based on probabilities from a deep learning policy network model or random actions;

[0061] Step S4-3: Execute the action, obtain the reward and next state according to the correspondence in the dataset. If the action is the same as the label value, the reward is 1; otherwise, it is 0.

[0062] Step S4-4: Store the state, action, reward, next state, and whether it has ended as a record in the experience replay cache;

[0063] Step S4-5: When the number of cached records is greater than or equal to the batch size, randomly sample batch records; otherwise, return to step S4-1.

[0064] Steps S4-6: Calculate the Q value of the current state using the deep learning policy network and the current state data in the selected batch records; calculate the Q value of the next state data using the deep learning target network and the next state data in the selected batch records.

[0065] Step S4-7: Calculate the expected Q-value of the reinforcement learning network based on the Q-value of the next state data and the reward results in the batch records. The formula for calculating the expected Q-value is: In the formula, For the expected Q value, The Q-value is the predicted next state of the target network. For the next state, For all possible actions in the next state, The immediate reward obtained in the current step. Discount factor;

[0066] Step S4-8: Calculate the loss value using the expected Q-value, the Q-value in the current state, and the loss function. The formula for the loss function is: In the formula, The predicted value output by the model. For the true label or target value, This is the error term, and its value is... ;

[0067] Step S4-9: Based on the loss value calculation results and the optimizer, update the parameters of the deep learning policy network through backpropagation, and assign the parameters in the deep learning policy network model to the deep learning target network at certain intervals;

[0068] Step S4-10: Return to step S4-1 and continue model training until all training data for all training cycles has been trained, then stop and output the deep learning policy network model.

[0069] Step S5: The signal to be identified is processed by steps S1 to S2 and then input into the trained deep learning policy network. The final signal type is output through a voting algorithm.

[0070] The effects that can be achieved by the present invention through the above technical solution include:

[0071] First, the method of this invention possesses high recognition capability and robustness. This invention introduces a reinforcement learning framework into signal type recognition. By constructing a state, action, and reward mechanism, the model not only learns the direct mapping relationship between features and labels but also actively explores effective recognition strategies during trial and error. Compared to traditional supervised learning methods that rely on training with all labels, this method utilizes reward signals for dynamic feedback, ensuring the agent receives reinforcement signals after each recognition attempt, thereby continuously optimizing its action selection strategy. The introduction of the ε-greedy strategy balances exploration and utilization, enabling the model to maintain broad search capabilities in the early stages and focus on accurate discrimination in the later stages, effectively improving the stability and convergence speed of recognition. Through this mechanism, the model is more capable of handling uncertainty and boundary samples, thereby improving overall recognition accuracy and robustness, making it particularly suitable for signal recognition tasks in real-world communication scenarios such as high noise and uneven distribution.

[0072] Second, this invention possesses a stable and efficient reinforcement learning training mechanism. The method employs an experience replay mechanism and a dual-network structure (policy network and target network) for training, effectively alleviating the training instability and convergence difficulties inherent in traditional Q-learning. Through the experience replay module, the system can extract samples from historical interactions, breaking the temporal correlation of state-action transitions, thereby improving sample utilization and data distribution diversity. The use of the target network to calculate the Q-value of the next state introduces a delayed update mechanism, effectively avoiding oscillations and divergence problems caused by frequent updates of the policy network. This design continuously adjusts the direction of Q-value estimation during training, enhancing the smoothness of gradient updates and ensuring the stability of the network training process. Furthermore, batch experience sampling and backpropagation jointly optimize the policy network parameters, enabling the model to possess good generalization ability and adapt to signal recognition tasks with different channel conditions, signal types, and noise levels.

[0073] The method of the present invention will be described below with reference to an exemplary embodiment.

[0074] In one exemplary embodiment, the method of the present invention can be implemented as follows, wherein the signal types involved in training or prediction are MS110A, MS110B, Link11A, Link11B, 2G-ALE, 3G-ALE, KG84, and ANDVT, respectively. The complete flowchart of this method can be found in [reference needed]. Figure 1 The method specifically includes:

[0075] Step 1: Perform zero-mean normalization and slicing of all IQ signal data used in training into 8192-bit segments to obtain IQ sample data. The zero-mean normalization formula is as follows: , , In the formula, For the normalized first 1000 sample values, with zero mean and unit standard deviation. For the original first Each sample value The average of all data. Let be the standard deviation of all data. This represents the number of samples.

[0076] Step 2: Split the IQ sample data into I-channel data and Q-channel data with a length of 4096. Construct a dataset based on the signal type labels corresponding to the sample data and shuffle it. The signal type labels are in one-hot format.

[0077] Step 3: Construct a deep learning policy network and a deep learning target network, setting the training batch size to 128, the training period to 50, the learning rate to 0.0001, the greedy probability decay rate to 20000, the discount factor to 0.99, and the maximum number of experience replay caches to 10000.

[0078] Step 4: Take a sample data point from the dataset in sequence as the current state;

[0079] Step 5, according to - A greedy algorithm selects actions, which are output based on probabilities from a deep learning policy network model or random actions.

[0080] Step 6: Execute the action, obtain the reward and next state based on the correspondence in the dataset. If the action and the label value are the same, the reward is 1; otherwise, the reward is 0.

[0081] Step 7: Store the status, action, reward, next status, and whether it has ended as a record in the experience replay cache;

[0082] Step 8: If the number of records in the experience replay cache is greater than 128, randomly select 128 records from the cache; otherwise, return to step 4.

[0083] Step 9: Calculate the Q value in the current state using the deep learning policy network model and the current state data from the selected 128 records; calculate the Q value in the next state using the deep learning target network model and the next state data from the selected 128 records.

[0084] Step 10: Calculate the expected Q-value of the reinforcement learning network using the Q-value in the next state and the reward results in the corresponding batch records. The calculation formula is as follows:

[0085]

[0086] in, For the expected Q value, Let Q be the Q-value predicted by the target network for the next state. γ represents the immediate reward obtained in the current step, and γ is the discount factor.

[0087] Step 11: Calculate the loss value using the expected Q-value, the Q-value in the current state, and the loss function. The formula for the loss function is: ,in, The predicted value output by the model. For the true label or target value, This is the error term, and its value is... ;

[0088] Step 12: Based on the loss value calculation results and the optimizer, update the parameters in the deep learning policy network using the backpropagation method, and assign the parameters in the deep learning policy network model to the deep learning target network at certain intervals;

[0089] Step 13: Return to Step 4 to continue model training. When all training data for all training cycles has been used, stop training and output the deep learning policy network model.

[0090] Step 14: Perform zero-mean normalization and slice operation of length W on the IQ data of the signal involved in the prediction to obtain IQ sample data;

[0091] Step 15: Split the IQ sample data into I-path data and Q-path data and feed them into the trained deep learning policy network model for inference to obtain the prediction result for each sample. Based on the inference results of all samples, a voting algorithm is used to finally obtain the reinforcement learning signal type identification result of the predicted signal.

[0092] The self-attention-based convolutional neural network designed in this invention can be found in [reference needed]. Figure 2 For details on the feature extraction and attention modules used in the network structure, please refer to [link / reference]. Figure 2 The attention module in the code first extracts features from the input feature matrix using convolutional kernels. The extracted features are then superimposed on the input data and normalized, followed by pooling. The attention module extracts features from the input feature matrix using a multi-head attention module. The extracted features are superimposed on the input features and generalized using dropout and regularization. The results are then processed twice with fully connected layers and dropout before being superimposed again and generalized using regularization.

[0093] The network has a dual-input, single-output structure. The input data are IQ signal data of size 1*4096. The network performs feature extraction operations on the IQ signal data using 128 and 64 convolutional kernel modules respectively, resulting in two 64*1024 feature matrices. The feature matrices are then processed by the attention mechanism module and 1*3 convolutional processing, and then superimposed to obtain a 256*1024 fusion matrix. The fusion matrix is ​​then subjected to high-order feature compression using 512, 512, 256, and 128 convolutional kernel modules respectively, resulting in a 128*64 compressed matrix. Global adaptive pooling is used to obtain a 1*128 feature vector. Finally, 512 and 8-dimensional fully connected layers are used for feature compression respectively.

[0094] It should be noted that the flowchart of the training part of the signal type recognition method of reinforcement learning in this invention is as follows: Figure 3As shown, the system first prepares data and sets training parameters. It sequentially extracts a sample from the dataset as the current state. Based on an ε-greedy policy, the agent selects the optimal action from the deep learning policy network with a certain probability, or randomly selects an action to explore. After the action is executed, the corresponding reward and next state are obtained based on the label relationships in the dataset. Then, the current state, selected action, reward value, next state, and whether the round has ended are combined to form an experience transfer record, which is stored in the experience replay cache. When the number of records in the cache reaches a set threshold, a batch of data is randomly sampled for model training. The policy network calculates the Q-value of the current state for each record, and the target network evaluates the corresponding Q-value of the next state. Combining the reward information and the Q-value of the next state, the expected Q-value of each record is calculated as the learning target. The error between the expected Q-value and the current Q-value is input into the loss function to obtain the loss value, and the parameters of the policy network are optimized using the backpropagation algorithm. During training, the system synchronously updates the parameters of the policy network to the target network at preset intervals to enhance training stability. This process iterates continuously until all training cycles are completed. The final output is the trained deep learning policy network model, which can be used for subsequent inference or deployment.

[0095] The schematic diagram of the Q-value update process based on the experience replay mechanism in the deep learning network of this invention is shown below. Figure 4 As shown, the system first samples batch data containing the current state and the next state from the experience buffer. The current state data is input into the policy network model to calculate the Q-value corresponding to the current state; the next state data is input into the target network model to obtain the maximum Q-value of the next state, and the expected Q-value is calculated by combining the reward information. Finally, the policy network is updated by calculating the loss between the current state Q-value and the expected Q-value, thereby improving the model's action decision-making ability.

[0096] According to a second aspect of the present invention, a computer device is also provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it is able to implement the steps of the signal type recognition method based on reinforcement learning in any of the technical solutions of the first aspect of the present invention.

[0097] It is understood that in this embodiment, the memory may include volatile memory, such as random access memory; the memory may also include non-volatile memory, such as read-only memory, flash memory, hard disk, or solid-state drive; furthermore, the memory may include combinations of the above types of memory. The present invention does not specifically limit this.

[0098] Similarly, the processor can implement or execute the various exemplary logical steps described in conjunction with the disclosure of this invention. The processor can be a central processing unit, a general-purpose processor, a digital signal processor, an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It can implement or execute the various exemplary logical steps described in conjunction with the disclosure of this invention. The processor can also be a combination that implements computational functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, etc.

[0099] According to a third aspect of the present invention, a computer-readable storage medium is also provided, on which a computer program is stored, characterized in that, when executed by a processor, the computer program is capable of implementing the steps of the signal type recognition method based on reinforcement learning in any of the technical solutions of the first aspect of the present invention.

[0100] In this embodiment, the computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of computer-readable storage media (a non-exhaustive list) include: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), a register, a hard disk, an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof, or any other form of computer-readable storage medium known in the art. An exemplary storage medium is coupled to a processor, enabling the processor to read information from and write information to the storage medium. Of course, the storage medium may also be a component of the processor. The processor and the storage medium may reside in an application-specific integrated circuit (ASIC). In embodiments of the present invention, a computer-readable storage medium may be any tangible medium that contains or stores a program that may be used by or in conjunction with an instruction execution system, apparatus, or device.

[0101] The above are merely specific embodiments of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A signal type recognition method based on reinforcement learning, characterized in that, Includes the following steps: Step S1: Perform zero-mean normalization and slicing on the IQ data of the training signal to generate IQ sample data; Step S2: Split the IQ sample data into I-channel data and Q-channel data, and construct a dataset with one-hot labels; Step S3: Construct a deep learning policy network and a deep learning target network with identical structures; Step S4: Iteratively execute the following training process: Step S4-1: Extract a sample data from the dataset in sequence as the current state; Step S4-2: According to The greedy algorithm selects actions, and the actions are output based on probabilities from a deep learning policy network model or random actions; Step S4-3: Execute the action, obtain the reward and next state according to the correspondence in the dataset. If the action is the same as the label value, the reward is 1; otherwise, it is 0. Step S4-4: Store the state, action, reward, next state, and whether it has ended as a record in the experience replay cache; Step S4-5: When the number of cached records is greater than or equal to the batch size, randomly sample batch records; otherwise, return to step S4-1. Steps S4-6: Calculate the Q value of the current state using the deep learning policy network and the current state data in the selected batch records; calculate the Q value of the next state data using the deep learning target network and the next state data in the selected batch records. Step S4-7: Calculate the expected Q-value of the reinforcement learning network based on the Q-value of the next state data and the reward results in the batch records. The formula for calculating the expected Q-value is: In the formula, For the expected Q value, The Q-value is the predicted next state of the target network. For the next state, For all possible actions in the next state, The immediate reward obtained in the current step. Discount factor; Step S4-8: Calculate the loss value using the expected Q-value, the Q-value in the current state, and the loss function. The formula for the loss function is: In the formula, The predicted value output by the model. For the true label or target value, This is the error term, and its value is... ; Step S4-9: Based on the loss value calculation results and the optimizer, update the parameters of the deep learning policy network through backpropagation, and assign the parameters in the deep learning policy network model to the deep learning target network at certain intervals; Step S4-10: Return to step S4-1 and continue model training until all training data for all training cycles has been trained, then stop and output the deep learning policy network model. Step S5: The signal to be identified is processed by steps S1 to S2 and then input into the trained deep learning policy network. The final signal type is output through a voting algorithm. The signal types include MS110A, MS110B, Link11A, Link11B, 2G-ALE, 3G-ALE, KG84, and ANDVT.

2. The signal type recognition method based on reinforcement learning according to claim 1, characterized in that, In step S1, the formula for zero-mean normalization is: , , In the formula, For the normalized first 1000 sample values, with zero mean and unit standard deviation. For the original first Each sample value The average of all data. Let be the standard deviation of all data. This represents the number of samples.

3. The signal type recognition method based on reinforcement learning according to claim 2, characterized in that, In step S1, the slice length W = 8192, and the generated sample size is 1×8192.

4. The signal type recognition method based on reinforcement learning according to claim 3, characterized in that, In step S2, the I-path data and Q-path data are split as follows: the 8192-dimensional sample is split into two 4096-dimensional vectors.

5. The signal type recognition method based on reinforcement learning according to claim 1, characterized in that, In step S3, the deep learning policy network and the deep learning target network adopt a convolutional neural network structure based on self-attention mechanism, which includes a feature extraction module and an attention fusion module. The feature extraction module first uses a convolution kernel to extract features from the input feature matrix, then superimposes the extracted results with the input data and performs standardization, and finally performs pooling. The attention fusion module uses a multi-head attention module to extract features from the input feature matrix. The extracted features are superimposed on the input features, and dropout and regularization are used for generalization. The results are then processed twice with full connection and dropout, and then superimposed again for a second generalization using regularization.

6. The signal type recognition method based on reinforcement learning according to claim 5, characterized in that, The self-attention-based convolutional neural network structure is a dual-input, single-output structure. The input data are IQ signal data of size 1*4096. The network performs feature extraction operations on the IQ signal data using 128 and 64 convolutional kernel modules respectively, resulting in two 64*1024 feature matrices. The feature matrices are then processed by the attention mechanism module and 1*3 convolution, and then superimposed to obtain a 256*1024 fusion matrix. The fusion matrix is ​​then subjected to high-order feature compression using 512, 512, 256, and 128 convolutional kernel modules respectively, resulting in a 128*64 compressed matrix. Global adaptive pooling is used to obtain a 1*128 feature vector. Finally, 512 and 8-dimensional fully connected layers are used for feature compression respectively.

7. The signal type recognition method based on reinforcement learning according to claim 1, characterized in that, In step S4, the key training parameters are set as follows: Training batch size: 128, training epochs: 50, learning rate: 0.0001, greedy probability decay rate: 20000, discount factor: [missing information]. The experience replay cache capacity is 10,000, and the target network parameter update interval is once every 10 policy network updates.

8. The signal type recognition method based on reinforcement learning according to claim 1, characterized in that, In step S5, the voting algorithm is as follows: A majority vote is conducted on the prediction results of multiple slice samples of the same signal.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it can implement the steps of the signal type recognition method based on reinforcement learning as described in any one of claims 1-8.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it can implement the steps of the signal type recognition method based on reinforcement learning as described in any one of claims 1-8.

Images