A drone identification method based on deep learning radio frequency fingerprint

Abstract

A method for identifying a drone based on deep learning radio frequency fingerprints belongs to the field of signal recognition and drone identification. The RF front end collects the radio frequency signal of the drone during flight, performs signal preprocessing through a low noise amplifier, a bandpass filter and a pre-amplifier, and then obtains an IQ signal through digital down-conversion. The amplitude of the IQ signal is calculated, and the amplitude sequence is further divided and spliced ​​to obtain the final input data of the deep learning network. If it is a model training process, data annotation is required, and a unique hot encoding is generated according to the drone category. The deep learning network model is composed of ViT blocks of different patch sizes, and each ViT block contains a multi-head attention mechanism and a multi-layer perceptron. By weighted summing the outputs of all ViT blocks, a vector consistent with the label dimension for predicting the drone category can be obtained. The present invention has the characteristics of fast calculation speed, high accuracy of drone identity recognition, and no information leakage.

Description

Technical Field

[0001] The invention belongs to the field of signal recognition and unmanned aerial vehicle identity recognition, and relates to a method for unmanned aerial vehicle identity recognition. Background Art

[0002] The machine learning technologies related to drone identification are divided into: drone identification systems based on radio frequency signals, drone identification systems based on visible light, drone identification systems based on voiceprints, drone detection systems based on radar, and hybrid drone identification systems. For example, the Chinese patent application number is 2017105254440, which discloses a drone identification method and management platform system, wherein the method includes: obtaining QR code information indicating the identity of the drone; decrypting the QR code information to obtain the drone's identity code; parsing the identity code to obtain the drone's identity information; matching the identity information with the identity information corresponding to each drone pre-stored in the filing library; if the match is successful, it is determined that the drone identity authentication is successful; if the match fails, it is determined that the drone identity authentication fails. The existing technical defects are: 1. Generating a QR code to indicate the identity of a drone carries a great risk of impersonation, and it is easy to copy and print the QR code of a registered drone on the surface of an unregistered drone; 2. The digital signature notarization of the QR code is not safe; 3. The QR code generation process is relatively cumbersome; 4. There is a risk that different drones will send the same authenticated QR code information; 5. The QR code information can be forged directly by copying an image; 6. The content of wireless communication is fixed, and there is a high risk of being attacked and cracked.

[0003] The advantage of the RF signal-based detection method is that it can operate during the day or night and in any weather conditions. Therefore, compared with other existing technologies, RF-based monitoring technology shows higher potential in UAV communication systems.

[0004] The difficulty of drone identification based on RF fingerprints is that environmental electromagnetic noise will interfere with the detection system. In order to reduce the impact of electromagnetic background noise on the clarity of RF signals, it is necessary to explore advanced signal processing methods to enhance the ability to extract discriminative features from RF signals, such as compressed sensing and adaptive filtering; develop machine learning models with adaptive learning functions that can continuously improve and adapt to drone signal changes, environmental changes, and new drone identification models; explore the integration of edge computing technology with drone identification systems based on RF fingerprints to achieve faster processing speeds, thereby enabling real-time decision-making in dynamic and resource-constrained environments; and study new methods to enhance drone detection and identification systems based on RF fingerprints to resist adversarial attacks, including intrusion detection mechanisms and encryption protocols.

[0005] Deep learning architectures have reached the state-of-the-art in many applications, such as speech recognition, computer vision, video analysis, natural language processing, and anomaly detection. Deep learning can recognize patterns using paradigms that are completely imperceptible to the human eye. These include transmissions that are undetectable to the human sensory system, such as radio frequency, optical, and audio. Deep learning methods can be divided into the following four types: supervised learning, semi-supervised learning, unsupervised learning, and reinforcement learning. Most deep learning-based drone detection techniques utilize supervised learning. Commonly used supervised learning methods for drone identification are CNN and RNN. Among them, CNN extracts image features through local convolution operations and is suitable for most visual tasks and time series pattern recognition tasks, especially when computing resources are limited. ViT is a deep learning architecture consistent with the application scenario of CNN. It captures global information through the self-attention mechanism and can handle more complex visual tasks than CNN, especially when there are large-scale data sets and sufficient computing resources. Specifically, 1D ViT applies the Transformer architecture to 1D data processing, which can capture global dependencies and is suitable for tasks such as time series analysis and text processing, especially when the data volume is large and complex dependencies need to be modeled. ViT has not yet been used for drone identification based on radio frequency fingerprints. Summary of the invention

[0006] In order to overcome the shortcomings of existing drone identification methods, the present invention provides a drone identification method based on deep learning radio frequency fingerprints, which has fast calculation speed, high drone identification accuracy and no information leakage.

[0007] To achieve the above technical objectives, the present invention adopts a drone identification method based on deep learning radio frequency fingerprint, which includes the following steps:

[0008] Step 1: The drone starts running and transmits radio frequency signals;

[0009] Step 2: The antenna receives the RF signal in the air and transmits it to the RF front end for preprocessing to obtain the preprocessed RF signal;

[0010] Step 3: digital down-conversion processing, including mixing, low-pass filtering, extraction, and generating IQ signals;

[0011] Step 4: Calculate the amplitude data of the IQ signal, which is the raw data that is then substituted into the network for training or inference;

[0012] Step 5: Divide the amplitude data according to a set fixed length. If radio frequency signals of multiple frequency bands are collected, that is, there are multiple radio frequency signals at the same time, multiple one-dimensional amplitude data are spliced ​​together to form a sample, that is, the spliced ​​amplitude data;

[0013] Step 6: Label the data according to the drone’s identity category, use one-hot encoding to label, and then divide the data set according to the ratio of the training set to the test set;

[0014] Step 7: Substitute the sample into the multi-scale ViT network. The entire ViT network is divided into three ViT blocks with different patch shapes and a weighted summation layer.

[0015] The samples are put into one of the ViT blocks in batches of 64. The features of the small blocks are first extracted through the convolution layer, and the features are embedded into the high-dimensional space and a classification tag is added. The position encoding is added without changing the dimension. Finally, regularization is performed through a random dropout layer.

[0016] Then it enters the Transformer encoder layer. There are 12 identical Transformer encoder layers in each ViT block. Each layer contains a self-attention mechanism and a multi-layer perceptron MLP, and the training effect is improved through normalization layers and residual connections;

[0017] The outputs of the six self-attention heads are concatenated together and residually connected with the input of the normalization layer before the multi-head attention layer. The residual is retained through the stochastic depth layer. The output of the multi-layer perceptron is residually connected with the input before the normalization layer before this, and then passes through a fully connected layer. The dimension of the output vector is consistent with the number of drone identity categories.

[0018] The outputs of the three ViT blocks are weighted summed up, and the output vector can be used to infer the drone identity category.

[0019] Furthermore, in step 2, the RF front end includes an antenna, a low noise amplifier, a band pass filter and a pre-amplifier. After the radio signal is collected by the antenna, it is processed by the low noise amplifier, the band pass filter and the pre-amplifier in sequence;

[0020] The low noise amplifier amplifies the received weak RF signal while minimizing the introduced noise, providing sufficient gain without significantly increasing the system noise, thereby improving the signal-to-noise ratio (SNR);

[0021] Bandpass filters filter out unwanted frequency components, allowing only signals in a specific frequency range to pass through, reducing interference from other frequency sources;

[0022] The preamplifier further amplifies the signal after being processed by the low noise amplifier and the band pass filter.

[0023] Furthermore, in step 3, mixing is to mix the preprocessed RF signal with the local oscillator LO signal to generate an intermediate frequency signal IF; low-pass filtering removes the high-frequency components generated after mixing and only retains the low-frequency components of the target frequency band; extraction is to reduce the sampling rate of the signal to reduce the amount of data while retaining the main information content; generating IQ signals refers to demodulating the signal through a digital signal processor DSP to extract the in-phase I and orthogonal Q components of the signal respectively.

[0024] In step 7, the self-attention mechanism has 6 heads to learn different features. In each head, the segmented sequence is mapped into query Q, key K and value V vectors using three different linear projections. The formula is,

[0025] ;

[0026] in, W Q , W K , W V is the weight matrix, X is the matrix after block embedding;

[0027] The similarity between query Q and key K is calculated by dot product to get the attention score matrix , the formula is,

[0028] ;

[0029] in, d k is the projection dimension, Softmax Convert the values ​​of each row of the matrix in brackets into a probability distribution so that the sum of the values ​​of each row is 1. The Softmax formula is:

[0030] ;

[0031] in,[ z 1 ,z 2 ,…,z n ] are all the values ​​of a row of the matrix, z i Represents the i-th value in a row, [ y 1 , y 2 ,…,y n ] is the probability distribution after Softmax transformation, y i represents the i-th probability value;

[0032] The attention score matrix With value vector V Multiply them together to get the self-attention output at each position , the formula is;

[0033] .

[0034] Furthermore, in step 7, during the multi-scale ViT network training process, the optimizer is Adam, and the loss function is the classification cross entropy loss, and the formula is,

[0035] ;

[0036] in, N is the number of categories, x i It is the component representing different drone categories in the label of the sample in the form of one-hot encoding, and the component under the correct category position x i is 1, and the rest of the positions x i is 0, is the probability distribution predicted by the model, that is, the probability of each category obtained by the Softmax function, log is the logarithmic function, and the measurement indicators are AUC and classification accuracy.

[0037] In step 7, the output dimension of the first layer of the multilayer perceptron is 1024, the output dimension of the second layer is consistent with the last dimension of the output matrix, the activation function type is GELU, and the discard rate is 0.

[0038] The drone identification method based on deep learning RF fingerprint of the present invention collects samples, processes data, and identifies a certain section of RF signal. The model of the present invention can also learn to distinguish RF signals that always exist in a changing environment when there is no drone, thereby reducing false detection of drones. RF fingerprint is a radio signal feature generated by random defects introduced during hardware industrial production. By extracting the features of the RF fingerprint of the drone during flight through a deep learning model, the drone identity classification accuracy of 99% can be achieved.

[0039] The beneficial effects of the present invention are mainly manifested in: 1. Compared with the existing radio communication to identify the identity of drones, radio frequency fingerprints are the characteristics of the physical layer, and there is no risk of identity leakage when identifying the identity of drones based on radio frequency fingerprints; and because the analysis of the hardware source of the radio frequency fingerprint is very complicated; 2. The proposed drone identity identification method does not involve Fourier transform processing of the signal, but can achieve higher classification accuracy than other machine learning methods involving Fourier transform. Therefore, the method of this case greatly shortens the computer processing time and significantly reduces the computer's memory consumption. The overall architecture is based on the Vision Transformer design, and it is the first to apply ViT to the classification of drone radio frequency signals. Compared with the existing machine learning drone identity recognition technology, the accuracy of the final classification can be greatly improved because the deep learning network based on ViT can capture the correlation information of different regions, and the multi-scale ViT network of the present invention takes into account the information at different scales and performs weighted fusion, so that the network can learn more key features during training. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 It is a flow chart of the drone identification method based on deep learning RF fingerprint.

[0041] Figure 2 It is an optional schematic diagram of the specific parameter details of the ViT network. DETAILED DESCRIPTION

[0042] The present invention will be further described below in conjunction with the accompanying drawings.

[0043] Reference Figure 1 and Figure 2 ,A method for identifying drone identity based on deep learning radio frequency fingerprint, includes the following steps:

[0044] Step 1: The drone starts running and transmits radio frequency signals;

[0045] Step 2: The antenna receives the RF signal in the air and transmits it to the RF front end for preprocessing to obtain the preprocessed RF signal;

[0046] The RF front end includes an antenna, a low noise amplifier, a band pass filter and a pre-amplifier. After the radio signal is collected by the antenna, it is processed by the low noise amplifier, the band pass filter and the pre-amplifier in sequence.

[0047] The low noise amplifier amplifies the received weak RF signal while minimizing the introduced noise, providing sufficient gain without significantly increasing the system noise, thereby improving the signal-to-noise ratio (SNR);

[0048] Bandpass filters filter out unwanted frequency components, allowing only signals in a specific frequency range to pass through, reducing interference from other frequency sources;

[0049] The preamplifier further amplifies the signal after being processed by the low-noise amplifier and bandpass filter;

[0050] Step 3, digital down-conversion processing, including mixing, low-pass filtering, extraction, and generating IQ signals in sequence, wherein mixing is to mix the pre-processed RF signal with the local oscillator LO signal to generate an intermediate frequency signal IF; low-pass filtering removes the high-frequency components generated after mixing and only retains the low-frequency components of the target frequency band; extraction is to reduce the sampling rate of the signal to reduce the amount of data while retaining the main information content; generating IQ signals means demodulating the signal through a digital signal processor DSP to extract the in-phase I and orthogonal Q components of the signal respectively;

[0051] Step 4: Calculate the amplitude data of the IQ signal, which is the raw data that is then substituted into the network for training or inference;

[0052] Step 5: Divide the amplitude data into fixed lengths (e.g., 20480). Since multiple devices may collect RF signals of different frequency bands during collection, it is assumed here that RF signals of two frequency bands are collected, that is, there are two RF signals at the same time. Then, two one-dimensional amplitude data with a length of 20480 are spliced ​​together to form a matrix of 20480 rows and 2 columns, which are then spliced ​​together to form samples, i.e., the spliced ​​amplitude data.

[0053] Step 6: Label the data according to the identity category of the drone. The labeling adopts one-hot encoding. For example, if there are 10 drones and the identity category is 10, it is labeled with a vector of length 10 consisting of only 0 and 1. Except for the vector position representing the true category, the other positions of the labeled vector are 0. Then the data set is divided according to the ratio of the training set and the test set;

[0054] Step 7. Substitute the sample into the multi-scale ViT network. The entire ViT network is divided into three ViT blocks with different patch shapes and a weighted summation layer. For example, the shape of the patch can be a matrix with 512 rows and 2 columns. The row dimension of the patch matrix here does not exceed the length when the amplitude data is divided, and the column dimension is consistent with the number of collected frequency bands. The following steps take the sample matrix with 20480 rows and 2 columns and the patch matrix with 512 rows and 2 columns as examples;

[0055] like Figure 2As shown in the figure, samples are put into one of the ViT blocks in batches of 64. First, the features of the patches are extracted through the convolutional layer and embedded into a high-dimensional space, such as 768 dimensions. After embedding, it becomes a matrix with 40 rows and 768 columns. After adding a classification token (CLS token), it becomes a matrix with 41 rows and 768 columns. Position encoding is added without changing the dimension. Finally, dropout is used for regularization to prevent overfitting.

[0056] Then enter the Transformer encoder layer, such as Figure 2 As shown in Figure 1, each ViT block has 12 identical Transformer encoder layers, each of which contains two main parts: a multi-head attention mechanism and a multi-layer perceptron. The normalization layer and residual connection are used to improve the training effect. The self-attention mechanism is shown in Figure 1. Figure 2 As shown, there are 6 heads learning different features. In each head, the segmented sequence is mapped into query Q, key K and value V vectors using three different linear projections (the projection dimension here is 768), and the formula is,

[0057] ;

[0058] in, W Q , W K , W V is the weight matrix, X is the matrix after block embedding;

[0059] The similarity between query Q and key K is calculated by dot product to get the attention score matrix , the formula is,

[0060] ;

[0061] in, d k is the projection dimension, Softmax Convert the values ​​of each row of the matrix in brackets into a probability distribution so that the sum of the values ​​of each row is 1. The Softmax formula is:

[0062] ;

[0063] in,[ z 1 ,z 2 ,…,z n ] are all the values ​​of a row of the matrix, z i Represents the i-th value in a row, [y 1 , y 2 ,…,y n ] is the probability distribution after Softmax transformation, y i represents the i-th probability value;

[0064] The attention score matrix With value vector V Multiply them together to get the self-attention output at each position , the formula is;

[0065] ;

[0066] Output a matrix of 41 rows and 768 columns;

[0067] The outputs of the six self-attention heads are concatenated together and residually connected with the input after patch segmentation and embedding and before being substituted into the multi-head attention layer, and the residual is retained by StochasticDepth; then pass through the normalization layer and the multi-layer perceptron, the first layer output dimension of the multi-layer perceptron is 1024, the second layer output dimension is consistent with the last dimension of the output matrix, the activation function type is GELU, and the dropout rate is 0; the output of the multi-layer perceptron is residually connected with the input before the normalization layer before this, and then pass through a fully connected layer, and the output vector dimension is consistent with the number of drone identity categories;

[0068] The outputs of the three ViT blocks are weighted and summed. The weights are learnable and the shape of the output vector remains unchanged. The drone identity category can be inferred from this vector.

[0069] Furthermore, in the training process of the multi-scale ViT network, the optimizer is Adam and the loss function is the categorical cross entropy loss, and the formula is;

[0070] ;

[0071] in, N is the number of categories, x i It is the component representing different drone categories in the label of the sample in the form of one-hot encoding, and the component under the correct category position x i is 1, and the rest of the positions x i is 0, is the probability distribution predicted by the model, that is, the probability of each category obtained by the Softmax function, log is the logarithmic function, and the measurement indicators are AUC and classification accuracy.

[0072] The drone identification method based on deep learning radio frequency fingerprint of this embodiment includes the following steps:

[0073] Step 1) The drone is turned on and starts transmitting radio frequency signals;

[0074] Step 2), the RF front end may include an antenna, a low noise amplifier, a bandpass filter, and a pre-amplifier, which receives and pre-processes the RF signal to obtain a pre-processed signal;

[0075] Step 3), digital down-conversion processing, which may include mixing, low-pass filtering, extraction, and generation of IQ signals, wherein the IQ signal is indispensable;

[0076] Step 4), the amplitude data of the IQ signal can be calculated, or other features can be selected as the original data. The following steps still take the amplitude as an example, and substitute the amplitude data into the deep learning network for training or reasoning;

[0077] Step 5), divide the amplitude data into a fixed length, for example, the length can be 20480. If multiple RF acquisition devices are used to collect data of multiple frequency bands, after the processing of step 4 is performed respectively, multiple sequences are spliced ​​to form a two-dimensional matrix, and then the following steps are performed, for example, data of two different frequency bands are collected, the amplitude is calculated, and a 20480*2 matrix is ​​obtained after division and splicing; if only data of one frequency band is collected, there is no splicing process, and after a fixed length division, the next step is directly performed, for example, a sequence with a length of 20480;

[0078] Step 6) Label the data according to the identity of the drone, use one-hot encoding for labeling, and then divide the data set according to the ratio of training set to test set, which can be 8:2;

[0079] Step 7), substitute the data after step 5 into the multi-scale ViT network. The entire ViT network can be divided into three ViT blocks with different patch shapes and a weighted summation layer;

[0080] Step 8), you can select 64 data after step 5) as a batch and substitute them into the ViT block. In any ViT block, you can consider the input as 20480*2 spliced ​​data. First, you can use a 512*2 convolution kernel to extract 40 features divided into small blocks, and embed the features into a high-dimensional space of 1024 dimensions, add a classification tag; add position encoding without changing the dimension; finally, you can use a random dropout layer for regularization;

[0081] Then it enters the Transformer encoder layer. Each ViT block can contain 12 consecutive identical Transformer encoder layers. Each layer contains a multi-head attention mechanism and a multi-layer perceptron, and the training effect is improved through normalization layers and residual connections.

[0082] Step 9), in the multi-head attention mechanism, 6 self-attention heads can be selected and their outputs can be spliced ​​together, and residual connection can be performed with the input of the normalization layer before the multi-head attention mechanism, and the residual can be retained by random depth; then through the normalization layer and the multi-layer perceptron, the first layer output dimension of the multi-layer perceptron can be 1024, the second layer output dimension can be consistent with the last dimension of the output matrix of the multi-head attention mechanism, the activation function type can be GELU, the drop rate can be 0, the output of the multi-layer perceptron is residually connected with the input before the normalization layer before this, and then through a fully connected layer, the output vector dimension is consistent with the number of drone identity categories;

[0083] Step 10), the output of all ViT blocks is weighted and summed, and the output vector can be used to infer the drone identity category. The optimizer for model training can be Adam, and the loss function can be classification cross entropy loss.

[0084] The contents described in the embodiments of this specification are merely enumerations of implementation forms of the inventive concept and are for illustrative purposes only. The protection scope of the present invention should not be considered to be limited to the specific forms described in this embodiment, and the protection scope of the present invention also extends to equivalent technical means that can be thought of by ordinary technicians in this field based on the inventive concept.

Claims

1. A drone identification method based on deep learning radio frequency fingerprint, characterized in that: The method comprises the following steps: Step 1: The drone starts running and transmits radio frequency signals; Step 2: The antenna receives the RF signal in the air and transmits it to the RF front end for preprocessing to obtain the preprocessed RF signal; Step 3: digital down-conversion processing, including mixing, low-pass filtering, extraction, and generating IQ signals; Step 4: Calculate the amplitude data of the IQ signal, which is the raw data that is then substituted into the network for training or inference; Step 5: Divide the amplitude data according to a set fixed length. If radio frequency signals of multiple frequency bands are collected, that is, there are multiple radio frequency signals at the same time, multiple one-dimensional amplitude data are spliced ​​together to form a sample, that is, the spliced ​​amplitude data; Step 6: Label the data according to the drone’s identity category, use one-hot encoding to label, and then divide the data set according to the ratio of the training set to the test set; Step 7: Substitute the sample into the multi-scale ViT network. The entire ViT network is divided into three ViT blocks with different patch shapes and a weighted summation layer. The samples are put into one of the ViT blocks in batches of 64. First, the features of the small blocks are extracted through the convolution layer, and the features are embedded into the high-dimensional space and a classification tag is added. The position encoding is added without changing the dimension. Finally, regularization is performed through the random dropout layer. Then it enters the Transformer encoder layer. There are 12 identical Transformer encoder layers in each ViT block. Each layer contains a self-attention mechanism and a multi-layer perceptron MLP, and the training effect is improved through normalization layers and residual connections; The outputs of the six self-attention heads are concatenated together and residually connected with the input of the normalization layer before the multi-head attention layer, and the residual is retained through the random depth layer; then the output of the multi-layer perceptron is residually connected with the input before the normalization layer before this, and then passes through a fully connected layer, and the output vector dimension is consistent with the number of drone identity categories; The outputs of the three ViT blocks are weighted summed up, and the output vector can be used to infer the drone identity category.

2. A method for identifying drones based on deep learning radio frequency fingerprints as claimed in claim 1, characterized in that: In step 2, the RF front end includes an antenna, a low noise amplifier, a band pass filter and a pre-amplifier. After the radio signal is collected by the antenna, it is processed in sequence by the low noise amplifier, the band pass filter and the pre-amplifier.

3. A method for identifying a drone based on deep learning radio frequency fingerprint as claimed in claim 1 or 2, characterized in that: In step 3, mixing is to mix the pre-processed RF signal with the local oscillator LO signal to generate an intermediate frequency signal IF; low-pass filtering removes the high-frequency components generated after mixing and only retains the low-frequency components of the target frequency band; extraction is to reduce the sampling rate of the signal to reduce the amount of data while retaining the main information content; generating IQ signals refers to demodulating the signal through a digital signal processor DSP to extract the in-phase I and orthogonal Q components of the signal respectively.

4. A method for identifying a drone based on deep learning radio frequency fingerprint as claimed in claim 1 or 2, characterized in that: In step 7, the self-attention mechanism has 6 heads to learn different features. In each head, the segmented sequence is mapped to query Q, key Q, and key Q using three different linear projections. K Sum V Vector, the formula is, ； in, W Q , W K , W V is the weight matrix, X is the matrix after block embedding; The similarity between query Q and key K is calculated by dot product to get the attention score matrix , the formula is, ； in, d k is the projection dimension, Softmax Convert the values ​​of each row of the matrix in brackets into a probability distribution so that the sum of the values ​​of each row is 1. The Softmax formula is: ； in,[ z 1 ,z 2 ,…,z n ] are all the values ​​of a row of the matrix, z i Represents the i-th value in a row, [ y 1 ,y 2 ,…,y n ] is the probability distribution after Softmax transformation, y i represents the i-th probability value; The attention score matrix With value vector V Multiply them together to get the self-attention output at each position , the formula is, 。 5. A method for identifying drones based on deep learning radio frequency fingerprints as claimed in claim 4, characterized in that: In the training process of the multi-scale ViT network, the optimizer is Adam and the loss function is the categorical cross entropy loss, and the formula is, ； in, N is the number of categories, x i It is the component representing different drone categories in the label of the sample in the form of one-hot encoding, and the component under the correct category position x i is 1, and the rest of the positions x i is 0, is the probability distribution predicted by the model, that is, the probability of each category obtained by the Softmax function, log is the logarithmic function, and the measurement indicators are AUC and classification accuracy.

6. A method for identifying drones based on deep learning radio frequency fingerprints as claimed in claim 1 or 2, characterized in that: In step 7, the output dimension of the first layer of the multilayer perceptron is 1024, the output dimension of the second layer is consistent with the last dimension of the output matrix, the activation function type is GELU, and the discard rate is 0.

Images