A short-burst SOQPSK carrier synchronization method

Abstract

The application discloses a short-burst SOQPSK carrier synchronization method based on an RPT-Net network, pre-processes a signal, and merges time domain data of the signal and data after quartic into input data; a multi-level residual convolution network is used to extract cross-symbol local phase features, a convolution operation is used to capture subtle phase changes in the SOQPSK signal through layer-by-layer stacking, and a residual connection mechanism is combined to relieve gradient degradation problems of a deep network and enhance local representation capabilities of phase jumps and frequency offset disturbances; a phase perception position coding module is used to convert the time domain signal into a complex form, extract an instantaneous phase, construct an orthogonal base coding table to represent phase changes, and then deeply fuse the position coding; a Transformer coding layer is introduced, a multi-head self-attention mechanism is used to capture long-distance symbol dependency relationships, global phase consistency constraints are optimized, long-term effects caused by frequency offset on the signal are captured, and time sequence error propagation is effectively inhibited.

Description

Technical Field

[0001] This invention belongs to the field of deep learning technology, specifically relating to a short burst SOQPSK carrier synchronization method. Background Technology

[0002] SOQPSK-MIL, as a specific modulation scheme, achieves a balance between high spectral efficiency and constant envelope through continuous phase characteristics and shaped pulse design. However, its short burst transmission mode and complex channel environment pose a dual challenge to receiver synchronization. Therefore, how to accurately synchronize short burst SOQPSK signals has become a major problem.

[0003] Synchronization is a crucial step that a receiving system must complete before signal demodulation, directly impacting the performance of subsequent demodulation processes. Good synchronization significantly improves demodulation accuracy and communication quality. The core task of synchronization is to estimate carrier frequency offset, carrier phase error, and symbol timing error in the received signal, and to compensate and correct the received signal based on these estimates. Through effective synchronization processing, the receiver can eliminate various deviations in the signal, providing an accurate reference for subsequent demodulation and ensuring correct information recovery. Traditional synchronization methods are mainly divided into data-assisted and non-data-assisted methods. Data-assisted algorithms offer higher accuracy but require the transmission of a synchronization header containing the training sequence, which consumes transmission time and reduces information transmission efficiency. Furthermore, due to channel time-varying characteristics, the timing error estimated for the synchronization header may not be accurately applicable to subsequent symbols. For blind reception of short burst SOQPSK signals, data assistance cannot be used. Fast synchronization requires a large loop bandwidth, introducing more phase noise and thus degrading synchronization performance, affecting the system's demodulation performance. Existing algorithms based on nonlinear fourth-order transforms are prone to phase abrupt changes under low signal-to-noise ratio conditions, leading to deteriorated synchronization performance.

[0004] In recent years, deep learning technology has provided new insights into the synchronization problem. Existing techniques incorporate the attention mechanism of wireless transformation networks into modulation and demodulation schemes, recovering received signals with random delays, frequency differences, phase differences, and delay spreads; however, their effectiveness has not been fully verified. Additionally, a channel synchronization algorithm for SOQPSK has been implemented using CSDNN networks, achieving fast convergence and effective synchronization even under strong interference. However, these existing techniques neglect the memory characteristics of SOQPSK signals and the long-term impact of frequency offset on the signal. Therefore, a more suitable network is needed to achieve carrier synchronization of SOQPSK signals. Summary of the Invention

[0005] To address the shortcomings of existing technologies and improve the carrier synchronization performance of short-burst SOQPSK signals, this invention adopts the following technical solution:

[0006] A short burst SOQPSK carrier synchronization method includes the following steps:

[0007] Step 1: Acquire burst SOQPSK signal sample data;

[0008] Step 2: Signal sample data preprocessing; downconvert and burst signal detection are performed on the signal sample data, and symbol timing synchronization is set to complete (symbol delay is 0), the influence of carrier frequency offset and phase offset is retained, the signal sample data is divided into real part and imaginary part, and fourth power processing is performed to suppress modulation influence and enhance the carrier information expression capability.

[0009] Step 3: In order to adapt short-time burst SOQPSK signals to the input requirements of deep neural networks, the preprocessed signals need to be image-constructed. The real, imaginary, and machine fourth-order real and imaginary parts are re-dimensioned to construct a feature map matrix. This allows for the full fusion of the original time-domain information and frequency offset features, enabling the network to model the changing trends of frequency and phase perturbations in the feature space and improve the synchronous recognition capability.

[0010] Step 4: Extract burst SOQPSK signal features;

[0011] Step 5: Model the global temporal characteristics of the SOQPSK signal and perform synchronization compensation. The extracted features are processed by a phase-aware position encoding module to increase the awareness of cumulative phase feature changes and frequency offset drift. Subsequently, a long-distance phase dependency relationship is established through a two-layer Transformer encoder module using a multi-head attention mechanism. This mechanism can capture the cumulative phase offset under the influence of frequency offset from a global modeling perspective. Combined with a fully connected layer, the carrier frequency offset error is further corrected to achieve accurate synchronization. Finally, the synchronization signal after deep network compensation is output. Its bit error rate approaches the theoretical limit under different signal-to-noise ratio conditions, showing better carrier synchronization performance than traditional methods and other deep networks (such as CSDNN).

[0012] Furthermore, in step 1, burst SOQPSK signal sample data under different signal-to-noise ratios are obtained through a real receiver and simulation. The SOQPSK signal received through the channel, after down-conversion, is represented in an additive white Gaussian noise environment as follows:

[0013]

[0014] Among them, E b T represents the energy of each symbol. b Let represent the symbol period, exp{·} represent the exponential function, j represent the imaginary unit, f represent the carrier residual frequency offset, t represent time, τ represent the symbol timing delay, w(t) represent additive Gaussian noise, and θ represent the fixed phase offset of the received signal. Represents the modulation phase function;

[0015] Since this invention only considers the implementation of SOQPSK carrier synchronization, it is assumed that the received signal has undergone ideal timing synchronization, i.e., τ is 0. It is expressed as follows:

[0016]

[0017] Where, α i represents the precoded transmission sequence, with values ​​ranging from {-1, 0, 1}, h represents the modulation index (SOQPSK-MIL has a modulation index of 1 / 2), q(·) represents the phase pulse shaping function, n represents the symbol index, and α represents the symbol sequence.

[0018] Further, in step 2, the real signal data acquired through front-end sampling is processed by automatic gain control (AGC) to adjust the power of the real signal data after front-end sampling to a smaller range so that the power of burst signals and noise in the channel is similar; the signal after automatic gain control processing is subjected to orthogonal downconversion and low-pass filtering to remove the second harmonic component, obtaining the baseband data of the complex signal; the baseband data of the complex signal is processed by point sliding fast Fourier transform (FFT) to extract frequency domain features, and the frequency points extracted by FFT results are sorted from smallest to largest, and the ratio of the peak value of the extracted frequency points to the mean value of the in-band noise floor frequency points is calculated to form a frequency domain ratio sequence; the frequency domain ratio sequence is analyzed using a sliding window, and each ratio in the window is compared with a set threshold. If the ratio in the sliding window that is greater than the threshold exceeds a specified number, it is determined to be the starting point of a burst signal; the signal is downconverted to a baseband signal using a frequency offset estimation method, and the baseband signal is converted to contain four sampling points per symbol by sampling rate transformation.

[0019] Furthermore, the dimension reorganization in step 3 involves reorganizing a one-dimensional sequence of length 2N into an M×N matrix, where M is the number of bits and N is the number of sampling points per bit. The matrix is ​​constructed such that each row represents one bit, different rows represent different symbol segments, and odd-numbered indices in each column represent the real part of the signal, while even-numbered indices represent the imaginary part. In this way, the mapping from a one-dimensional signal to two-dimensional image features is completed, constructing an input matrix with spatial structure, which facilitates subsequent convolutional networks in modeling the phase structure of the signal between bits and sampling points.

[0020] The specific implementation of the graphical construction in step 3 is as follows:

[0021] If the preprocessed signal is sampled N times, the discrete sampling mathematical model of each bit of the SOQPSK signal after sampling can be expressed as:

[0022]

[0023] The sampled signal r n (α) is divided into real parts and the virtual part The following matrix R is obtained:

[0024]

[0025] One row represents the real part of the signal, and the other row represents the imaginary part;

[0026] r n (α) is processed to the fourth power to obtain:

[0027]

[0028] (r) n (α)) 4 Let X[k] represent the index of the sampling point, and construct a matrix with one row of real parts and one row of imaginary parts. Then merge it with matrix R to obtain a new matrix I.

[0029]

[0030] Matrix I is used as input data and passed to the deep learning model.

[0031] Furthermore, in step 4, the residual connection convolution module is used to extract local temporal features. The feature map is first extracted by one-dimensional convolutional layer, batch normalization and activation functions (Mish, LeakyReLU) to extract pulse shaping structure and phase perturbation features. Then, the residual connection module is introduced to enhance the stability of feature flow transmission and the training effect of deep network, and avoid the gradient vanishing problem. The residual connection convolution module effectively enhances the local representation ability of the signal under frequency offset interference.

[0032] Furthermore, in the residual connection module, a convolutional neural network is used to extract features from the feature map matrix. First, local temporal features are extracted through a one-dimensional convolutional layer (64 filters, kernel size 8) to capture the pulse shaping characteristics of the signal. Then, a residual structure is introduced to enhance network performance. The residual block includes two one-dimensional convolutional layers (each using 8 filters), each followed by batch normalization and a first activation function, Mish, to improve nonlinear modeling ability and generalization. The residual path directly adds the input to the convolutional output, achieving cross-layer feature fusion and mitigating gradient vanishing. In addition, a second activation function, LeakyReLU, is used at the residual output to effectively avoid dead neurons and enhance model stability. The residual connection module can significantly extract frequency offset and phase features, laying the foundation for subsequent synchronization processing.

[0033] Furthermore, the formula expression for the first activation function shown is the Mish activation function:

[0034] f(x) = x * tanh(ln(1+e) x ))

[0035] Where x represents the input feature, f(x) represents the output feature of the activation function, and e represents the base of the natural logarithm; by using the Mish activation function, nonlinear characteristics can be introduced, enabling the network to learn more complex signal features;

[0036] The second activation function is the LeakyReLU activation function, which further alleviates the gradient vanishing problem and improves the robustness of the model. Unlike the traditional ReLU activation function, the LeakyReLU activation function retains a small gradient in the negative range, thus effectively avoiding the dead neuron phenomenon and ensuring the stability of the network during training. Through this design, the residual convolutional layer can not only efficiently extract the phase features and frequency offset information of the SOQPSK signal, but also enhance the gradient fluidity and information transfer capability of the model in deep network training.

[0037] Furthermore, step 5 includes the following steps:

[0038] First, the extracted feature map is input into the phase-aware position encoding module to calculate the instantaneous phase of the input signal, which is a complex number consisting of real and imaginary parts, and then a sine-cosine mapping is performed to extract its phase change features. At the same time, a position encoding vector based on sine and cosine functions is constructed and fused with the phase features to enhance the model's ability to perceive phase accumulation changes and frequency offset drift, thereby improving synchronization accuracy.

[0039] Subsequently, the fused features are input into a two-layer Transformer encoder module. Each layer consists of a multi-head self-attention mechanism (4 heads in total, each with 8 dimensions) and a feedforward neural network (FFN). The self-attention mechanism calculates the relationship between the query, key, and value to globally model the temporal dependency in the signal, effectively capturing the influence of frequency offset and phase transition at different time steps. Each encoder layer of the Transformer encoder module introduces residual connections and layer normalization to improve training stability and alleviate the gradient vanishing problem.

[0040] Finally, the synchronization characteristics output by the Transformer encoder module are used as a signal compensation reference to achieve accurate carrier synchronization of SOQPSK signals under complex conditions.

[0041] Furthermore, the phase-sensing position encoding module in step S5 employs a bidirectional short-duration memory module, and the implementation process is as follows:

[0042] First, during the initialization process, the input dimension d model The high-dimensional representation used to determine the position code is calculated based on periodic functions (sine and cosine functions), which have the following form:

[0043]

[0044] Where t represents the time step, i represents the dimension index of the location encoding, and d model This encoding method represents the dimension of the model, which can transform time series information into a high-dimensional vector with periodicity and increasing properties, thereby enabling the model to better capture time dependencies.

[0045] Then, to further improve synchronization accuracy, this module calculates the instantaneous phase of the signal by representing the input signal in complex form, defined as:

[0046] θ(t) = arg(x(t))

[0047] Where x(t) represents the complex form of the SOQPSK signal, and then, by performing sine and cosine mapping on the instantaneous phase θ(t), the phase coding features are obtained:

[0048] θ enc =[sin(θ(t)),cos(θ(t))]

[0049] Phase-encoded features are mapped to the model dimension d through a fully connected layer. model A consistent high-dimensional space enhances its expressive power;

[0050] Finally, the processed phase code and position code are added element-wise to form the output of the phase-aware position code:

[0051]

[0052] This fusion method can preserve the low-level features of the original signal, avoid information loss, and effectively model the phase change characteristics of the SOQPSK signal, enhancing the network's ability to model linear phase accumulation, thereby compensating for frequency offset and phase error in the signal. Furthermore, by performing fourth-power processing on the signal to weaken the modulation component and highlight the four-fold component of the carrier phase, it can more effectively model the phase change characteristics of the SOQPSK signal and improve the accuracy of carrier synchronization. Compared with traditional synchronization methods, the phase-aware position coding module proposed in this invention can significantly improve synchronization performance under low signal-to-noise ratio, dynamic frequency offset, and short burst environments.

[0053] Furthermore, the dual-layer Transformer encoder module in step S5 includes a multi-head self-attention mechanism and a feedforward neural network in its encoding layer;

[0054] Multi-head self-attention mechanisms learn different subspace features of the input signal in parallel through multiple heads. Each head captures temporal dependencies by calculating the relationship between the query, key, and value of the input sequence. Through this calculation, the model can focus on different parts of the sequence at each time step, thus effectively capturing the temporal dependencies of the signal. Especially in carrier synchronization tasks, this mechanism can help the model learn the effects of phase and frequency offsets on the signal. The calculation formula for the head attention mechanism is as follows:

[0055]

[0056] Where Attention(·) represents a multi-head attention operation, Q, K, and V represent the query, key, and value matrices, respectively, and d k The dimension representing the key;

[0057] Each coding layer includes a feedforward neural network (FFN), comprising multiple fully connected layers and non-linear mapping through activation functions. FFN helps improve the model's expressive power, enabling it to learn more complex signal features. In this implementation, the structure of the feedforward neural network is as follows:

[0058] FNN(x)=Dense(ReLU(W1x+b1))W2+b2

[0059] Where Dense(·) represents the fully connected operation, ReLU represents the activation operation, x represents the input feature, W1 and b1 represent the weight matrix and bias term of the activation operation, W2 and b2 represent the weight matrix and bias term of the fully connected layer, and FFN performs nonlinear transformation on the output of each layer, thereby enhancing the learning ability of the model, especially when processing communication signals with nonlinear characteristics, it can effectively improve signal processing capabilities;

[0060] After the attention mechanism and feedforward neural network in each layer, residual connections and layer normalization are added. Residual connections help prevent gradient vanishing, enabling the network to be trained more deeply and stably. Layer normalization helps accelerate training and improves the stability and convergence speed during the training process.

[0061] Through this design, the Transformer encoder can fully extract the features of the input signal, and at the same time, combined with the fourth power signal enhancement, it can enhance the sensitivity to phase error, capture the cumulative phase change of the signal under long distance conditions, obtain residual frequency offset information, and perform effective synchronization error correction on this basis. The introduction of the Transformer architecture makes the model perform well when processing complex signals, especially in the presence of Doppler frequency offset and phase offset, with stronger robustness and generalization ability.

[0062] The advantages and beneficial effects of this invention are as follows:

[0063] This invention addresses the carrier synchronization difficulties of SOQPSK signals in short-burst transmission scenarios due to complex channel environments (Doppler shift, noise interference, and multipath effects). It utilizes an RPT-NET network to achieve carrier synchronization of SOQPSK signals, aiming to improve carrier synchronization accuracy. In preprocessing, the original time-domain signal is concatenated with its fourth-order transform features, and a residual convolutional network is used to extract local phase details with gradient stability, enhancing the representation of burst phase changes. Subsequently, dynamic orthogonal basis coding is generated through complex-domain instantaneous phase decomposition, deeply fusing phase continuity with location information to improve adaptability to frequency offset disturbances. Finally, a multi-head attention mechanism of the Transformer is used to establish long-range cross-symbol associations, and accumulated frequency offset errors are compensated through global phase consistency constraints. Simulation results show that the bit error rate obtained by demodulating SOQPSK signals synchronized using this invention approaches the theoretical bit error rate, and the carrier synchronization performance is superior to traditional methods and CSDNN networks. Attached Figure Description

[0064] Figure 1a This is a flowchart of the method of the present invention.

[0065] Figure 1b This is a structural diagram of the RPT-Net network in an embodiment of the present invention.

[0066] Figure 2a This is a waveform diagram of the real part of the modulation signal in an embodiment of the present invention.

[0067] Figure 2b This is a waveform diagram of the imaginary part of the modulation signal in an embodiment of the present invention.

[0068] Figure 3a This is a waveform diagram of the real part of the received signal in an embodiment of the present invention.

[0069] Figure 3b This is a waveform diagram of the imaginary part of the received signal in an embodiment of the present invention.

[0070] Figure 4 This is a schematic diagram of the phase-sensing position encoding module in an embodiment of the present invention.

[0071] Figure 5a This is a comparison diagram of the real part data of the original signal and the received signal at 0dB in an embodiment of the present invention.

[0072] Figure 5b This is a comparison diagram of the imaginary part data of the original signal and the received signal at 0dB in an embodiment of the present invention.

[0073] Figure 5cThis is a comparison diagram of the real part data of the original signal and the received signal at 12dB in an embodiment of the present invention.

[0074] Figure 5d This is a comparison diagram of the imaginary part data of the original signal and the received signal at 12dB in an embodiment of the present invention.

[0075] Figure 6a This is a comparison chart of the real part data of the original signal and the synchronized signal at 0dB in an embodiment of the present invention.

[0076] Figure 6b This is a comparison diagram of the imaginary part data of the original signal and the synchronized signal at 0dB in an embodiment of the present invention.

[0077] Figure 6c This is a comparison chart of the real part data of the original signal and the synchronized signal at 12dB in an embodiment of the present invention.

[0078] Figure 6d This is a comparison diagram of the imaginary part data of the original signal and the synchronized signal at 12dB in an embodiment of the present invention.

[0079] Figure 7 This is a comparison chart of the phase mean square error of different synchronization algorithms with different signal-to-noise ratios in the embodiments of the present invention.

[0080] Figure 8 This is a comparison chart of bit error rates for different synchronization algorithms with different signal-to-noise ratios in embodiments of the present invention.

[0081] Figure 9a This is a comparison graph (mean square error, MSE) of ablation experimental results of different networks in the embodiments of the present invention.

[0082] Figure 9b This is a comparison graph (Bit Error Rate BER) of ablation experiments using different networks in this embodiment of the invention. Detailed Implementation

[0083] The specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are for illustration and explanation only and are not intended to limit the present invention.

[0084] This invention proposes a short-burst SOQPSK carrier synchronization method based on the RPT-Net network, aiming to improve carrier synchronization accuracy. First, the signal is preprocessed by merging the time-domain data and the fourth-order processed data into the input data. Then, a multi-level residual convolutional network is used to extract cross-symbol local phase features. Subtle phase changes in the SOQPSK signal are captured through layer-by-layer stacked convolution operations, and a residual connection mechanism is used to mitigate the gradient degradation problem of deep networks, enhancing the local representation capability of phase jumps and frequency offset disturbances. Second, a phase-aware position coding module is designed. By converting the time-domain signal into complex form and extracting the instantaneous phase, an orthogonal basis coding is constructed to represent phase changes, which is then deeply fused with the position coding. This module not only fully utilizes the phase continuity characteristics of the SOQPSK signal but also dynamically adapts to the received signal under different frequency and phase offset conditions. Finally, a Transformer coding layer is introduced, utilizing its multi-head self-attention mechanism to capture long-distance symbol dependencies, optimize global phase consistency constraints, capture the long-term effects of frequency offset on the signal, and effectively suppress the propagation of timing errors. Even under low signal-to-noise ratio conditions, this network can still effectively compensate for the phase of the signal through global phase modeling, ensuring the stability of synchronization performance. Simulation results show that the bit error rate of the SOQPSK signal after synchronization using this invention and subsequent demodulation is close to the theoretical bit error rate, and the carrier synchronization performance is higher than that of traditional methods and CSDNN networks.

[0085] like Figure 1a , Figure 1b As shown, the present invention specifically includes the following steps:

[0086] Step 1: Simulate and obtain signal sample data including burst signals. The signal sample data is obtained through simulation under different signal-to-noise ratios. Using Matlab simulation, short burst modulated signals are generated according to the P-band SOQPSK-MIL protocol, and Gaussian white noise is added to simulate a complex channel environment. Training and test data are set under different frequency offsets and phase offsets to form a large-scale sample set for subsequent synchronization tasks. The waveform of the signal is shown below. Figure 2a , Figure 2b , Figure 3a , Figure 3b As shown.

[0087] Furthermore, SOQPSK is developed based on OQPSK. Unlike QPSK signals, SOQPSK signals have a slope in their phase changes. The phase within each symbol is a function of time, and its modulation signal is a continuous phase modulation signal. SOQPSK signals also differ from ordinary CPM signals. The phase change of a SOQPSK signal within a symbol can only be constant or ±π / 2.

[0088] Specifically, the received signal model is implemented as follows:

[0089] Burst SOQPSK signal sample data under different signal-to-noise ratios were obtained through real receivers and simulations. The SOQPSK signal received through the channel, after down-conversion, can be represented in an additive white Gaussian noise environment as:

[0090]

[0091] Among them, E b Let w(t) be the energy of each symbol, be additive Gaussian noise with power spectral density N0, and let f and θ represent the carrier residual frequency offset and the fixed phase offset of the received signal, respectively. Let τ represent the symbol timing delay. Since this invention only considers the implementation of SOQPSK carrier synchronization, it is assumed that the received signal has undergone ideal timing synchronization, i.e., τ is 0, and the modulation phase... It can be represented as:

[0092]

[0093] Where h represents the modulation index, the modulation index of SOQPSK-MIL is 1 / 2, and α i q(t) represents the pre-encoded transmission sequence, with values ​​ranging from {-1, 0, 1}; q(t) represents the phase pulse shaping function.

[0094] Step 2: Burst signal preprocessing; The received SOQPSK signal is first down-converted and burst signal detected, and it is assumed that symbol timing synchronization has been completed (symbol delay is 0), while retaining the effects of carrier frequency offset and phase offset; Subsequently, the signal is divided into real and imaginary parts, and fourth-order processing is performed to suppress modulation effects and enhance the carrier information expression capability.

[0095] Furthermore, firstly, the real signal data sampled by the front end is processed by Automatic Gain Control (AGC) to adjust the power of the sampled real signal data to a smaller range so that the power of burst signals and noise in the channel is similar. Then, the signal processed by AGC undergoes orthogonal downconversion and low-pass filtering to remove the second harmonic component and obtain the baseband data of the complex signal. Next, the complex signal data is processed by point sliding fast Fourier transform (FFT) to extract frequency domain features, and the frequency points in the FFT results are sorted from smallest to largest. The ratio of the peak frequency point to the mean of the in-band noise floor frequency point is calculated to form a frequency domain ratio sequence. Then, the frequency domain ratio sequence is analyzed using a sliding window. Each ratio in the window is compared with a set threshold. If the ratio in the sliding window exceeds a specified number, it is determined to be the starting point of a burst signal. Then, a frequency offset estimation algorithm is applied to downconvert the signal to a baseband signal, and the sampling rate is transformed to make the signal contain four sampling points per symbol.

[0096] Step 3: In order to adapt short-time burst SOQPSK signals to the input requirements of deep neural networks, the signals need to be image-constructed. The real part, imaginary part and the complex information after fourth power processing are organized into a unified 4×N feature map matrix, which can fully integrate the original time domain information and frequency offset features, so that the network can model the changing trend of frequency and phase perturbation in the feature space and improve the synchronous recognition capability.

[0097] Furthermore, firstly, the preprocessed signal (including the original real part, imaginary part, and its fourth-order real and imaginary parts) undergoes dimensionality reorganization. The original signal is a one-dimensional sequence of numbers of length 2N, where N represents the number of sampling points per bit (e.g., for 100 bits, with 2 sampling points per bit, N = 200). This sequence is then reorganized into an M×N matrix, where M is the number of bits and N is the number of sampling points per bit. This matrix is ​​constructed as follows: each row represents one bit, and different rows represent different symbol segments; odd-numbered indices in each column represent the real part of the signal, and even-numbered indices represent the imaginary part. In this way, the mapping from a one-dimensional signal to two-dimensional image features is completed, constructing an input matrix with spatial structure, which facilitates subsequent convolutional networks in modeling the phase structure of the signal between bits and sampling points.

[0098] Specifically, the signal visualization construction is implemented as follows:

[0099] The preprocessed signal data is sampled N times, where N represents the number of sampling points. Due to the preprocessing operation, the value of N is 2. Therefore, the mathematical model of the discrete sampling form of each bit of the SOQPSK signal after sampling can be expressed as:

[0100]

[0101] Next, the sampled signal r n (α) is divided into real and imaginary parts. Since the burst signal in this experiment is 100 bits of information, two 1*200 row vectors are constructed respectively.

[0102] Assume the real and imaginary parts of the sampled signal are respectively and The following matrix can then be obtained:

[0103]

[0104] This matrix R is a 2*200 matrix, where one row represents the real part of the signal and the other row represents the imaginary part;

[0105] r n (α) is processed to the fourth power to obtain (r) n (α)) 4 Its expression is:

[0106]

[0107] (r) n (α)) 4 Let X[k] be denoted as X[k]. Then transform X[k] into a 2*200 matrix, where one row is the real part and the other row is the imaginary part.

[0108] Then merge it with matrix R to obtain a new matrix I, which has a size of 4*200;

[0109]

[0110] Matrix I is passed as input data to a deep learning model for processing. This matrix represents the preprocessed input signal, where each column contains information about the real and imaginary parts of the signal, as well as the real and imaginary parts after being raised to the fourth power. The matrix data is then fed into a deep neural network, whose input layer receives and processes the features of these signals.

[0111] Through progressive processing by network layers, the data undergoes multiple convolutions, activations, and pooling operations, gradually extracting key features from the signal. These features are then used for tasks such as carrier synchronization and phase compensation, ultimately generating a synchronized signal through the output layer. By combining the signal's time-domain information with its fourth-order derivative, the phase changes of the signal are fully extracted. Furthermore, the characteristics of the fourth-order derivative of the SOQPSK signal are utilized to eliminate self-interference of symbols, effectively improving carrier synchronization accuracy.

[0112] Step 4: Extract burst SOQPSK signal features; use the residual connection convolution module to extract local temporal features. The feature map is first extracted by one-dimensional convolutional layer, batch normalization and activation functions (Mish, LeakyReLU) to extract pulse shaping structure and phase perturbation features. Then, the residual connection module is introduced to enhance the stability of feature flow propagation and the training effect of deep network, and avoid the gradient vanishing problem. This module effectively enhances the local representation ability of the signal under frequency offset interference.

[0113] Furthermore, in the residual connection module, this invention uses a convolutional neural network to extract features from the preprocessed SOQPSK signal. The input consists of four components, including the real part, the imaginary part, and their fourth power. First, the signal is passed through a one-dimensional convolutional layer (64 filters, kernel size 8) to extract local temporal features, used to capture the pulse shaping characteristics of the signal. Then, a residual structure is introduced to enhance network performance. The residual block includes two one-dimensional convolutional layers (each using 8 filters), each followed by batch normalization, with the Mish activation function used to improve nonlinear modeling ability and generalization. The residual path directly adds the input to the convolutional output, achieving cross-layer feature fusion and mitigating gradient vanishing. In addition, the LeakyReLU activation function is used at the residual output to effectively avoid dead neuron problems and enhance model stability. This module can significantly extract frequency offset and phase features, laying the foundation for subsequent synchronous processing.

[0114] Specifically, the signal feature extraction is implemented as follows:

[0115] The input signal is the output after data preprocessing, containing four components: the real part, the imaginary part, and the real and imaginary parts of the fourth power of the signal. In this stage, the input data is first pre-processed through a one-dimensional convolutional layer. This convolutional layer contains 64 filters, each with a kernel size of 8, to ensure that the temporal dimension of the output after the convolution operation remains consistent with the input. Local temporal features are extracted from the input signal to capture the pulse shaping characteristics of the SOQPSK signal.

[0116] To further enhance the model's representational ability and training stability, this invention introduces a residual network structure. The residual block directly adds the input to the output of the convolutional layer via a shortcut connection, effectively alleviating the gradient vanishing problem in deep network training and promoting cross-layer information transfer. Each residual block contains two consecutive convolutional layers, each using eight filters, and batch normalization is applied after each convolution operation to ensure that the output of each layer maintains a stable distribution during training. Furthermore, a Mish activation function is applied after the convolutional layers, introducing non-linear characteristics and enabling the network to learn more complex signal features. The formula for the Mish activation function is:

[0117] f(x) = x * tanh(ln(1+e) x (7)

[0118] It can retain negative value information while also having regularization properties, thus improving generalization ability.

[0119] To further alleviate the vanishing gradient problem and improve the model's robustness, this invention introduces the LeakyReLU activation function at the output of the residual block. Unlike the traditional ReLU activation function, LeakyReLU retains a small gradient in the negative range, effectively avoiding dead neurons and ensuring the stability of the network during training. Through this design, the residual convolutional layer can not only efficiently extract the phase features and frequency offset information of the SOQPSK signal, but also enhance the gradient fluidity and information transfer capability of the model during deep network training.

[0120] In summary, the combination of convolutional layers and residual blocks in this stage provides a foundation for subsequent carrier synchronization tasks. By extracting local time-domain features of the signal, this structure significantly improves the model's robustness and training performance under complex channel conditions, providing a high-precision signal representation for subsequent synchronization steps and signal demodulation.

[0121] Step 5: Model the global temporal features of the SOQPSK signal and perform synchronization compensation. The features, after passing through a phase-aware position encoding module, are fed into a two-layer Transformer structure, where a multi-head attention mechanism is used to establish long-distance phase dependencies. This mechanism can capture the cumulative phase shift under frequency offset from a global modeling perspective, and further correct carrier frequency offset errors using a fully connected layer, achieving accurate synchronization. The final output is a synchronization signal compensated by a deep network, whose bit error rate approaches the theoretical limit under different signal-to-noise ratio conditions, demonstrating superior carrier synchronization performance compared to traditional methods and other deep networks (such as CSDNN).

[0122] Furthermore, the global timing characteristics of the SOQPSK signal are modeled and synchronization compensation is implemented as follows:

[0123] First, the feature map output from step 4 is input to the phase-aware position encoding module. This module extracts the phase change features by calculating the instantaneous phase of the input signal (a complex number consisting of real and imaginary parts) and performing sine and cosine mapping. Simultaneously, a position encoding vector based on sine and cosine functions is constructed and fused with the phase features to enhance the model's ability to perceive phase accumulation changes and frequency drift, thereby improving synchronization accuracy.

[0124] Subsequently, the fused features are input into a two-layer Transformer encoder module. Each layer consists of a multi-head self-attention mechanism (4 heads in total, each with 8 dimensions) and a feedforward neural network (FFN). The self-attention mechanism calculates the relationship between Query, Key, and Value, globally modeling the temporal dependencies in the signal and effectively capturing the effects of frequency offset and phase transitions at different time steps.

[0125] Each encoder layer incorporates residual connections and layer normalization to improve training stability and mitigate the vanishing gradient problem. Finally, the synchronization features output by the Transformer serve as a signal compensation reference, achieving accurate carrier synchronization of the SOQPSK signal under complex conditions.

[0126] Specifically, the bidirectional short-duration memory module is implemented as follows:

[0127] First, during the initialization process, the input dimension d model The high-dimensional representation used to determine the position code is calculated based on periodic functions (sine and cosine functions), which have the following form:

[0128]

[0129] Where t represents the time step, i is the dimension index of the location encoding, and d model This refers to the dimension of the model, which is set to 64 in this invention. This encoding method transforms temporal information into a periodic and increasing high-dimensional vector, enabling the model to better capture time dependencies. To further improve synchronization accuracy, this module calculates the instantaneous phase of the input signal by representing it in complex form, defined as:

[0130] θ(t)=arg(x(t)) (9)

[0131] Where x(t) represents the complex form of the SOQPSK signal, and then, by performing sine and cosine mapping on the instantaneous phase θ(t), the phase coding features are obtained:

[0132] θ enc =[sin(θ(t)),cos(θ(t))] (10)

[0133] This phase encoding is processed by a fully connected layer so that it can be mapped to the model dimension d. model A consistent high-dimensional space enhances its expressive power; such as Figure 4 As shown, the processed phase code and position code are finally added element-wise to form the output of the phase-aware position code:

[0134]

[0135] This fusion approach preserves the low-level features of the original signal, avoids information loss, and effectively models the phase change characteristics of the SOQPSK signal. It enhances the network's ability to model linear phase accumulation, thereby compensating for frequency offset and phase errors in the signal. Furthermore, by performing fourth-power processing on the signal, it weakens the modulation component and highlights the four-fold component of the carrier phase, thus more effectively modeling the phase change characteristics of the SOQPSK signal and improving carrier synchronization accuracy. Compared to traditional synchronization methods, the phase-aware position coding module proposed in this invention significantly improves synchronization performance under low signal-to-noise ratio, dynamic frequency offset, and short-burst environments.

[0136] The core components of the Transformer encoder layer are a multi-head self-attention mechanism and a feedforward neural network. In this stage, the invention uses two encoder layers to extract deep features from the signal.

[0137] Multi-head self-attention mechanisms can learn different subspace features of the input signal in parallel using multiple "heads". Each head captures temporal dependencies by calculating the relationships between queries, keys, and values ​​in the input sequence. The calculation formula for multi-head attention is:

[0138]

[0139] Where Q, K, and V are the matrices of query, key, and value, respectively, and d k This refers to the dimension of the key. Through this computation, the model can focus on different parts of the sequence at each time step, effectively capturing the temporal dependencies of the signal. Especially in carrier synchronization tasks, this mechanism helps the model learn the effects of phase and frequency offsets on the signal. In this network, the Transformer coding layer uses four heads, each with a dimension of eight, balancing computational complexity and expressive power. Each coding layer also includes an FFN, which typically consists of two fully connected layers and uses a non-linear mapping through an activation function. The FFN helps improve the model's expressive power, enabling it to learn more complex signal features. In this implementation, the structure of the FFN is as follows:

[0140] FNN(x)=Dense(ReLU(W1x+b1))W2+b2 (13)

[0141] Where W1 and W2 are weight matrices, and b1 and b2 are bias terms. FFN performs a nonlinear transformation on the output of each layer, thereby enhancing the model's learning ability, especially when processing communication signals with nonlinear characteristics, effectively improving signal processing capabilities.

[0142] Residual connections and layer normalization are added after the attention mechanism and feedforward neural network in each layer. Residual connections help prevent gradient vanishing, allowing the network to be trained deeper and more stably; layer normalization helps accelerate training and improves stability and convergence speed during training. Through this design, the Transformer encoder can fully extract the features of the input signal, and at the same time, it combines the fourth power signal enhancement to improve sensitivity to phase error, capturing the cumulative phase change of the signal over long distances, obtaining residual frequency offset information, and performing effective synchronization error correction based on this. The introduction of the Transformer architecture makes the model perform well when handling complex signals, especially in the presence of Doppler frequency offset and phase offset, exhibiting stronger robustness and generalization ability.

[0143] Example:

[0144] 1. Dataset:

[0145] During the experiment, according to the P-band short burst SOQPSK-MIL protocol specifications, SOQPSK-MIL short burst signals were generated using Matlab, and Gaussian white noise was added. Because this invention targets non-data-assisted carrier synchronization, all data was randomly generated, with only the first symbol used as a reference phase for phase correction to prevent phase ambiguity. This invention primarily focuses on the carrier synchronization module of the signal; therefore, it is assumed that the symbol delay of the signal is 0 and does not pass through the symbol synchronization module. To improve the robustness of this model to fixed frequency and phase offsets, different fixed frequency and phase offsets were artificially introduced for each signal sample. Table 1 shows the parameters of the experimental dataset:

[0146] Table 1 Experimental Dataset

[0147]

[0148] 2. Experimental environment and network configuration:

[0149] The experiment used an NVIDIA TITAN RTX 3090 GPU as the server GPU and TensorFlow as the deep learning framework. The model training phase used 100,000 samples and 40 training epochs. An adaptive moment estimation optimizer was used to optimize the loss function. The learning rate was set to 1e-3, the weight decay coefficient was set to 1e-4, and the mean squared error (MSE) was used as the loss function.

[0150] 3. Performance simulation:

[0151] In addition to the network proposed in this invention, this example also uses a nonlinear quartic method, a decision-based carrier synchronization algorithm, and a CSDNN network carrier synchronization algorithm.

[0152] 3.1 Synchronization Algorithm Performance:

[0153] To verify the carrier synchronization performance of this network, the time-domain components of the transmitted and received signals were compared to obtain the effects of 1 / 4 phase offset and 0.01Rb frequency offset on the time-domain components of the signals. The transmitted signal was then compared with the synchronization signal to obtain the network synchronization result. To verify the robustness of the model to different signal-to-noise ratios, experiments were conducted using signals of 0dB and 12dB, respectively. Figure 5a This is a comparison chart of the real parts of the original signal and the received signal at 0dB. Figure 5b This is a comparison chart of the corresponding imaginary part data; Figure 5c This is a comparison chart of the real parts of the original signal and the received signal at 12dB. Figure 5d This is a comparison chart of the corresponding imaginary part data.

[0154] By observing the time-domain waveforms of the received and original signals, significant fluctuations can be observed, which is consistent with the effects of carrier frequency offset and phase deviation on the receiving end during transmission. An RPT-Net network model was trained using the experimental dataset. The received signal, after preprocessing, was then fed into the network model for carrier synchronization to obtain synchronized data. The synchronized data was then compared with the original signal to verify the effectiveness of the RPT-Net synchronization network. Figure 6a This is a comparison chart of the real parts of the original signal and the synchronized signal at 0dB. Figure 6b A comparison chart of the corresponding imaginary part data; Figure 6c This is a comparison chart of the real part data of the original signal and the synchronized signal at 12dB. Figure 6d This is a comparison chart of the corresponding imaginary part data.

[0155] from Figures 6a to 6d As can be seen, the received signal after passing through the carrier synchronization network of this invention has already matched the original signal, and basically no synchronization time is required. It is more suitable for carrier synchronization of short burst SOQPSK signals than phase-locked loops, and can still match the original signal under low signal-to-noise ratio conditions. It has more advantages than the nonlinear fourth method, which suffers from performance degradation under low signal-to-noise ratio conditions. Figures 6a to 6d The fit is better at 12dB than at 0dB, which is consistent with the theoretical situation.

[0156] (1) Compare the phase variance of carrier synchronization algorithms with different signal-to-noise ratios:

[0157] Under an AWGN channel, SOQPSK-MIL burst signal types with a fixed phase offset of 1 / 3 and a fixed frequency offset of 0.005Rb under different signal-to-noise ratio (SNR) environments are selected as the research object. The burst signal length is 100 bits, the timing error is set to 0, and the number of burst signal test samples is 2000 for each SNR. The bits in each signal test sample are randomly generated. The simulation results are shown in the figure below. Figure 7 As shown.

[0158] As the signal-to-noise ratio (SNR) increases, the errors of all methods decrease, consistent with theory. Among them, the nonlinear fourth-order method performs the worst, especially under low SNR conditions, because the noise variance increases when performing fourth-order operations on the signal, easily leading to phase abrupt changes. Direct decision-based carrier synchronization techniques reduce errors to some extent, but the need for fast synchronization in bursty signals introduces significant loop noise, thus its performance is inferior to deep learning methods. Although the CSDNN network performs better than traditional methods, it performs worse than the RPT-Net network because it ignores the memory characteristics of SOQPSK signals and lacks long-range dependencies. In contrast, the RPT-Net network's phase error is close to the theoretical lower bound MCRB, its estimation accuracy is higher, and its performance remains good under low SNR, showing a 1-2 dB performance improvement over the CSDNN network.

[0159] (2) Compare the demodulation bit error rate at different signal-to-noise ratios.

[0160] In an AWGN channel, a set of SOQPSK burst signal types with a fixed phase offset of 1 / 3 and a fixed frequency offset of 0.005Rb under different signal-to-noise ratio environments was selected as the research object. The burst signal length was 100 bits, the timing error was set to 0, and the number of burst signal test samples was 100,000. The signal was carrier synchronized using various algorithms, and the carrier-synchronized data was sent to the same demodulator for demodulation to obtain the bit error rate result. The simulation results are shown in the figure below. Figure 8 As shown.

[0161] As the signal-to-noise ratio (SNR) increases, the bit error rate (BER) of all methods decreases. At low SNR, the nonlinear fourth-order method exhibits the highest BER, consistent with the analysis above. The BER obtained by carrier synchronization using the RPT-Net network proposed in this invention is close to the theoretical BER. Under low SNR conditions, it offers a 1–2 dB performance improvement over traditional methods and approximately a 1 dB performance improvement over the CSDNN network.

[0162] To comprehensively verify the contribution of each module to the model, from the perspective of model training, the network composed of the residual connection module, the transformer module, and the position encoding module is called RT-Net, and the network composed of the phase-aware encoding module and the transformer module is called PT-Net. The above experiments were repeated to obtain... Figure 9a and Figure 9b The result.

[0163] from Figure 9a and Figure 9b As can be seen, the phase-aware position encoding module and the residual connection module of this invention contribute to the model, with the phase-aware position encoding module having a greater effect than the residual connection module. At a bit error rate of 1e-4, the RPT-Net network shows a 0.5dB performance improvement over the PT-Net network and a 0.7dB performance improvement over the RT-Net network, demonstrating their contributions to the RPT-Net network proposed in this invention.

[0164] In summary, this invention solves the problem of poor carrier synchronization performance of short burst SOQPSK signals under low signal-to-noise ratio conditions, and also solves the problem of poor performance of short burst SOQPSK signals under fast synchronization conditions. This invention has the characteristics of better carrier synchronization performance and strong robustness, and has better carrier synchronization performance under both low signal-to-noise ratio and fast synchronization conditions.

[0165] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A short burst SOQPSK carrier synchronization method, characterized in that... Includes the following steps: Step 1: Acquire burst SOQPSK signal sample data; Step 2: Signal sample data preprocessing; down-convert and burst signal detection are performed on the signal sample data, and symbol timing synchronization is set to complete the process. The signal sample data is divided into real and imaginary parts and then processed to the fourth power. Step 3: Construct an image from the preprocessed signal by reorganizing the real and imaginary parts (machine fourth power real and imaginary parts) to build a feature map matrix. Step 4: Extract burst SOQPSK signal features; Step 5: Model the global timing features of the SOQPSK signal and perform synchronization compensation; the extracted features are processed by a phase-aware position encoding module to increase the awareness of cumulative phase feature changes and frequency offset drift. Subsequently, a two-layer Transformer encoder module is used to establish long-distance phase dependence using a multi-head attention mechanism. Combined with a fully connected layer, the carrier frequency offset error is further corrected, and finally, the compensated synchronization signal is output; including the following steps: First, the extracted feature map is input into the phase-aware position encoding module to calculate the instantaneous phase of the input signal, which is a complex number consisting of real and imaginary parts, and then a sine-cosine mapping is performed to extract its phase change features. Simultaneously, a position encoding vector based on sine and cosine functions is constructed and fused with the phase features. The phase-aware position encoding module employs a bidirectional short-duration memory module, and the implementation process is as follows: First, the dimensions of the input. The high-dimensional representation used to determine the position code is calculated based on a periodic function, which has the following form: Where t represents the time step, and i represents the dimension index of the location encoding. Indicates the dimension of the model; Then, by representing the input signal in complex form, the instantaneous phase of the signal is calculated and defined as: in, Representing the complex form of the SOQPSK signal, and then, by considering the instantaneous phase... By performing sine and cosine mapping, the phase-encoded features are obtained: Phase-encoded features are mapped to the model dimension through a fully connected layer. Consistent high-dimensional space; Finally, the processed phase code and position code are added element-wise to form the output of the phase-aware position code: ； Subsequently, the fused features are input into a two-layer Transformer encoder module. Each layer consists of a multi-head self-attention mechanism and a feedforward neural network. The self-attention mechanism calculates the relationship between the query, key, and value to globally model the temporal dependence in the signal, thereby capturing the influence of frequency offset and phase transition at different time steps. Each encoder layer of the Transformer encoder module introduces residual connections and layer normalization. Finally, the synchronization characteristics output by the Transformer encoder module are used as a reference for signal compensation.

2. The short burst SOQPSK carrier synchronization method according to claim 1, characterized in that: In step 1, burst SOQPSK signal sample data under different signal-to-noise ratios are obtained through a real receiver and simulation. The SOQPSK signal received through the channel is down-converted and represented in an additive white Gaussian noise environment as follows: in, Represents the energy of each symbol. Let exp{·} denote the symbol period, exp{·} denote the exponential function, j denote the imaginary unit, f denote the carrier residual frequency offset, and t denote time. Indicates a timed delay for the symbol. This represents additive Gaussian noise. This indicates a fixed phase shift in the received signal. Represents the modulation phase function; Assuming the received signal has been timed and synchronized, that is If it is 0, then It is expressed as follows: in, This represents the pre-coded transmission sequence, where h represents the modulation index. This represents the phase pulse shaping function, where n represents the symbol index. Represents a code element sequence.

3. The short burst SOQPSK carrier synchronization method according to claim 1, characterized in that: In step 2, the acquired real signal data is subjected to automatic gain control processing to make the power of burst signals and noise in the channel similar; the signal after automatic gain control processing is subjected to quadrature downconversion and low-pass filtering to obtain the baseband data of the complex signal. Frequency domain features are extracted from the baseband data of the complex signal, and the ratio of the extracted peak frequency point to the mean of the in-band noise floor frequency point is calculated to form a frequency domain ratio sequence. Each ratio in the sliding window is compared with a set threshold. If the number of ratios in the sliding window that are greater than the threshold exceeds a certain number, it is determined to be the starting point of the burst signal. The signal is downconverted to baseband signal by frequency offset estimation, and the baseband signal is converted to contain four sampling points per symbol by sampling rate transformation.

4. The short burst SOQPSK carrier synchronization method according to claim 1, characterized in that: The dimension reorganization in step 3 is to reorganize a one-dimensional sequence of length 2N into an M×N matrix, where M is the number of bits and N is the number of sampling points corresponding to each bit. The matrix is ​​constructed such that each row represents one bit, different rows represent different symbol segments, and odd-numbered indices of each column represent the real part of the signal, while even-numbered indices represent the imaginary part. The specific implementation of the graphical construction in step 3 is as follows: The preprocessed signal is sampled N times, and the discrete sampling form of each bit after sampling is represented by the mathematical model as follows: The sampled signal Divided into real part and the virtual part The following matrix R is obtained: One row represents the real part of the signal, and the other row represents the imaginary part; Will The result of the fourth power process is: Will Record k represents the index of the sampling point, which is used to construct a matrix with one row of real parts and one row of imaginary parts. This matrix is ​​then merged with matrix R to obtain a new matrix I. Matrix I is used as input data and passed to the deep learning model.

5. The short burst SOQPSK carrier synchronization method according to claim 1, characterized in that: In step 4, local temporal features are extracted using a residual connection convolution module. The feature map is first extracted by a one-dimensional convolutional layer, batch normalization and activation function to extract pulse shaping structure and phase perturbation features, and then introduced into the residual connection module.

6. The short burst SOQPSK carrier synchronization method according to claim 5, characterized in that: In the residual connection module, a convolutional neural network is used to extract features from the feature map matrix; firstly, local temporal features are extracted through a one-dimensional convolutional layer to capture the pulse shaping characteristics of the signal; Subsequently, a residual structure is introduced to enhance network performance. The residual block consists of two one-dimensional convolutional layers, each followed by batch normalization and a first activation function. The residual path directly adds the input to the convolutional output. In addition, a second activation function is used at the residual output.

7. A short burst SOQPSK carrier synchronization method according to claim 6, characterized in that: The first activation function shown is the Mish activation function, and its formula expression is: Where x represents the input feature, f(x) represents the output feature of the activation function, and e represents the base of the natural logarithm; The second activation function is the LeakyReLU activation function, which retains a small gradient in the negative range.

8. The short burst SOQPSK carrier synchronization method according to claim 1, characterized in that: The dual-layer Transformer encoder module in step 5 includes a multi-head self-attention mechanism and a feedforward neural network in its encoding layer. Multi-head self-attention mechanisms learn different subspace features of the input signal in parallel through multiple heads. Each head captures temporal dependencies by calculating the relationship between the query, key, and value of the input sequence. The calculation formula for the head attention mechanism is as follows: Where Attention(·) represents a multi-head attention operation, and Q, K, and V represent the query, key, and value matrices, respectively. The dimension representing the key; Each coding layer comprises a feedforward neural network, which includes multiple fully connected layers and performs non-linear mapping through activation functions. In this implementation, the structure of the feedforward neural network is as follows: Where Dense(·) represents a fully connected operation, ReLU represents an activation operation, and x represents the input feature. and This represents the weight matrix and bias terms for the activation operation. and This represents the weight matrix and bias terms of the fully connected layer. FFN performs a non-linear transformation on the output of each layer. Residual connections and layer normalization are added after the attention mechanism and feedforward neural network in each layer.

Images