A probability shaping signal equalization method based on an improved LSTM nonlinear equalizer
By improving the LSTM nonlinear equalizer by using the MSE loss function and optimizing hyperparameters, the overfitting problem in the existing technology is solved, the signal equalization performance of the optical fiber communication system is improved, and better nonlinear impairment compensation and system capacity enhancement are achieved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- FUDAN UNIVERSITY
- Filing Date
- 2022-11-03
- Publication Date
- 2026-07-17
AI Technical Summary
Existing LSTM nonlinear equalizers use cross-entropy as the loss function when processing probability-shaped M-QAM and M-PAM signals, which leads to overfitting, reduces the model's generalization ability, and affects the signal equalization effect.
The structure of the LSTM nonlinear equalizer is improved by using mean square error (MSE) as the loss function. The sign fitting is improved by amplitude fitting. Combined with preprocessing steps such as resampling, frequency phase offset estimation and signal synchronization, the hyperparameters are optimized to reduce computational complexity.
It effectively avoids overfitting, improves the model's generalization ability, enhances the nonlinear impairment compensation effect of the optical fiber communication system, and improves signal quality and system capacity.
Smart Images
Figure CN115765876B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of nonlinear compensation technology in optical fiber communication, and in particular to a probabilistic shaping signal equalization method for optical fiber communication systems. Background Technology
[0002] With the rapid development of the information age, broadband data services such as cloud computing have grown rapidly. With the popularization of 5G technology, new services and applications requiring high speed, low latency, and wide connectivity, such as 4K / 8K video, VR, and autonomous driving, are constantly emerging, leading to a dramatic increase in data traffic. However, the current capacity of fiber optic communication networks is insufficient to meet the rapid growth of internet traffic. Therefore, the demand for ultra-high capacity, ultra-high speed, and ultra-long distance fiber optic communication has become increasingly urgent. To improve the transmission capacity and spectral efficiency of fiber optic communication, high-order PAM and high-order QAM modulation formats have become popular research directions. However, signals are subject to varying degrees of nonlinear impairment during fiber optic communication, which greatly affects the transmission rate and quality, hindering further improvements in system capacity. These nonlinear impairments mainly originate from various optical / electronic devices in the system, such as the square-law detection of photodiodes (PDs), the cosine characteristics of Mach-Zehnder modulators (MZMs), and the nonlinear effects of amplifiers and lasers; furthermore, as the transmission distance increases, the fiber optic cable also introduces severe nonlinear impairments.
[0003] Probabilistic Shaping (PS) technology can effectively mitigate the nonlinear effects of signals in optical channels. By changing the probability distribution of constellation points and altering the entropy of information, different data rates can be flexibly achieved, thereby increasing system capacity. The signal constellation points satisfy the Maxwell-Boltzmann distribution by utilizing a distribution matcher (DM).
[0004] Due to their high nonlinearity, neural networks can form arbitrary nonlinear decision boundaries, thus they can also be applied to nonlinear effect compensation. LSTM-based nonlinear equalizers (NNEs) are currently used in fiber optic system receivers for signal demodulation. Existing LSTM nonlinear equalizers use cross-entropy as the loss function to demodulate fourth-order pulse amplitude modulation (PAM-4) signals in intensity modulation / direct detection (IM / DD) systems, offering better performance and lower complexity compared to traditional equalization algorithms like Volterra. However, in reality, after PS (Power Sequencing), the constellation point distribution probabilities of M-QAM and M-PAM signals differ, making them non-uniform input samples for neural networks. Using cross-entropy as the loss function can lead to output symbols biased towards points with higher probabilities, resulting in overfitting, poorer equalization performance, and decreased model generalization ability. Summary of the Invention
[0005] The purpose of this invention is to overcome the shortcomings of the prior art and provide a probabilistic shaping signal equalization method for optical fiber communication systems.
[0006] The objective of this invention can be achieved through the following technical solutions:
[0007] A probabilistic shaping signal equalization method based on an improved LSTM nonlinear equalizer, the method comprising the following steps:
[0008] Sampling is used to obtain probabilistically shaped signals in fiber optic transmission systems;
[0009] Preprocess the probability-shaped signal;
[0010] The processed probability-shaped signal is input into an improved LSTM nonlinear equalizer trained with MSE as the loss function for nonlinear compensation to recover the original transmitted symbol.
[0011] As a preferred technical solution, the optical fiber transmission system includes a direct-modulation and direct-detection optical fiber transmission system, a coherent optical transmission system, and an optically generated millimeter-wave system.
[0012] As a preferred technical solution, in the direct-modulation and direct-detection optical fiber transmission system and the coherent optical transmission system, the probability-shaping signal obtained by sampling in the optical fiber transmission system is:
[0013] A photodetector is used to convert the optical signal transmitted through the optical fiber into an electrical signal, which is then sampled by an oscilloscope to obtain a probability-shaped signal.
[0014] As a preferred technical solution, in the optically generated millimeter-wave system, the probability-shaped signal obtained by sampling from the optical fiber transmission system is:
[0015] In the optically generated millimeter-wave system, a horn antenna is used to receive high-frequency electrical signals. After passing through a low-noise amplifier and a mixer, the high-frequency signal is down-converted to an intermediate frequency, and then sampled by an oscilloscope to obtain a probability-shaped signal.
[0016] As a preferred technical solution, the preprocessing includes pre-convergence, frequency phase offset estimation, and signal synchronization.
[0017] As a preferred technical solution, the preprocessing process includes:
[0018] The probabilistically shaped signal is resampled, and the CMA algorithm is used to pre-converge the resampled signal.
[0019] The frequency offset and phase offset of the pre-converged data were estimated using the Viterbi-Viterbi algorithm and the BPS algorithm.
[0020] Synchronize the frequency offset and phase offset estimated signals with the transmitting signals.
[0021] As a preferred technical solution, the process of nonlinear compensation performed by the improved LSTM nonlinear equalizer is as follows:
[0022] The real and imaginary parts of the N probabilistic shaped signals are extracted to form a real column vector of length 2N. The real column vector is used to form a dataset and input into an improved LSTM equalizer. The recovered signal is output from the output layer through the LSTM network and the fully connected layer. The number of neurons in the output layer is 2K, where K equals N. The 2K neurons in the output layer represent the real and imaginary parts of the N probabilistic shaped signals, respectively.
[0023] As a preferred technical solution, the training process of the improved LSTM nonlinear equalizer is as follows:
[0024] The acquired probability-shaped signal is input into the improved LSTM nonlinear equalizer to obtain the output layer recovery signal;
[0025] Using MSE as the loss function, the loss function of the real and imaginary parts of the recovered signal from the output layer is calculated.
[0026] By using the gradient descent method, the partial derivative of the loss function with respect to each weight in the neural network is calculated to determine the direction and magnitude of the correction of each weight parameter, thereby correcting the weights in the neural network.
[0027] As a preferred technical solution, the improved LSTM nonlinear equalizer has 2N neurons in the input layer, 4H neurons in the LSTM layer, L neurons in the fully connected layer, and 2K neurons in the output layer, with the goal of low complexity of the LSTM equalizer.
[0028] As a preferred technical solution, the formula for calculating the complexity C of the LSTM equalizer is:
[0029] C = 4H(2N+H) + HL + 2KL(MACC)
[0030] In the formula, N is the number of input probability-shaped signals, H is the number of neurons in each gated unit of the LSTM layer, L is the number of neurons in the fully connected layer, K is half the number of neurons in the output layer, and MACC is the multiply-accumulate number.
[0031] Compared with the prior art, the present invention has the following beneficial effects:
[0032] 1) This invention provides a probabilistic shaping signal equalization method based on an improved LSTM nonlinear equalizer. In the loss function, mean square error (MSE) replaces cross-entropy, thereby improving sign fitting to amplitude fitting, avoiding overfitting, and enhancing the model's generalization ability. This effectively compensates for nonlinear impairments in optical fiber communication.
[0033] 2) The various hyperparameters of the improved LSTM nonlinear equalizer provided by this invention, such as the number of hidden layers, the number of neurons, the learning rate, and the number of training iterations, can be selected according to specific circumstances. This invention also provides a method for calculating the complexity of the equalizer, setting the parameters in the LSTM equalizer with the goal of minimizing complexity, thereby improving the system's performance. Attached Figure Description
[0034] Figure 1 This is a schematic diagram illustrating the process of signal equalization using an improved LSTM nonlinear equalizer in this invention.
[0035] Figure 2 This is a schematic diagram of the LSTM equalizer training process of the present invention;
[0036] Figure 3 This is a schematic diagram of the fully connected layer in an LSTM equalizer.
[0037] Figure 4 This is a schematic diagram of the LSTM network structure. Detailed Implementation
[0038] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments. These embodiments are based on the technical solution of the present invention and provide detailed implementation methods and specific operating procedures. However, the scope of protection of the present invention is not limited to the following embodiments.
[0039] This invention provides an improved LSTM nonlinear equalization algorithm suitable for non-uniformly distributed probability-shaped signals such as PS-M-QAM and PS-M-PAM signals. The specific principle is as follows:
[0040] In the loss function, mean squared error (MSE) is used instead of cross-entropy, thereby improving the sign fitting to signal amplitude fitting, which compensates for the shortcomings of cross-entropy and improves system performance. The hyperparameters of the improved LSTM equalizer, such as the number of hidden layers, the number of neurons, and the number of training iterations, can all be adjusted.
[0041] S1: Sampling to obtain the probabilistically shaped signal in the optical fiber transmission system;
[0042] S2: Preprocess the probability-shaping signal;
[0043] S3: Input the processed probability-shaped signal into an improved LSTM nonlinear equalizer trained with MSE as the loss function for nonlinear compensation to recover the original transmitted symbol.
[0044] Since the QAM signal is composed of two orthogonal PAM signals, and the equalization method of this invention is for real number sequences, this embodiment provides an equalization method as follows: Figure 2The method steps shown are for using an improved LSTM equalizer to equalize more complex PS-M-QAM signals.
[0045] First, the signal used in this invention needs to be acquired in a communication experimental system. Specifically, the PS signal is emitted by the transmitting end. In a direct-modulation direct-detection fiber optic transmission / coherent optical transmission system, a photodetector converts the optical signal transmitted through the fiber optic cable into an electrical signal, which is then sampled by an oscilloscope to obtain the PS signal. In a light-generated millimeter-wave system, after the horn antenna receives the high-frequency electrical signal, it is down-converted to an intermediate frequency by a low-noise amplifier and a mixer, and then sampled by an oscilloscope to obtain the PS signal.
[0046] The sampled signal needs to be preprocessed. First, resampling is performed, followed by pre-convergence using the CMA algorithm. Due to frequency and phase offsets between the fiber optic communication system and the optical millimeter-wave communication system, further frequency and phase offset estimation is required using the Viterbi-Viterbi algorithm and the BPS algorithm to correct them. To ensure that the data volume at the transmitting and receiving ends is the same for training and testing the neural network, the frequency and phase offset corrected signal needs to be synchronized with the transmitting signal to complete the preprocessing.
[0047] The acquired signals are divided into training and test sets proportionally, with the training set being larger than the test set. The training process for applying the improved LSTM equalizer to PS-M-QAM signals using the training set is as follows:
[0048] like Figure 2 As shown, during a training process, the real and imaginary parts of N PS-M-QAM symbols are extracted into a real column vector of length 2N, which serves as the input to an LSTM equalizer. This vector then passes through an LSTM layer with four gated units for processing, forgetting, and selectively outputting data, with a total of 4H neurons, followed by a fully connected layer with L neurons, finally reaching the output layer. The output layer performs amplitude fitting with the ideal real and imaginary parts of the PS-M-QAM data from the transmitter, calculating the MSE as the loss function, thus completing one forward propagation. Subsequently, using gradient descent, the partial derivative of the loss function with respect to each weight in the neural network is calculated to determine the direction and magnitude of parameter correction, thereby adjusting the weights in the neural network and completing one backpropagation. This process is repeated until training is complete.
[0049] The parameters of the improved LSTM nonlinear equalizer, such as the number of hidden layers, the number of neurons, the learning rate, and the number of training iterations, can be selected according to specific circumstances. Due to the strong learning ability of neural networks, in order to prevent overfitting, the number of layers in LSTM networks used in communication systems is generally small, and the number of neurons in the hidden layers should not be too many. The parameters should be set with the goal of minimizing computational complexity while ensuring the improvement of system performance.
[0050] Complexity analysis of the improved LSTM equalizer:
[0051] Because offline DSPs and chip-based algorithms differ in their actual implementation, the floating-point operations per second (FLOPS) obtained from the computer side have limited reference value. Furthermore, the main computational structure of neural networks and traditional Volterra series equalizer-based models involves matrix / tensor multiplication. Here, we use the multiply-accumulate (MACC) / symbol corresponding to each symbol output, where one MACC corresponds to one multiplication and one accumulation, as a reference indicator for algorithm complexity.
[0052] First, calculate the computational complexity of a fully connected layer, such as... Figure 3 As shown, in a fully connected layer of a neural network, all inputs are connected to all outputs.
[0053] y = x·W + b
[0054] Its x can be stored in a 1×I matrix, the weight W can be stored in an I×J matrix, and b can be stored in a 1×J matrix.
[0055] Add=(I-1)×J+J=I×J
[0056] Mul=I×J
[0057] With an input batch size of 1, the computational cost of a fully connected layer is the input vector length I × the output length J. Since the activation function does not perform a dot product, the MACC cannot be directly calculated. In large networks, the MACC of the activation function can be ignored, so it is omitted here.
[0058] Then calculate the appendix Figure 3 The complexity of the LSTM network shown is as follows:
[0059] Three gated z in LSTM f z i z o And the input data z is composed of x t and h t-1 The concatenated vectors are obtained by multiplying the weight matrix, and they all have the same dimension (let's call them H). Since adding a circle to a matrix does not change the vector dimension, h... t and ct The dimension is also the same as the four states, so the computational complexity of the LSTM part is:
[0060] C = (2N + H) × H × 4 (MACC)
[0061] In summary, the complexity of the improved LSTM equalizer proposed in this invention is...
[0062] C = 4H(2N+H) + HL + 2KL(MACC)
[0063] The improved LSTM equalizer provided in this embodiment was tested using a test set:
[0064] The test set is input into the trained LSTM nonlinear equalizer, and the test dataset is calculated. Finally, the real and imaginary parts of the equalized PS-M-QAM signal are obtained in the output layer. The signal is then compared with the standard PS-M-QAM symbol transmitted by the transmitter, and the bit error rate is calculated.
[0065] The preferred embodiments of the present invention have been described in detail above. It should be understood that those skilled in the art can make numerous modifications and variations based on the concept of the present invention without creative effort. Therefore, all technical solutions that can be obtained by those skilled in the art based on the concept of the present invention through logical analysis, reasoning, or limited experimentation on the basis of existing technology should be within the scope of protection defined by the claims.
Claims
1. A probabilistic shaping signal equalization method based on an improved LSTM nonlinear equalizer, characterized in that, The method includes the following steps: Sampling is used to obtain probabilistically shaped signals in fiber optic transmission systems; Preprocess the probability-shaped signal; The processed probability-shaped signal is input into an improved LSTM nonlinear equalizer trained with MSE as the loss function for nonlinear compensation to recover the original transmitted symbol. The process of nonlinear compensation by the improved LSTM nonlinear equalizer is as follows: The real and imaginary parts of the N probabilistic shaped signals are extracted to form a real column vector of length 2N. The real column vector is used to form a dataset and input into an improved LSTM equalizer. The recovered signal is output from the output layer through the LSTM network and the fully connected layer. The number of neurons in the output layer is 2K, where K equals N. The 2K neurons in the output layer represent the real and imaginary parts of the N probabilistic shaped signals, respectively.
2. The probabilistic shaping signal equalization method based on an improved LSTM nonlinear equalizer according to claim 1, characterized in that, The optical fiber transmission system includes a direct-modulation and direct-detection optical fiber transmission system, a coherent optical transmission system, and an optically generated millimeter-wave system.
3. The probabilistic shaping signal equalization method based on an improved LSTM nonlinear equalizer according to claim 2, characterized in that, In the direct-modulation and direct-detection optical fiber transmission system and the coherent optical transmission system, the probability-shaping signal obtained by sampling in the optical fiber transmission system is: A photodetector is used to convert the optical signal transmitted through the optical fiber into an electrical signal, which is then sampled by an oscilloscope to obtain a probability-shaped signal.
4. The probabilistic shaping signal equalization method based on an improved LSTM nonlinear equalizer according to claim 2, characterized in that, In the optically generated millimeter-wave system, the probability-shaped signal obtained by sampling in the optical fiber transmission system is: In the optically generated millimeter-wave system, a horn antenna is used to receive high-frequency electrical signals. After passing through a low-noise amplifier and a mixer, the high-frequency signal is down-converted to an intermediate frequency, and then sampled by an oscilloscope to obtain a probability-shaped signal.
5. The probabilistic shaping signal equalization method based on an improved LSTM nonlinear equalizer according to claim 1, characterized in that, The preprocessing includes pre-convergence, frequency phase offset estimation, and signal synchronization.
6. The probabilistic shaping signal equalization method based on an improved LSTM nonlinear equalizer according to claim 5, characterized in that, The preprocessing process includes: The probabilistically shaped signal is resampled, and the CMA algorithm is used to pre-converge the resampled signal. The frequency offset and phase offset of the pre-converged data were estimated using the Viterbi-Viterbi algorithm and the BPS algorithm. Synchronize the frequency offset and phase offset estimated signals with the transmitting signals.
7. The probabilistic shaping signal equalization method based on an improved LSTM nonlinear equalizer according to claim 1, characterized in that, The training process of the improved LSTM nonlinear equalizer is as follows: The acquired probability-shaped signal is input into the improved LSTM nonlinear equalizer to obtain the output layer recovery signal; Using MSE as the loss function, the loss function of the real and imaginary parts of the recovered signal from the output layer is calculated. By using the gradient descent method, the partial derivative of the loss function with respect to each weight in the neural network is calculated to determine the direction and magnitude of the correction of each weight parameter, thereby correcting the weights in the neural network.
8. The probabilistic shaping signal equalization method based on an improved LSTM nonlinear equalizer according to claim 1, characterized in that, The improved LSTM nonlinear equalizer has 2N neurons in the input layer, 4H neurons in the LSTM layer, L neurons in the fully connected layer, and 2K neurons in the output layer, with the goal of low complexity for the LSTM equalizer.
9. A probabilistic shaping signal equalization method based on an improved LSTM nonlinear equalizer according to claim 8, characterized in that, The formula for calculating the complexity C of the LSTM equalizer is: In the formula, N is the number of input probability-shaping signals, H is the number of neurons in each gated unit of the LSTM layer, L is the number of neurons in the fully connected layer, and K is half the number of neurons in the output layer. It is a multiplicative addend.