A deep learning decoding method for LDPC codes based on hypernetwork

Abstract

The present invention discloses a deep learning decoding method for LDPC codes based on a hypernetwork. At present, the learning weights of channel neural decoding are trained for a specific channel environment, which makes it difficult to adapt to changes in the channel environment. In addition, the number of network layers of the channel decoder is difficult to flexibly change according to the application environment. In view of these two existing problems, a new channel neural decoder based on a hypernetwork is proposed. The decoder is composed of a hypernetwork and an NNMS network, wherein the NNMS network is the main network for channel neural decoding, and the hypernetwork generates learning weights for the main network; the hypernetwork is composed of a dense neural network, and its input and the input of the main network are both log-likelihood ratio LLR value vectors received by the channel. Experimental results show that compared with the existing channel neural decoding algorithm, the proposed decoding algorithm based on the hypernetwork can achieve better performance on LDPC, BCH and Hamming codes. In addition, the proposed framework can provide adaptability of learning weights and flexibility of network structure.

Description

Technical Field

[0001] The present invention relates to a deep learning decoding method of LDPC codes based on a hypernetwork. Background Art

[0002] Channel neural decoding can be mainly divided into two categories, one is the decoder based on data-driven deep learning, and the other is the decoder based on model-driven deep learning. The decoder based on data-driven deep learning uses a general neural network architecture, such as convolutional neural network (CNN), recurrent neural network (RNN), etc., to decode without using communication expert knowledge, which is a data-driven approach.

[0003] The authors in the paper [T.Gruber, S.Cammerer, J.Hoydis, and S.ten Brink, “On deep learning-based channel decoding,” in 2017 51st Annual Conference on Information Sciences and Systems (CISS). IEEE, 2017, pp. 1–6.] use a general neural network to decode random codes and achieve maximum a posteriori (MAP) decoding performance for very short block codes.

[0004] Due to the disadvantage that the decoder based on neural network (NN) is only effective for very short block codes, the authors in the literature [S.Cammerer, T.Gruber, J.Hoydis, and S.Ten Brink, "Scaling deep learning-based decoding of polar codes via partitioning," in GLOBE-COM 2017-2017IEEE globalcommunications conference. IEEE, 2017, pp.1–6.] proposed to replace the subcomponents of the polar decoding algorithm with a decoder based on neural network. In this way, the scalability of channel decoding based on deep learning is achieved.

[0005] The authors of the literature [H.Kim, Y.Jiang, R.Rana, S.Kannan, S.Oh, and P.Viswanath, "Communication algorithms via deep learning," arXiv preprintarXiv:1805.09317, 2018.] and [Y.Jiang, S.Kannan, H.Kim, S.Oh, H.Asnani, and P.Viswanath, "Deepturbo: Deepturbo decoder," in 2019IEEE 20th International Workshop on Signal ProcessingAdvances in Wireless Communications (SPAWC). IEEE, 2019, pp.1–5.] proposed RNN-based and CNN-based decoders to decode convolutional codes and Turbo codes respectively. The authors in the paper [A. Bennatan, Y. Choukroun, and P. Kisilev, "Deep learning for decoding of linear codes - a syndrome-based approach," in 2018 IEEE International Symposium on Information Theory (ISIT). IEEE, 2018, pp. 1595–1599.] use syndrome decoding to eliminate the overfitting problem that often occurs when training codeword sets. Model-free methods generally use fully connected neural networks or RNNs to implement the iterative process of the decoding algorithm. However, this method lacks the interpretability of the decoding performance, makes it difficult to analyze the reliability of the decoding results, and usually requires a large number of parameters.

[0006] Model-driven deep learning decoders are obtained by parameterizing traditional decoding algorithms, where the Tanner graph is unfolded into a neural network where learning weights are added to each edge. The unfolding method was proposed in the literature [JRHershey, JLRoux, and F.Weninger, "Deep unfolding: Model-based inspirationof novel deep architectures," arXiv preprint arXiv:1409.2574, 2014.] and has now become a bridge between traditional communication iterative algorithms and communication algorithms based on model-driven deep learning. Using the unfold method, the authors of the literature [E.Nachmani, Y.Be'ery, and D.Burshtein, "Learning todecode linear codes using deep learning," in 2016 54th Annual Allerton Conference on Communication, Control, and Computing (Allerton). IEEE, 2016, pp.341–346.] and [E.Nachmani, E.Marciano, L.Lugosch, WJGross, D.Burshtein, and Y.Be'ery, "Deep learning methods for improved decoding of linear codes," IEEE Journal ofSelected Topics in Signal Processing, vol.12, no.1, pp.119–131, 2018.] proposed the neural belief propagation (NBP) algorithm, which first unfolds the BP algorithm into a neural network and adds a trainable learning weight Tanner graph to each edge. Experiments show that the bit error rate performance of the NBP decoding algorithm for high-density parity-check (HDPC) codes exceeds that of the BP decoding algorithm.

[0007] The authors in the literature [L. Lugosch and WJ Gross, "Neural offset min-sum decoding," in 2017 IEEE International Symposium on Information Theory (ISIT). IEEE, 2017, pp. 1361–1365.] expanded the offset minimum sum (OMS) into a neural network form and proposed the neural offset minimum sum (NOMS) algorithm, which has lower computational complexity than the NBP algorithm. The authors then applied the syndrome loss to the neural normalized minimum sum (NNMS) algorithm (such as the literature ["Learning from the syndrome," in 2018 52nd Asilomar Conference on Signals, Systems, and Computers. IEEE, 2018, pp. 594–598.]), thereby achieving unsupervised training of the channel neural decoding algorithm. The authors in the literature [Q.Wang, S.Wang, H.Fang, L.Chen, L.Chen, and Y.Guo, “A model-driven deep learning method for normalized min-sumldpc decoding,” in 2020IEEE International Conference on CommunicationsWorkshops(ICC Workshops).IEEE, 2020, pp.1–6.] and [Q.Wang, Q.Liu, S.Wang, L.Chen, H.Fang, L.Chen, Y.Guo, and Z.Wu, “Normalized min-sum neural network for ldpcdecoding,” IEEE Transactions on Cognitive Communications and Networking, 2022.] applied the NNMS algorithm to long low-density parity-check (LDPC) codes.The literature [J.Dai, K.Tan, Z.Si, K.Niu, M.Chen, HVPoor, and S.Cui, "Learning to decode protograph ldpc codes," IEEE Journal on Selected Areas in Communications, vol.39, no.7, pp.1983–1999, 2021.] proposed a high-performance neural minimum sum (MS) decoding method that fully utilizes the lifting structure of the original graph LDPC (P-LDPC) code. Channel neural decoding surpasses traditional decoding in decoding performance, but there are still many difficulties in highly mobile time-varying channel environments and noise levels. Once the neural decoder is fixed after training, the internal parameters and number of layers cannot be changed. Once the noise level or the number of layers changes, retraining is required, and the stability and optimality of the algorithm cannot be guaranteed.

[0008] Hypernetworks use one network to generate the weight coefficients of another network, which was first proposed in the literature [D.Ha,A.Dai,and QVLe,“Hypernetworks,”arXiv preprint arXiv:1609.09106,2016.]. In order to improve the adaptability of multiple-input multiple-output (MIMO) detection based on machine learning (ML) to different channel environments and noise levels, the authors of the literature [M.Goutay, FAAoudia, and J.Hoydis, "Deep hypernetwork-based mimodetection," in 2020IEEE 21st International Workshop on Signal ProcessingAdvances in Wireless Communications (SPAWC). IEEE, 2020, pp.1–5.] and [J.Zhang, C.-K.Wen, and S.Jin, "Adaptive mimo detector based on hypernetwork: Design, simulation, and experimental test," IEEE Journal on Selected Areas in Communications, vol.40, no.1, pp.65–81, 2021.] introduced hypernetworks into ML-based MIMO detection and proposed HyperMMNet and HyperEPNet respectively. The authors in the paper [Y.Liu and O.Simeone, "Learning how to transfer from uplink to downlink via hyper-recurrent neural network for fddmassive mimo," IEEE Transactions on Wireless Communications, 2022.] proposed a HyperRNN architecture for an end-to-end downlink channel estimation scheme for massive MIMO frequency division duplex (FDD) systems, achieving lower normalized mean square error (NMSE) channel estimation and higher sum of beamforming rates.The authors in the literature [W.-C.Tsai, C.-W.Chen, C.-F.Teng, and A.-Y.Wu, "Low-complexity compressive channel estimation for irs-aided mmwave systems with hypernetwork-assisted lampnetwork," IEEE Communications Letters, 2022.] applied hypernetworks to the channel estimation algorithm in reconfigurable smart surface (RIS)-assisted communication systems.

[0009] The authors of the paper [E.Nachmani and L.Wolf, "Hyper-graph-network decoders for blockcodes," Advances in Neural Information Processing Systems, vol. 32, 2019.] transformed the message graph of algebraic block codes into a graph neural network, whose learning weights were generated using a hypernetwork. This solved the challenge of difficult training of graph neural networks for message passing. However,

[20] only focused on the BP decoding algorithm and did not involve the min-sum algorithm, which is more hardware-friendly. Compared with the NBP algorithm, the NNMS algorithm has lower computational complexity and is more suitable for hardware implementation. In recent years, based on the NNMS algorithm, the literature [Y. Liang, C.-T. Lam, and BKNg, “A low-complexity neural normalized min-sum ldpc decoding algorithm using tensor-train decomposi-tion,” IEEE Communications Letters, vol. 26, no. 12, pp. 2914–2918, 2022.] and [“Joint-way compression for ldpc neural decoding algorithm with tensor-ring decomposition,” IEEE Access, 2023.] proposed a higher performance version of the NNMS+ algorithm and a lower complexity version of the TT-NNMS+ algorithm, so that different versions can be selected for application as needed.

[0010] Hypernetworks can improve the adaptability of channel neural decoders and avoid repeated training due to changes in channel environment and number of network layers. However, deploying additional hypernetworks in hardware will increase the computational complexity of neural decoding algorithms, making the already limited communication hardware resources even more challenging. In recent years, tensor training (TT) and tensor ring (TR) decomposition have been widely used in model compression because they can overcome the curse of dimensionality. In recent years, some scholars have applied them to intelligent communications. The authors of the literature [Y. Liang, C.-T. Lam, and BK Ng, “A low-complexity neural normalized min-sum ldpc decoding algorithm using tensor-train decomposition,” IEEE Communications Letters, vol. 26, no. 12, pp. 2914–2918, 2022.] and [“Joint-way compression for ldpc neural decoding algorithm with tensor-ring decomposition,” IEEE Access, 2023.] introduced TT decomposition and joint-way compression into channel neural decoding, greatly reducing the required storage space and computational complexity while maintaining the decoding performance.

[0011] However, to the best of our knowledge, TT and TR decompositions have not been applied to hypernetworks. Tensor decomposition has some similarities with neural networks in mathematical principles and structure. The authors of [N. Cohen, O. Sharir, and A. Shashua, "On the expressive power of deep learning: A tensor analysis," in Conference on learning theory. PMLR, 2016, pp. 698–728.] and [V. Khrulkov, A. Novikov, and I. Oseledets, "Expressive power of recurrent neural networks," arXiv preprint arXiv: 1711.00811, 2017.] use hierarchical Tucker (HT) tensor decomposition and TT decomposition to explain the expressive power of CNN and RNN, respectively. The literature [M.Astrid and S.-I.Lee, "Cp-decomposition with tensorpower method for convolutional neural networks compression," in 2017 IEEE International Conference on Big Data and Smart Computing (BigComp). IEEE, 2017, pp. 115–118.] and [V.Lebedev, Y.Ganin, M.Rakhuba, I.Oseledets, and V.Lempit-sky, "Speeding-up convolutional neural networks using fine-tuned cp-decomposition," arXiv preprint arXiv:1412.6553, 2014.] decomposes the convolution kernel into a canonical multivariate (CP) format and establishes a deep bottleneck architecture. The authors in the literature [Y.Chen, X.Jin, B.Kang, J.Feng, and S.Yan, "Sharing residual units through collective tensor factorization to improved eep neural networks." in IJCAI, 2018, pp.635–641.] use tensor block term decomposition (BTD) decomposition to establish a unified framework to express a variety of significantly different residual functions.The literature [J.-T.Chien and Y.-T.Bao, "Tensor-factorized neural networks," IEEE transactions on neural networks and learning systems, vol. 29, no. 5, pp. 1998–2011, 2017.] regards the entire process of Tucker decomposition as a neural network process and proposes a tensor factorization neural network (TFNN). The literature [J.Kossaifi, ZC Lipton, A.Kolbeinsson, A.Khanna, T.Furlanello, and A.Anandkumar, "Tensorregression networks," The Journal of Machine Learning Research, vol. 21, no. 1, pp. 4862–4882, 2020.] incorporates tensor contraction as a differentiable layer into the neural network, and on this basis proposes a tensor contraction layer and a tensor regression layer respectively. Summary of the invention

[0012] The technical problem to be solved by the present invention is to provide a deep learning decoding method for LDPC codes based on a hypernetwork, and to use the hypernetwork for the NNMS algorithm and its low-complexity version. The hypernetworks designed for the channel neural decoder can be divided into two types: static and dynamic. Static hypernetworks can provide learning weights for the main network, but cannot flexibly change the number of layers of the decoding network. On the other hand, dynamic hypernetworks can extend the channel neural decoder to any number of layers without retraining.

[0013] The present invention adopts the following technical solutions to solve the above technical problems:

[0014] A deep learning decoding method for LDPC codes based on a hypernetwork, the method is a combination of a hypernetwork and an NNMS network, wherein the NNMS network is a main network for channel neural decoding, and the hypernetwork is used to generate learning weights for the main network;

[0015] The super network consists of a dense neural network, and its input, like the input of the main network, is the log-likelihood ratio (LLR) value vector received by the channel.

[0016] Furthermore, the tanh function is used as the activation function of the hypernetwork; under the guidance of the cross entropy loss function, stochastic gradient descent SGD is used to train the hypernetwork.

[0017] The cross entropy function is defined as:

[0018]

[0019] where o i is the i-th bit of the output of the NNMS network estimation, x i is the i-th bit of the transmitted codeword, and N is the number of bits of the transmitted codeword.

[0020] Furthermore, the hypernetwork is a static hypernetwork, and generates the learning weights of the main network based on the following expression:

[0021]

[0022] where is the learning weight of the static hypernetwork , is the message transmitted from the CN node c to the VN node v at time t, is the message transmitted from the VN node v ′ to the CN node c at time t - 1, M(c) is the set of all VN nodes connected to the CN, and M(c)\v is the set of VN nodes in M(c) except the VN node v.

[0023] Furthermore, the hypernetwork is a dynamic hypernetwork, and generates the learning weights of the main network based on the following expression:

[0024]

[0025] where W α is the learning weight of the dynamic hypernetwork f (t-1) (:,W α ), is the message transmitted from the CN node c to the VN node v at time t, is the message transmitted from the VN node v ′ to the CN node c at time t - 1, M(c) is the set of all VN nodes connected to the CN, and M(c)\v is the set of VN nodes in M(c) except the VN node v.

[0026] Furthermore, the dynamic hypernetwork is composed of T hypernetwork units corresponding one-to-one to each layer of the main network. Each hypernetwork unit generates the learning weights for the corresponding layer of the main network. The input of the first hypernetwork unit is the vector of log-likelihood ratio (LLR) values received by the channel, and the input of the t-th hypernetwork unit is the output of the (t - 1)-th layer of the main network, where 1 < t ≤ T and T is the number of layers of the main network.

[0027] There is another deep learning decoding method for LDPC codes based on a hypernetwork. This method combines a hypernetwork and a TR-NNMS+ network. Among them, the TR-NNMS+ network is the main network for channel neural decoding, and the hypernetwork generates the learning weights for the main network;

[0028] The hypernetwork consists of a dense neural network, and the inputs of both it and the main network are vectors of log-likelihood ratio (LLR) values received by the channel.

[0029] Furthermore, the hypernetwork is a static hypernetwork. The number of network layers of the static hypernetwork is the same as the number of factor tensors in the TR decomposition. Each layer of the static hypernetwork generates a learning weight tensor for the corresponding factor tensor, and the size of the learning weight tensor of the static hypernetwork is the same as the size of the factor tensor in the TR decomposition.

[0030] The input of the static hypernetwork is the LLR received by the channel or the information tensor output by the previous layer of the main network.

[0031] Furthermore, activation functions are added between the learning weight tensors to form an m-layer static hypernetwork.

[0032] Furthermore, the hypernetwork is a dynamic hypernetwork composed of T hypernetwork units corresponding one-to-one to each layer of the main network. Each hypernetwork unit generates a learning weight for the corresponding layer of the main network. The input of the first hypernetwork unit is the vector of log-likelihood ratio (LLR) values received by the channel, and the input of the t-th hypernetwork unit is the output of the (t - 1)-th layer of the main network, where 1 < t ≤ T and T is the number of layers of the main network.

[0033] Compared with the prior art, the present invention adopts the above technical solutions and has the following technical effects:

[0034] The present invention proposes a novel channel neural decoder framework based on a hypernetwork to adaptively generate learning weights and adjust the number of network layers according to the application environment. The addition of the hypernetwork leads to higher computational complexity and storage requirements. To reduce the complexity of the proposed framework, based on the structural similarity between the tensor ring (TR) decomposition and the hypernetwork, the storage space of the learning weights is reused between the hypernetwork and the channel neural decoder (referred to as TRHyper). By sharing the storage space, TRHyper greatly reduces the storage space and computational complexity of the hypernetwork-based algorithm. Numerical results show that compared with the existing channel neural decoding algorithms, the proposed hypernetwork-based decoding algorithm can achieve better performance on LDPC, BCH, and Hamming codes. In addition, the proposed framework can provide adaptability of learning weights and flexibility of network structure. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 is the static hypernetwork based on NNMS;

[0036] Figure 2 is the dynamic hypernetwork based on NNMS and TR-NNMS+;

[0037] Figure 3 is the schematic diagram of TRHyper-NNMS+, where m is equal to 4. DETAILED DESCRIPTION

[0038] The embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and cannot be interpreted as limiting the present invention.

[0039] It will be understood by those skilled in the art that, unless expressly stated, the singular forms "one", "said", and "the" used herein may also include plural forms. It should be further understood that the term "comprising" used in the specification of the present invention refers to the presence of the features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof. It should be understood that when we refer to an element as being "connected" or "coupled" to another element, it may be directly connected or coupled to the other element, or there may be an intermediate element. In addition, the "connection" or "coupling" used herein may include wireless connection or coupling. The term "and / or" used herein includes any unit and all combinations of one or more associated listed items.

[0040] It will be understood by those skilled in the art that, unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as those generally understood by those skilled in the art in the art to which the present invention belongs. It should also be understood that terms such as those defined in common dictionaries should be understood to have meanings consistent with the meanings in the context of the prior art, and will not be interpreted with idealized or overly formal meanings unless defined as herein.

[0041] The technical solution of the present invention is further described in detail below in conjunction with the accompanying drawings:

[0042] The channel neural decoding algorithm is superior to the traditional decoding algorithm and has been widely studied in recent years. However, the learning weights of the current channel neural decoding are trained for a specific channel environment, which makes it difficult to adapt to changes in the channel environment. Once the channel neural decoding algorithm is deployed, the learning weights and the number of network layers cannot be changed.

[0043] This paper proposes a new channel neural decoder framework based on a hypernetwork to adaptively generate learning weights and adjust the number of network layers according to the application environment. However, the addition of the hypernetwork leads to higher computational complexity and storage requirements.

[0044] In order to reduce the complexity of the proposed framework, a low-complexity hypernetwork technology (called TRHyper) is proposed based on the similarity between tensor ring (TR) decomposition and hypernetwork in mathematical principles and structure. The storage space of the learning weights of the TR-NNMS+ algorithm proposed in

[22] is reused to store the learning weights of the hypernetwork, so that the learning weights of the hypernetwork no longer occupy additional storage resources. The hypernetwork in the TRHyper solution generates the optimal learning weights for the TR-NNMS+ algorithm, saving the training cost required from NNMS+ to TR-NNMS+.

[0045] 1. NNMS decoding algorithm based on hypernetwork

[0046] The NNMS algorithm has two limitations that we aim to address. First, the learning weights of the channel neural decoding cannot be automatically updated as the noise level changes, resulting in inability to guarantee stability and optimality and requiring frequent retraining. Second, the number of layers of the channel neural decoding cannot be adaptively changed as required by different application scenarios. To overcome the above problems, we combine the hypernetwork and the channel neural decoding and propose the following channel neural decoding algorithm based on the hypernetwork, where the hypernetwork generates learning weights for the main network.

[0047] The hypernetwork solution consists of two neural networks, one is the channel neural decoding network, which is the main network, and the other is the hypernetwork, which generates learning weights for the main network. The hypernetwork consists of a dense neural network with L layers, each with 32 neurons. We use the tanh function as the activation function of the hypernetwork. The input of the hypernetwork is the same as the input of the main network, that is, the received log-likelihood ratio (LLR) value vector. Stochastic gradient descent (SGD) is used to train the hypernetwork under the guidance of the cross entropy loss function. According to the difference in information interaction between the hypernetwork and the main network, the hypernetwork can be divided into two types: static and dynamic.

[0048] 1. Static HyperNNMS (S-HyperNNMS)

[0049] Let the hypernetwork be represented by a parameterized function f(:,W), where W is the learning weight of the hypernetwork. The learning weights of the NNMS algorithm are generated by a static hypernetwork and can be expressed as:

[0050]

[0051] in, It is a static hypernetwork The learning weights of .

[0052] like Figure 1 As shown, the static super network generates all the learning weights required by the main network NNMS This eliminates the need to retrain the NNMS main network when the channel environment changes.

[0053] (II) Dynamic HyperNNMS (D-HyperNNMS)

[0054] The static supernetwork solution can generate learning weights for the main network, avoiding retraining the main network when the channel environment or noise level changes. However, the number of learning weight vectors output by the supernetwork is fixed. Once the number of layers of the main network changes, the output dimension of the supernetwork needs to be changed, so the supernetwork needs to be retrained. Therefore, the present invention further proposes a dynamic supernetwork to solve this problem. In order to be compatible with the static supernetwork, the dynamic supernetwork still uses a dense neural network. But in the dynamic solution, the supernetwork dynamically generates learning weights for the main network, which can change online across layers.

[0055] The learning weights in the NNMS algorithm are generated by dynamic D-HyperNNMS and can be expressed as:

[0056]

[0057] Among them, W α is a dynamic hypernetwork (t-1) (:,W α )’s learning weights.

[0058] Different from the S-HyperNNMS algorithm, in the D-HyperNNMS algorithm, the hypernetwork implements information interaction with the NNMS network at each layer. Figure 2 As shown in the figure, the hypernetwork dynamically provides learning weights to the NNMS network of each layer, and the NNMS network provides the hypernetwork with the information required for calculation. Each G is a hypernetwork unit composed of a dense neural network. Unlike the S-HyperNNMS algorithm, each hypernetwork unit in the D-HyperNNMS algorithm can share learning weights across layers. Therefore, when the number of NNMS network layers changes, retraining is not required.

[0059] The present invention proposes static and dynamic HyperNNMS algorithms based on NNMS. The static HyperNNMS algorithm, which consists of a hypernetwork and an unfolding algorithm called NNMS, can improve the adaptability of the NNMS algorithm under different noise levels. The hypernetwork generates learning weights for the NNMS algorithm to avoid repeated training due to changes in the channel environment. The dynamic HyperNNMS algorithm, in which learning weights and messages are exchanged between the hypernetwork and the main network at each layer, in addition to the advantages of the static HyperNNMS, the number of layers of the dynamic HyperNNMS algorithm can be arbitrarily expanded without retraining.

[0060] TRHyper

[0061] The static hypernetwork and dynamic hypernetwork proposed above can solve the problem that the channel neural decoding algorithm needs to be frequently retrained when the noise level changes or the number of network layers changes. However, the hardware implementation of the hypernetwork requires a lot of memory storage and computing resources, which is a huge challenge for communication hardware. Especially for the TR-NNMS+ network, due to the special learning weight structure, multiple dense networks are required to generate learning weights simultaneously, which requires more memory storage and computing resources, but the performance is still difficult to meet the requirements. To meet these challenges, we propose a low-complexity hypernetwork technology with TR decomposition for channel neural decoding, called TRHyper.

[0062] 1. Hypernetwork based on TR decomposition

[0063] Assume that a tensor W is decomposed into m factor tensors W1, W2, ..., W by TR. m . If we use these m factor tensors as the learning weights of the neural network. Then add an activation function f between each two tensors. Finally, give the neural network an input X. We can build an m-layer dense network as follows:

[0064] Y=f(f(…f(f(X*W1)*W2)*…)*W m )

[0065] The above process builds a bridge between m tensors and m-layer neural network. After processing, the m tensors can be used as learning weights of the neural network.

[0066] The hypernetwork is composed of a neural network with multiple dense layers. Using the relationship between the tensors and neural networks mentioned above, the m-layer hypernetwork is established based on the m factor tensors obtained after TR decomposition of the learning weights. At the same time, the similarities between them are used to reuse storage space and weights, thereby reducing the memory storage and computing resources required by the hypernetwork.

[0067] (II) TRHyper solution

[0068] TRHyper uses the structural similarity between TR decomposition and dense neural networks. During the training process, the storage space required for learning weights can be reused, saving a lot of storage resources. It also avoids the inverse calculation of TR decomposition required for TR-NNMS+ algorithm training, thereby saving a lot of computing resources. In addition, the training of factor tensors required by TR-NNMS+ is no longer required, saving training costs. According to the information interaction between the hypernetwork and the main network, the TRHyper solution is also divided into static and dynamic.

[0069] 1. Static TRHyper (S-TRHyper)

[0070] The S-TRHyper solution proposed in the present invention is a super network composed of m layers of dense neural networks, and the size of each layer is specially designed so that the storage space can be reused. TR-NNMS+ is obtained by compressing the NNMS+ algorithm based on TR decomposition. Therefore, the learning weights of each layer in the TR-NNMS+ algorithm are composed of $m$ factor tensors. These m factor tensors can participate in the calculation of the layer after the inverse operation of TR decomposition during training. These learning weights need to be trained for a specific channel environment to achieve the purpose of optimization. The size of the m layer in the super network is designed to be the size of $m$ factor tensors, so that the storage space for storing m factor tensors can be reused to store the learning weights of the m layer in the super network, thus saving the storage resources of the learning weights of the m layer of the super network.

[0071] In addition, in the TRHyper solution proposed in the present invention, the forward calculation of the hypernetwork is designed to complete the inverse calculation of TR decomposition required in TR-NNMS+ training. The output of the hypernetwork participates in the calculation of the corresponding layer of the TR-NNMS+ network as the result of the inverse operation. This design saves the computing resources required for the inverse calculation of TR decomposition. The output of the hypernetwork after training is the optimal learning weight of the TR-NNMS+ algorithm, avoiding the training of learning weights in the factor tensor format required for the optimization of the TR-NNMS+ algorithm. Therefore, the TRHyper solution also saves a lot of training costs.

[0072] like Figure 3 As shown, the S-TRHyper solution can be divided into the following 5 steps:

[0073] Step 1: Design a compression scheme based on TR decomposition for the learning weights of the NNMS+ decoding algorithm, and obtain the number m and factor tensors of the TR decomposition scheme The size of {D1,…,D m}. According to m and Design storage space O to store the factor tensor after TR decomposition (i.e., the learning weight of the TR-NNMS+ algorithm). Set the number of network layers of the hypernetwork to be equal to the number of factor tensors, which is also m. The learning weight tensor W of the hypernetwork i , i = 1, ..., m and the size of the factor tensor D i Consistent, that is In this way, the designed storage space O can also be used to store the learning weights of the designed hypernetwork.

[0074] Step 2: During training, the hypernetwork uses memory space O to store the learned weights W i,i=1,…,m. Add m tanh activation functions between the learning weight tensors to form an m-layer hypernetwork. The designed hypernetwork does not have its own loss function. It uses the cross entropy loss function with the main network. The input of the hypernetwork corresponds to the input of the main network, that is, the LLR received by the channel or the information tensor output of the upper layer of the main network.

[0075] Step 3: If Figure 3 As shown, the established super network is trained together with the main network, and the learning weights are updated under the guidance of the loss function and the back-propagation mechanism.

[0076] Step 4: After training, we will get the optimized TR-NNMS+ algorithm based on TRHyper. The learning weights of the m-layer hypernetwork use the size of m factor tensors respectively. The forward calculation process of the hypernetwork can be regarded as the inverse process of TR decomposition. Therefore, the output of the hypernetwork is the optimal learning weight of the TR-NNMS+ algorithm. In other words, from the NNMS+ algorithm to the TR-NNMS+ algorithm, it is no longer necessary to train separately as in the previous step, thus saving a lot of training costs.

[0077] Step 5: If the channel environment changes, repeat steps 3-4.

[0078] 2. Dynamic TRHyper (D-TRHyper)

[0079] Unlike S-TRHyper, D-TRHyper interacts with the main network at every layer. Figure 2 As shown in Figure 1, each hypernetwork unit G consists of a dense neural network, similar to the static TRHyper. Each layer of the main network accepts the learning weights generated by the hypernetwork to complete the calculation of the current layer. Then the output of each layer is not only used as the input of the next layer, but also provided to the hypernetwork as the calculation of the next iteration. The remaining steps are consistent with the S-TRHyper scheme.

[0080] The S-TRHyper and D-TRHyper schemes designed by the present invention fully utilize the structural similarity between the factor tensor obtained after tensor decomposition and the dense neural network to reuse the storage space. The TRHyper scheme saves a lot of storage resources required for learning weights and saves the training cost required from NNMS+ to TR-NNMS+ algorithm.

[0081] Compared with the NNMS algorithm, the TR-NNMS+ algorithm has better performance and lower complexity. Each learning weight tensor of the TR-NNMS+ algorithm is decomposed into multiple factor tensors by TR. Therefore, neither the static hypernetwork scheme nor the dynamic hypernetwork scheme of the NNMS algorithm can be directly applied to the TR-NNMS+ algorithm. For the learning weight structure of TR-NNMS+, we use multiple dense neural networks as hypernetworks, each of which generates learning weights for the factor tensor in TR format. According to this criterion, we designed the corresponding static hypernetwork scheme and dynamic hypernetwork scheme for the TR-NNMS+ algorithm.

[0082] In order to reduce the computational complexity and required storage resources of the hypernetwork, we proposed the TRHyper scheme, which fully exploits the mathematical principles and structural similarities between TR decomposition and the hypernetwork. TRHyper reuses the learning weight storage space of TR-NNMS+ to store the learning weights of the hypernetwork. Therefore, the learning weights of the hypernetwork no longer require additional storage space, saving a lot of storage resources. In addition, the hypernetwork in the TRHyper scheme generates the optimal learning weights for the TR-NNMS+ algorithm, without the need to train from the NNMS+ algorithm to the TR-NNMS+ algorithm, saving a lot of training costs.

[0083] 3. Complexity Analysis

[0084] We assume that the number of learned weights of the dense network used in the hypernetwork-based channel neural decoding algorithm is W, and the required memory storage is O(W). As shown in Table 1, the memory storage required by the hypernetwork used in the HyperNNMS solution, HyperTR-NNMS+ solution, and TRHyper solution is listed.

[0085] For the S-HyperNNMS solution, the hypernetwork is a dense neural network, the number of learning weights of the hypernetwork is W, and the required memory storage is O(W). For the D-HyperNNMS solution, the learning weights and the memory storage occupied by the learning weights are the same as those of S-HyperNNMS. Since the S-HyperTR-NNMS+ algorithm requires d dense networks to generate learning weights for the main network, where d is the number of factor tensors obtained by TR decomposition. Therefore, memory storage d*O(W) is required to store the learning weights of the hypernetwork of S-HyperTR-NNMS+. For the D-HyperTR-NNMS solution, the learning weights and the memory storage occupied by the learning weights are the same as those of S-HyperTR-NNMS+. In the D-TRHyper-NNMS+ and S-TRHyper-NNMS+ solutions, since the memory storage of the learning weights of the TR-NNMS+ algorithm is reused, no additional storage space is required to store the learning weights of the hypernetwork.

[0086] Table 1

[0087]

[0088] In the above, we only calculated the memory storage required for the learning weights of the hypernetwork. It should be pointed out that the addition of the hypernetwork, in addition to introducing additional learning weights, also includes activation functions that need to be calculated and stored. For the S-HyperNNMS algorithm, only one dense neural network is used as the hypernetwork. We assume that there are L activation functions in this dense network. We also assume that the computational complexity required for the L activation functions during training is O(L). For D-HyperNNMS, the hypernetwork unit consists of a dense neural network with L activation functions. Therefore, D-HyperNNMS contains a total of T*L activation functions, requiring a computational complexity of T*O(L), where T is the number of layers of the main NNMS network. Since the S-HyperTR-NNMS+ algorithm requires d dense networks, d is the number of factor tensors obtained by TR decomposition. Therefore, the S-HyperTR-NNMS+ algorithm requires dL activation functions, which requires a computational complexity of dO(L). Similar to the hypernetwork in D-HyperNNMS, the hypernetwork in D-HyperTR-NNMS+ requires dT*L activation functions, which requires a computational complexity of dT*O(L). Although the TRHyper solution reuses the storage space for learning weights, the computational complexity of the activation functions is not reduced. Similar to HyperNNMS, S-TRHyper requires L activation functions and a computational complexity of O(L), and D-TRHyper requires T*L activation functions and a computational complexity of T*O(L).

[0089] The above description is only a specific implementation of the present invention, but the protection scope of the present invention is not limited thereto. Any person familiar with the technology can understand and think of any changes or substitutions within the technical scope disclosed by the present invention, which should be included in the scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. A deep learning decoding method for LDPC codes based on a hypernetwork, characterized in that, the method combines a hypernetwork and an NNMS network, where the NNMS network is the main channel neural decoding network, and the hypernetwork generates learning weights for the main network; The hypernetwork consists of a dense neural network, and its input and the input of the main network are both the vector of log-likelihood ratio (LLR) values received by the channel; the hypernetwork is a static hypernetwork and generates the learning weights of the main network based on the following expression: in, It is a static hypernetwork The learning weights of is the message transmitted from CN node c to VN node v at time t, is the time from VN node v at time t-1 ′ For messages transmitted to CN node c, M(c) is the set of all VN nodes connected to CN, and M(c)\v is the set of VN nodes in M(c) except VN node v; Alternatively, the hypernetwork is a dynamic hypernetwork and generates the learning weights of the main network based on the following expression: Among them, W α is a dynamic hypernetwork (t-1) (:,W α ), is the message transmitted from CN node c to VN node v at time t, is the time from VN node v at time t-1 ε The message transmitted to CN node c, M(c) is the set of all VN nodes connected to CN, and M(c)\v is the set of VN nodes in M(c) except VN node v.

2. The deep learning decoding method for LDPC codes based on a hypernetwork according to claim 1, characterized in that, the tanh function is used as the activation function of the hypernetwork; under the guidance of the cross-entropy loss function, stochastic gradient descent (SGD) is used to train the hypernetwork.

3. The deep learning decoding method for LDPC codes based on a hypernetwork according to claim 1, characterized in that, the dynamic hypernetwork consists of T hypernetwork units corresponding one-to-one to each layer of the main network. Each hypernetwork unit generates learning weights for the corresponding layer of the main network. The input of the first hypernetwork unit is the vector of log-likelihood ratio (LLR) values received by the channel, and the input of the t-th hypernetwork unit is the output of the (t - 1)-th layer of the main network, where 1 < t ≤ T and T is the number of layers of the main network.

4. A deep learning decoding method for LDPC codes based on a hypernetwork, characterized in that, the method combines a hypernetwork and a TR-NNMS+ network, where the TR-NNMS+ network is the main channel neural decoding network, and the hypernetwork generates learning weights for the main network; The hypernetwork consists of a dense neural network, and its input and the input of the main network are both the vector of log-likelihood ratio (LLR) values received by the channel; the hypernetwork is a static hypernetwork. The number of layers of the static hypernetwork is the same as the number of factor tensors in the TR decomposition. Each layer of the static hypernetwork generates a learning weight tensor for the corresponding factor tensor, and the size of the learning weight tensor of the static hypernetwork is the same as the size of the factor tensor in the TR decomposition; the input of the static hypernetwork is the LLR received by the channel or the information tensor output by the previous layer of the main network; Alternatively, the hypernetwork is a dynamic hypernetwork consisting of T hypernetwork units corresponding one-to-one to each layer of the main network. Each hypernetwork unit generates learning weights for the corresponding layer of the main network. The input of the first hypernetwork unit is the vector of log-likelihood ratio (LLR) values received by the channel, and the input of the t-th hypernetwork unit is the output of the (t - 1)-th layer of the main network, and the rest is the same as the static hypernetwork, where 1 < t ≤ T and T is the number of layers of the main network.

5. The deep learning decoding method for LDPC codes based on a hypernetwork according to claim 4, characterized in that, an activation function is added between the learning weight tensors to form a static hypernetwork with m layers.

Images