A pulse neural network word selection method and device based on Bayesian posterior probability and a computer readable storage medium

Abstract

The application relates to the field of artificial intelligence and natural language processing technology, and discloses a pulse neural network word selection method and device based on Bayesian posterior probability and a computer readable storage medium. The method replaces the word selection mechanism based on a normalized exponential function in a traditional language model, introduces a learnable logarithmic prior probability vector in an output layer, performs Bayesian fusion on a context conditional likelihood and a word frequency prior through logarithmic domain addition, and generates a posterior probability distribution. Compared with a traditional normalized exponential function output, the method can explicitly model and utilize word frequency prior information, control the sharpness of the posterior distribution through a temperature parameter in a decoding process, and limit a candidate space through a top K truncation. The method is suitable for the output layer of a pulse neural network and a traditional deep neural network.

Description

Technical Field

[0001] This application relates to the fields of artificial intelligence and natural language processing technology, and in particular to a word selection method, apparatus and computer-readable storage medium based on a spiking neural network with Bayesian posterior probability. Background Technology

[0002] The output layer of current language models typically adopts the following structure: the representation vector of the last hidden layer is mapped to a vector of vocabulary size through linear projection, and then normalized to a probability distribution using a normalized exponential function. The mathematical expression of this structure is P(w|ctx) = normalized exponential function(X × Wᵀ), where X is the continuous representation of the hidden layer, and W is the projection weight matrix.

[0003] The structure has the following limitations: (1) It does not explicitly model word frequency priors - high-frequency words and low-frequency words enjoy the same baseline probability in the projection, and word frequency information can only be implicitly encoded through the training of W, which increases the learning burden; (2) The learning efficiency for low-frequency words is low - low-frequency words appear less frequently in the training data, and the updates of the corresponding rows in the projection matrix are sparse, making it difficult for the model to learn accurate conditional probabilities for low-frequency words; (3) The temperature and truncation strategies of the normalized exponential function are only used as post-processing during inference and do not participate in optimization during training.

[0004] Bayes' theorem provides a framework for explicitly fusing prior knowledge and observational evidence: P(w|ctx) ∝ P(ctx|w) × P(w), where P(w) is the prior probability (word frequency), P(ctx|w) is the likelihood (the support of the word in the context), and P(w|ctx) is the posterior probability (the true probability of the word in the given context).

[0005] Therefore, a method is needed to incorporate Bayesian priors into the output layer of a language model, enabling the model to explicitly utilize word frequency prior information and improve the discrimination between low-frequency and high-frequency words. Summary of the Invention

[0006] To address the aforementioned technical problems, this application provides a word selection method, apparatus, and computer-readable storage medium based on a spiking neural network using Bayesian posterior probability.

[0007] In a first aspect, this application provides a word selection method for spiking neural networks based on Bayesian posterior probability, applied to the output layer of a spiking neural network model. The method includes the following steps: Step 1, receiving a continuous representation tensor from the final processing layer of the spiking neural network; Step 2, likelihood calculation: mapping the continuous representation tensor to the word space through a linear projection matrix to obtain the log-likelihood value of each word; Step 3, prior fusion: performing element-wise addition of a preset learnable log-prior probability vector and the log-likelihood value obtained in Step 2 in the logarithmic domain to obtain the log-posterior probability of each word; Step 4, posterior decoding: performing temperature scaling and candidate space constraints on the log-posterior probability, normalizing it to a posterior probability distribution through a normalized exponential function, and sampling the output word from the distribution.

[0008] Secondly, this application provides a word selection device for a spiking neural network based on Bayesian posterior probability, comprising: a processor and a memory, wherein the memory stores a computer program, and the computer program, when executed by the processor, implements the method described in the first aspect; or, the device is applied to the output layer of a spiking neural network model, the device comprising: a receiving module configured to receive a continuous representation tensor from the final processing layer of the spiking neural network; a likelihood calculation module configured to map the continuous representation tensor to a word space through a linear projection matrix to obtain a log-likelihood value for each word; a prior fusion module configured to perform element-wise addition of a preset learnable log-prior probability vector and the log-likelihood value obtained by the likelihood calculation module in the logarithmic domain to obtain a log-posterior probability for each word; and a posterior decoding module configured to perform temperature scaling and candidate space constraints on the log-posterior probabilities, normalize them to a posterior probability distribution through a normalized exponential function, and sample output words from the distribution.

[0009] Thirdly, this application provides a computing device including a processor and a memory, wherein the memory stores a program, and the processor executes the program to implement the method described in the first aspect or any of its possible designs.

[0010] Fourthly, this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method described in the first aspect.

[0011] The method is based on the logarithmic domain transformation of the following Bayesian formula:

[0012] log P(w|ctx) = log P(ctx|w) + log P(w) − log P(ctx)

[0013] The normalization constant log P(ctx) is the same for all tokens and is canceled out in the normalization exponential function normalization, therefore:

[0014] Log-posterior probability (w) = Log-likelihood (w) + Log-prior probability (w)

[0015] That is, log-posterior = log-likelihood + log-prior.

[0016] Compared with the prior art, the beneficial effects of the present invention include:

[0017] (1) Explicit prior modeling. The learnable log-prior vector provides an independent frequency bias for each lexical, enabling the model to distinguish between two word selection motivations: “strong contextual support” and “selected because of high frequency”.

[0018] (2) Low additional cost. The additional parameter is only a V-dimensional vector (about 200 kilobytes when V=50257), which is negligible compared to the projection matrix W. The additional computation is only a single vector addition.

[0019] (3) Training stability. The zero-initialized prior vector does not affect the gradient direction of the loss function in the early stage of training. The prior bias gradually emerges as training progresses, without introducing training instability.

[0020] (4) Interpretability. After training converges, the prior vector can be checked to directly observe the word frequency bias learned by the model: a positive value indicates that the word is boosted by the prior, and a negative value indicates that it is suppressed by the prior.

[0021] (5) Compatible with early exit mechanism. In the information entropy gated early exit scenario, each exit layer shares the same Bayesian output layer, and the prior vector provides consistent word frequency correction in all exit layers. Attached Figure Description

[0022] Figure 1 The overall flowchart of the Bayesian word selection method provided in the embodiments of this application is shown.

[0023] Figure 2 This is a schematic diagram of logarithmic field addition for Bayesian fusion provided in an embodiment of this application.

[0024] Figure 3 The distribution histogram of the logarithmic prior vector after training, provided in the embodiments of this application.

[0025] Figure 4 The diagram shows the effect of temperature parameters on the sharpness of the posterior distribution, as provided in the embodiments of this application.

[0026] Figure 5 This is a schematic diagram illustrating the reduction of the candidate space by truncating the top K candidates, as provided in an embodiment of this application.

[0027] Figure 6 A comparison diagram of the Bayesian output layer and the normalized exponential function output layer provided in the embodiments of this application. Detailed Implementation

[0028] The technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings.

[0029] The following descriptions of the embodiments are for the purpose of helping to understand this application and do not limit the scope of protection of this application in any way.

[0030] Example 1: Complete Data Flow of Bayesian Word Selection

[0031] Reference Figure 1 This embodiment uses a vocabulary size of V=50257, an embedding dimension of D=768, a sequence length of T=256, and a batch size of B=8 as an example to describe the complete word selection data flow.

[0032] Step 1: Input tensor X ∈ R^{8×256×768}, the continuous representation output from the 12th sparse processing layer of a spiking neural network. This tensor undergoes 12 layers of sparse routing and pulse feedforward processing, encoding the semantic information of the input context. In the information entropy-gated early exit scenario, the continuous representation can come from any exit layer.

[0033] Step 2: Likelihood Projection LL = X × Wᵀ ∈ R^{8×256×50257}. The projection matrix W ∈ R^{50257×768} represents the weights of the unbiased linear layer, with approximately 38.6 million parameters (50257 × 768 ≈ 38.6 million). This projection maps the 768-dimensional semantic space to a 50257-dimensional vocabulary space. LL[b,t,w] represents the support of the context semantics for the word w at the t-th position of the b-th sample.

[0034] Step 3: Prior fusion log-posterior probability = LL + π[None, None, :]. π is a 50257-dimensional learnable parameter vector, initialized to all zeros, and broadcast in the shape (1, 1, 50257) to (8, 256, 50257). The computational cost of the addition operation is 8 × 256 × 50257 ≈ 103 million floating-point additions, which accounts for approximately 0.3% of the likelihood projection computation.

[0035] Step 4 (Inference Stage): Obtain the logarithmic posterior probability [:, -1, :] to get an (8, 50257) dimensional tensor. Temperature scaling τ=0.8. Truncate the top K terms to K=50, setting the logarithmic posterior probability of terms ranked 51 to 50257 to negative infinity. Normalize the exponential function to a probability distribution. Multinomial sampling yields the (8, 1) output term index.

[0036] Example 2: Mathematical Derivation of Bayesian Fusion

[0037] Reference Figure 2 This embodiment derives the mathematical equivalence of Bayesian fusion.

[0038] Bayes' Theorem: P(w|ctx) = P(ctx|w) × P(w) / P(ctx)

[0039] Taking the logarithm: log P(w|ctx) = log P(ctx|w) + log P(w) − log P(ctx)

[0040] For all lexical units w, the normalization constant log P(ctx) is the same. In the normalization exponential function normalization, the denominator Σ_w exp(log probability value (w)) is automatically normalized, and the log P(ctx) term is canceled out. Therefore:

[0041] Log-posterior probability(w) = log P(ctx|w) + log P(w) = LL(w) + π(w)

[0042] LL(w) is implicitly parameterized by the projection matrix W (the w-th row of W encodes the semantic representation of the word w), and π(w) is explicitly parameterized by independent prior vectors.

[0043] Comparison with the output layer of the standard normalized exponential function: Comparison items Standard output layer Bayesian output layer of the present invention Prior modeling None (implicitly encoded in W) Explicit (independent π vector) Output formula Normalized exponential function (X × Wᵀ) Normalized exponential function (X × Wᵀ + π) Additional parameters 0 V vectors (one word size) Low-frequency word processing Depends on W's generalization ability π provides frequency correction Training methods Only W is associated with the loss function W and π are optimized together.

[0044] Cross-entropy loss during training: L = −log P(w_target|ctx) = −log normalized exponential function (log-posterior probability) [w_target]. The gradient flows simultaneously to W (updating the likelihood model) and π (updating the prior bias), achieving end-to-end Bayesian optimization.

[0045] Example 3: Training behavior of logarithmic prior vectors

[0046] Reference Figure 3 This embodiment illustrates the evolution of the π vector during the training process.

[0047] In the initial training phase: π is all zero, and the Bayesian output layer is equivalent to the standard normalized exponential function output layer. The model behaves exactly the same as without prior knowledge, ensuring stability at the start of training.

[0048] Mid-training: π begins to differentiate. The π values ​​of common function words (such as "the", "is", "of") rise to the positive range (+0.5 to +2.0), reflecting the high frequency of these word units in the training corpus. The π values ​​of rare proper nouns fall to the negative range (-1.0 to -3.0).

[0049] In the later stages of training: π tends to stabilize. Its distribution approximates the logarithm of word frequencies in the training corpus, i.e., π(w) ≈ log f(w), where f(w) is the normalized frequency of word w in the training corpus.

[0050] This evolution process shows that the π vector automatically learns the word frequency distribution of the training corpus as prior knowledge, without the need for manual setting.

[0051] Example 4: Temperature scaling and top K truncation

[0052] Reference Figure 4 and Figure 5 This embodiment provides a detailed description of the posterior decoding strategy during the inference phase.

[0053] Temperature scaling: Scaled log-posterior probability = log-posterior probability / τ. τ value Effect Applicable Scenarios 0.1 Extremely sharp, almost like greedy decoding Fact-based question and answer (requires high certainty) 0.5 More sharp, tending towards high-probability word groups Translation and abstract (must be faithful to the original text) 0.8 Moderate (Recommended default value) General Dialogue 1.0 No scaling Benchmark comparison 1.5 Flatter surface, increasing diversity Creative Writing

[0054] Top K truncation: The K words with the highest log-posterior probabilities are retained, and the rest are set to negative infinity. When K=50, the top 50 words with the highest probabilities are selected from 50257 candidates. After truncation, normalization is performed only on these 50 candidates, and probabilities are redistributed to these 50 words. This strategy prevents the model from wasting sampling opportunities on long-tail, low-probability words while maintaining sufficient diversity.

[0055] Example 5: Processing during the training phase

[0056] This embodiment illustrates how the Bayesian output layer is used during the training phase.

[0057] During the training phase, the temperature scaling and top K truncation in step four are not performed. The log-posterior probability output from step three is directly used as input to the cross-entropy loss function: L = cross-entropy(log probability value, target word).

[0058] The cross-entropy loss function incorporates normalized exponential function normalization (i.e., log-normalized exponential function + negative log-likelihood loss). The gradient of the log probability value, ∂L / ∂log probability value, flows to W (through ∂LL / ∂W) and π (through ∂log probability value / ∂π = 1, the gradient is directly passed to π).

[0059] Since the gradient of π is ∂L / ∂π = normalized exponential function (log probability value) − one-hot encoding (target word), when a high-frequency word is frequently predicted correctly, its π value is pushed in the positive direction; when a low-frequency word is frequently predicted incorrectly, its π value is pushed in the negative direction. This automatic adjustment mechanism requires no manual intervention.

[0060] Example 6: Structure of a Bayesian word selection device

[0061] Corresponding to the spiking neural network word selection method based on Bayesian posterior probability described in Examples 1 to 5, this application also provides a spiking neural network word selection device based on Bayesian posterior probability. This device is applied to the output layer of a spiking neural network model and includes: a receiving module configured to receive a continuous representation tensor from the final processing layer of the spiking neural network; a likelihood calculation module configured to map the continuous representation tensor to a word space using a linear projection matrix to obtain the log-likelihood value of each word; a prior fusion module configured to perform element-wise addition of a preset learnable log-prior probability vector and the log-priority value obtained by the likelihood calculation module in the logarithmic domain to obtain the log-posterior probability of each word; and a posterior decoding module configured to perform temperature scaling and candidate space constraints on the log-posterior probabilities, normalize them to a posterior probability distribution using a normalized exponential function, and sample the output words from the distribution. The detailed execution logic of each of the above functional modules corresponds completely to steps 1 to 4 of the aforementioned method.

[0062] Example 7: Collaboration with Early Exit via Information Entropy Gating

[0063] Reference Figure 6 This embodiment illustrates the collaborative working method of the Bayesian output layer and the information entropy gating early exit mechanism.

[0064] In the adaptive computation depth mechanism, the output of each sparse processing layer generates an exit log-posterior probability through a Bayesian output layer. The information entropy gating module calculates the information entropy of this log-posterior probability distribution to determine whether to exit prematurely. Here, the Bayesian output layer is a single instance shared by the entire model, and all 12 exit decisions use the same set of projection matrix W and prior vector π.

[0065] The technical effects of this shared design are: (1) The prior vector π provides consistent word frequency correction in all exit layers, ensuring that the prior baseline of the output distribution is the same regardless of which layer to exit from; (2) When exiting from a shallow layer, the semantic information of the continuous representation is weak, and the frequency correction effect of the prior vector is relatively greater, making shallow exit more inclined to select high-frequency common words and reducing the risk of illusion; (3) When exiting from a deep layer, the semantic information of the continuous representation is sufficient, the likelihood term dominates the word selection decision, and the influence of the prior vector is relatively weakened.

[0066] Example 8: Application Scenario Description

[0067] Application Scenario 1: Generative Dialogue. In chatbot applications, the prior bias of the Bayesian output layer makes the model more inclined to generate natural, high-frequency expressions in uncertain contexts, reducing the unexpected appearance of obscure words. Recommended temperature τ=0.8, K=50.

[0068] Application Scenario 2: Specialized Domain Output. In specialized fields such as law or medicine, fine-tuning the π vector enhances the prior knowledge of specialized terminology, making the model more inclined to use validated professional vocabulary. π can be fine-tuned independently while W is frozen, allowing for adaptation to different specialized domains at extremely low cost.

[0069] Application Scenario 3: Security Filtering. By setting the π value of harmful words to a very large negative number (such as -100), the generation of harmful content can be effectively suppressed without modifying the model weights. This method is more efficient than post-processing filtering because harmful words are probabilistically eliminated before sampling.

[0070] Example 9: Computing Device and Storage Medium

[0071] This method can be executed on a variety of computing devices, including but not limited to: general-purpose central processing units, graphics processing units, dedicated artificial intelligence accelerators (such as neuromorphic processors, spiking neural network accelerator chips), field-programmable gate arrays, cloud server clusters, edge computing devices (such as embedded development boards, industrial gateways), and mobile terminals (such as smartphones, tablets, and wearable devices). The computing device includes a processor and a memory, wherein the memory stores a computer program implementing the above method.

[0072] Those skilled in the art should understand that the above embodiments are merely illustrative of the technical solutions of the present invention and not intended to limit it. Modifications and equivalent substitutions to the above embodiments without departing from the spirit and scope of the present invention should fall within the protection scope of the present invention.

Claims

1. A word selection method based on a spiking neural network using Bayesian posterior probability, characterized in that, The method, applied to the output layer of a spiking neural network model, includes the following steps: Step 1: Receive the continuous representation tensors from the final processing layer of the spiking neural network; Step 2, Likelihood Calculation: The continuous representation tensor is mapped to the vocabulary space through a linear projection matrix to obtain the log-likelihood value of each word. Step 3, Prior Fusion: The preset learnable log prior probability vector and the log likelihood value obtained in Step 2 are added element-wise in the log domain to obtain the log posterior probability of each word. Step 4, Posterior Decoding: After applying temperature scaling and candidate space constraints to the logarithmic posterior probability, it is normalized to a posterior probability distribution using a normalized exponential function, and output tokens are sampled from the distribution.

2. The method as described in claim 1, characterized in that, The linear projection matrix mentioned in step two is an unbiased linear transformation; the continuous representation tensor is multiplied by the transpose of the linear projection matrix to calculate the log-likelihood matrix composed of the log-likelihood values ​​of each word under different samples and different time steps.

3. The method as described in claim 1, characterized in that, The learnable log-prior probability vector mentioned in step three is part of the model parameters and is optimized together with the linear projection matrix through backpropagation during training. The specific method of element-wise addition is to add the log-likelihood value of the target word in the given context to the log-prior value of the target word in the learnable log-prior probability vector to calculate the log-posterior probability of the target word.

4. The method as described in claim 3, characterized in that, The learnable logarithmic prior probability vector is initialized to an all-zero vector in the early stage of training, and the frequency bias of each word is adaptively learned during the training process; after training converges, the prior value of the position corresponding to the high-frequency word is positive, and the prior value of the position corresponding to the low-frequency word is negative.

5. The method as described in claim 1, characterized in that, The specific method of temperature scaling in step four is as follows: divide the logarithmic posterior probability by a temperature parameter that is greater than zero, and then perform normalization using a normalized exponential function; when the temperature parameter is less than one and greater than zero, the distribution tends to be sharp; when the temperature parameter is greater than one, the distribution tends to be flat; when the temperature parameter is equal to one, it is equivalent to no scaling.

6. The method as described in claim 1, characterized in that, The candidate space restriction in step four adopts a pre-preset number name truncation strategy: only the pre-preset number of words with the highest logarithmic posterior probability are retained as the candidate set, and the probability of the remaining words is set to negative infinity; the value limit of the pre-preset number is from one to the total number of words in the vocabulary.

7. The method as described in claim 1, characterized in that, The specific method for sampling from the posterior probability distribution in step four is multinomial sampling: weighted random sampling is performed with the posterior probability of each candidate word as the weight to obtain an output word.

8. The method as described in claim 1, characterized in that, The method directly uses the log-posterior probability output in step three as the input to the cross-entropy loss function during the training phase, without performing the temperature scaling and candidate space constraint in step four.

9. A word selection device based on a spiking neural network using Bayesian posterior probability, characterized in that, The device, applied to the output layer of a spiking neural network model, comprises: The receiving module is configured to receive continuous representation tensors from the final processing layer of the spiking neural network; The likelihood calculation module is configured to map the continuous representation tensor to the vocabulary space through a linear projection matrix to obtain the log-likelihood value of each word. The prior fusion module is configured to perform element-wise addition in the log domain of the preset learnable log prior probability vector and the log likelihood value obtained by the likelihood calculation module to obtain the log posterior probability of each word. The posterior decoding module is configured to perform temperature scaling and candidate space constraints on the logarithmic posterior probability, normalize it to a posterior probability distribution using a normalized exponential function, and sample the output tokens from the distribution.

10. A computing device comprising a processor and a memory, the memory storing a program, wherein the processor executes the program to implement the method as described in any one of claims 1 to 8.

11. A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method as described in any one of claims 1 to 8.

Images