A natural language processing method and system based on spiking neural network

Abstract

The present application discloses a natural language processing method and system based on a spiking neural network, which relates to the technical field of natural language processing. The method is based on a spiking neural network and a Transformer structure, and constructs a natural language processing model including an embedding layer, a multi-step pulse coding layer, several pulse Transformer layers and a prediction head connected in sequence, so as to fully tap the energy efficiency potential of pulse computing; then, based on the principle of maintaining a stable neuron pulse firing rate, the parameters in the natural language processing model are initialized; finally, based on a knowledge distillation algorithm, the knowledge in the artificial neural network model trained on the task set is distilled into the natural language processing model, making full use of the pre-training of the existing artificial neural network model, eliminating the need for a large-scale pre-training process from scratch, and effectively distilling the knowledge of the artificial neural network model into the spiking neural network model, thus avoiding the consumption of massive computing resources.

Description

Technical Field

[0001] The present application relates to the field of natural language processing technology, and in particular to a natural language processing method and system based on a pulse neural network. Background Art

[0002] Natural Language Processing (NLP) is a key branch of artificial intelligence, encompassing tasks such as text classification, semantic analysis, machine translation, and conversational question-answering. These tasks often involve more advanced semantic understanding and reasoning, and are more complex than other tasks. Currently, the field of NLP primarily uses large language models (LLMs) to accomplish various tasks. However, these models consume significant computing resources, time, and energy, making such high training costs unacceptable in practical applications.

[0003] With the rise of large pre-trained models, artificial neural networks (ANNs) have reached new heights in natural language processing. However, as the complexity of the task increases, the network model structure also becomes increasingly complex and large, which in turn brings with it enormous computing power overhead. Computational resources and time costs have become a non-negligible issue in practical applications. Among them, the Transformer is a fundamental architecture of artificial neural networks. Its core is its unique self-attention mechanism, which allows the model to simultaneously consider the dependencies between all positions in the input sequence. It is composed of several identical layers stacked together, each of which contains two sublayers: a multi-head self-attention sublayer and a feed-forward fully connected network sublayer. Both sublayers have residual connections and layer normalization to stabilize training and optimize performance. However, due to the large number of floating-point matrix multiplication operations relied on by the Transformer computation process, training large language models requires massive computing resources, often involving the parallel use of thousands of GPUs or TPUs, consuming a large amount of time and energy. Such high training costs are often unacceptable. Furthermore, such a large-scale model consumes extremely high power consumption during inference, resulting in low inference efficiency. Therefore, it cannot be deployed on terminal devices with limited computing resources or in edge application scenarios, limiting its scope of application. Therefore, the field of natural language processing urgently needs to find a more efficient implementation method. Summary of the Invention

[0004] The purpose of this application is to provide a natural language processing method and system based on a spiking neural network, which can improve the Transformer architecture based on the spiking neural network, fully tap the energy efficiency potential of spiking computing, and save a lot of computing resources.

[0005] To achieve the above objectives, this application provides the following solutions:

[0006] In a first aspect, the present application provides a natural language processing method based on a spiking neural network, comprising the following steps:

[0007] A natural language processing model is constructed based on a spiking neural network and a Transformer model structure. The natural language processing model includes an embedding layer, a multi-step pulse coding layer, several pulse Transformer layers, and a prediction head, which are connected in sequence. The pulse Transformer layer includes several spiking neurons. The embedding layer is used to convert the natural language text input into the natural language processing model into a sentence vector. The multi-step pulse coding layer is used to convert the sentence vector output by the embedding layer into a pulse sequence. The pulse Transformer layer is used to extract feature information of the pulse sequence. The prediction head is used to predict and output the natural language processing result based on the feature information output by the final pulse Transformer layer.

[0008] Based on the principle of maintaining a stable neuronal pulse firing rate, the parameters in the natural language processing model are initialized.

[0009] Based on the knowledge distillation algorithm, the knowledge in the artificial neural network model trained on the task set is distilled into the natural language processing model to obtain a trained natural language processing model; the task set includes several natural language texts and corresponding natural language processing results.

[0010] Obtain the natural language text to be recognized, input it into the trained natural language processing model, and obtain the natural language processing result.

[0011] Optionally, the pulse Transformer layer includes a pulse attention module and a feedforward module, and both the pulse attention module and the feedforward module have a pulse residual connection.

[0012] Optionally, based on a knowledge distillation algorithm, knowledge in the artificial neural network model trained on the task set is distilled into the natural language processing model to obtain a trained natural language processing model, specifically comprising the following steps:

[0013] The Logits distillation loss function is determined according to the output distribution of the artificial neural network model and the natural language processing model trained on the task set.

[0014] The attention distillation loss function is determined based on the mean square error between the artificial neural network model trained on the task set and the attention matrix of the last layer of the natural language processing model.

[0015] Based on the Logits distillation loss function and the attention distillation loss function, a comprehensive distillation loss function is determined.

[0016] With the goal of minimizing the comprehensive distillation loss function, the parameters of the natural language processing model are optimized to obtain a trained natural language processing model.

[0017] In the second aspect, corresponding to the aforementioned natural language processing method based on a pulse neural network, the present invention also provides a natural language processing system based on a pulse neural network. When the natural language processing system based on a pulse neural network is run by a computer, it executes the natural language processing method based on a pulse neural network as described above.

[0018] According to the specific embodiments provided in this application, this application discloses the following technical effects:

[0019] The present application provides a natural language processing method and system based on a spiking neural network. Based on the spiking neural network and Transformer model structure, a natural language processing model is constructed, which includes an embedding layer, a multi-step pulse coding layer, several pulse Transformer layers and a prediction head connected in sequence. Based on the principle of maintaining a stable neuron pulse firing rate, the parameters in the natural language processing model are initialized. Finally, based on a knowledge distillation algorithm, the knowledge in the artificial neural network model trained on a task set is distilled into the natural language processing model. The above-mentioned scheme of the present application abandons the computing components incompatible with the spiking neural network, such as layer normalization and original self-attention mechanism in the traditional Transformer network structure, constructs a pure spiking Transformer architecture, fully exploits the energy efficiency potential of spiking computing, and also designs a multi-step pulse coding strategy for text data, as well as a parameter initialization method based on the principle of stable pulse firing rate. In addition, the knowledge obtained by training the artificial neural network model on the task set is effectively transferred to the natural language processing model, making full use of the pre-training of the existing artificial neural network model, eliminating the need for a large-scale pre-training process from scratch, and avoiding the consumption of massive computing resources. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0021] Figure 1 This is a diagram of the application environment of a natural language processing method based on a pulse neural network in one embodiment of the present application.

[0022] Figure 2 A flowchart of a natural language processing method based on a spiking neural network is provided in accordance with one embodiment of the present application.

[0023] Figure 3 A schematic diagram of the structure of a natural language processing model in a natural language processing method based on a pulse neural network provided in one embodiment of the present application.

[0024] Figure 4 This is a detailed flowchart of step S203 in a natural language processing method based on a spiking neural network provided in one embodiment of the present application.

[0025] Figure 5 A schematic diagram of the functional modules of a natural language processing system based on a spiking neural network provided in one embodiment of the present application.

[0026] Figure 6 A schematic diagram of the structure of a computer device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0027] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0028] In order to make the above-mentioned purposes, features and advantages of the present application more obvious and easy to understand, the present application is further described in detail below with reference to the accompanying drawings and specific implementation methods.

[0029] The natural language processing method based on pulse neural network provided in the embodiment of the present application can be applied to Figure 1In the application environment shown. Among them, the terminal 102 communicates with the server 104 through the network. The data storage system can store the data that the server 104 needs to process. The data storage system can be set up separately, integrated on the server 104, or placed on the cloud or other servers. The terminal 102 can send the natural language text to be processed to the server 104. After the server 104 receives the natural language text to be processed, for the natural language text to be processed, the server 104 constructs a natural language processing model based on the pulse neural network and Transformer model structure; based on the principle of maintaining the stability of the neuron pulse firing rate, the parameters in the natural language processing model are initialized; based on the knowledge distillation algorithm, the knowledge in the artificial neural network model trained on the task set is distilled into the natural language processing model to obtain a trained natural language processing model; the natural language text to be recognized is input into the trained natural language processing model to obtain a natural language processing result. The server 104 can feedback the obtained natural language processing result to the terminal 102. In addition, in some embodiments, the natural language processing method based on the pulse neural network can also be implemented independently by the server 104 or the terminal 102. For example, the terminal 102 can directly process the natural language text to be processed, or the server 104 can obtain the natural language text to be processed from the data storage system and process the natural language text to be processed.

[0030] Terminal 102 may include, but is not limited to, various desktop computers, laptops, smartphones, tablet computers, IoT devices, and portable wearable devices. IoT devices may include smart speakers, smart TVs, smart air conditioners, and smart car devices. Portable wearable devices may include smart watches, smart bracelets, and head-mounted devices. Server 104 may be implemented as a standalone server or a server cluster consisting of multiple servers, or may be a cloud server.

[0031] In an exemplary embodiment, Figure 2 The flowchart shown in FIG. 1 provides a natural language processing method based on a pulse neural network. The method is executed by a computer device, specifically a computer device such as a terminal or a server, or a terminal and a server. In the embodiment of the present application, the method is applied to Figure 1 Taking the server 104 in FIG. 1 as an example, the method includes the following steps S201 to S204:

[0032] S201, based on the pulse neural network and Transformer model structure, a natural language processing model is constructed. Figure 3The model structure diagram shown in the figure shows that the natural language processing model includes an embedding layer, a multi-step pulse coding layer, several pulse Transformer layers and a prediction head connected in sequence, and the pulse Transformer layer includes several pulse neurons; the embedding layer is used to convert the natural language text input into the natural language processing model into a sentence vector; the multi-step pulse coding layer is used to convert the sentence vector output by the embedding layer into a pulse sequence; the pulse Transformer layer is used to extract the feature information of the pulse sequence; the prediction head is used to predict and output the natural language processing result based on the feature information output by the pulse Transformer layer of the last layer.

[0033] S202. Based on the principle of maintaining a stable neuronal pulse firing rate, initialize the parameters in the natural language processing model.

[0034] S203, based on the knowledge distillation algorithm, distill the knowledge in the artificial neural network model trained on the task set into the natural language processing model to obtain a trained natural language processing model; the task set includes several natural language texts and corresponding natural language processing results; Figure 4 In the detailed flowchart shown, step S203 specifically includes the following steps:

[0035] S2031. Determine the Logits distillation loss function based on the output distribution of the artificial neural network model and the natural language processing model trained on the task set. The Logits distillation loss function is shown in the following formula:

[0036] L logits =Loss(z Teacher / t Teacher ,z Student / t Student ).

[0037] Among them, L logits is the Logits distillation loss function value, z Teacher is the output distribution of the artificial neural network model trained on the task set, t Teacher is the distillation temperature of the artificial neural network model trained on the task set, z Student is the output distribution of the natural language processing model, t Student is the distillation temperature of the natural language processing model.

[0038] S2032. Determine the attention distillation loss function based on the mean square error between the artificial neural network model trained on the task set and the attention matrix of the last layer of the natural language processing model. The attention distillation loss function is shown in the following formula:

[0039] L attention =MSE(A Teacher ,A Student ).

[0040] Among them, L attention is the attention distillation loss function value, T is the number of time steps.

[0041] S2033. Determine a comprehensive distillation loss function based on the Logits distillation loss function and the attention distillation loss function. The comprehensive distillation loss function is shown in the following formula:

[0042] L=(1-η)L logits +ηL attention .

[0043] Among them, L is the comprehensive distillation loss function value, and η is the weight of the attention distillation loss function.

[0044] S2034. Optimize the parameters of the natural language processing model with the goal of minimizing the comprehensive distillation loss function to obtain a trained natural language processing model.

[0045] S204: Obtain the natural language text to be recognized, input it into the trained natural language processing model, and obtain the natural language processing result.

[0046] In this embodiment, the pulse Transformer layer includes a pulse attention module and a feedforward module, and both the pulse attention module and the feedforward module have a pulse residual connection.

[0047] Specifically, the pulse attention module is as follows:

[0048] Attention=QK T V / d.

[0049] Where, Q = LIF (XW Q ),K=LIF(XW K ),V=XW V , X is the input pulse sequence, W Q 、W K and W V are all learnable weight parameters, Q is the information to be queried, K is the queried vector, V is the value obtained by the query, d is the size of the pulse attention module, T is the transpose, LIF() is the pulse neuron function, which represents a pulse neuron, and Attention is the output pulse attention.

[0050] The pulse residual connection is shown as follows:

[0051] X'=LIF(X+αSublayer(X)).

[0052] Where X,X'∈{0,1} s×d are the input and output in pulse form, and Sublayer(X) is the sublayer output in floating point form (the sublayer can be an attention layer, a feedforward layer, such as Figure 3 As shown), α is the scaling factor of the sub-layer output.

[0053] Spiking neurons perform spike calculations according to the following formula:

[0054]

[0055] in, is the membrane potential of the l-layer neuron at time t, τ is the membrane potential decay constant, W is the synaptic connection weight of the spiking neuron, is the pulse emitted by the neurons in layer l at time t, is a step function, u th is the pulse emission threshold.

[0056] The multi-step pulse coding layer performs multi-step pulse coding according to the following formula:

[0057]

[0058] Among them, S t ∈{0,1} s×d is the pulse sequence generated by the multi-step pulse coding layer at the tth time step, X BERT is the sentence vector output by the embedding layer, W t and b t are the weights and biases of multi-step pulse coding, respectively.

[0059] In this embodiment, the parameters in the natural language processing model are initialized according to the following formula:

[0060]

[0061] Among them, w ij is the synaptic connection weight between the j-th spiking neuron in the previous hidden layer and the i-th spiking neuron in the current hidden layer, N is the number of spiking neurons in the previous hidden layer, p represents the pulse firing rate of the spiking neurons in the previous hidden layer, k is the constant used to approximate the cumulative density function of the standard normal distribution, and u th is the pulse emission threshold, σ(w ij ) is the standard deviation of the weights.

[0062] In a specific embodiment, the spiking neurons in the natural language processing model constructed in step S201 are all implemented using LIF neurons. LIF (Leaky Integrate and Fire) neurons are the most widely used neuron model in spiking neural networks. Their computational process is relatively simple, yet they are highly flexible and easily adjustable to form different variants. The behavior of LIF neurons is described by the following four processes:

[0063] 1) Leak: When not stimulated, the membrane potential will gradually decrease until it reaches the resting potential.

[0064] 2) Integrate: accumulate impulse stimulation from upstream neurons.

[0065] 3) Fire: When the membrane potential reaches the pulse firing threshold, a pulse signal is generated.

[0066] 4) Reset: After the pulse is released, the membrane potential drops back down.

[0067] In different LIF neuron variants, the implementation methods of the above four processes are different. The basic LIF neuron used in this embodiment can be described by the following equations:

[0068]

[0069] in, represents the membrane potential of the l-layer neuron at time t, τ is the membrane potential decay constant, which makes the membrane potential decay exponentially; represents the pulse emitted by neurons in layer l at time t, and W is the synaptic connection weight; the above formula (1) describes the three processes of leakage, integration and reset, and the membrane potential is multiplied by After the pulse is released in the previous step, the membrane potential is reset to 0; the above formula (2) describes the pulse release process, where is a step function, when Greater than the pulse emission threshold U th A pulse signal is generated.

[0070] In the above LIF neuron model, the membrane potential decay constant τ and the pulse firing threshold u th are two adjustable parameters that control the behavior of LIF neurons. In many studies, researchers often use τ and u thSet as a learnable parameter and update along with training. In this embodiment, the membrane potential decay constant τ is set to be learnable. According to formula (1), τ controls the neuron's ability to remember previous stimuli. Neurons in different layers and modules learn their own values, which can make neuron behavior more heterogeneous, thus helping to improve the spatiotemporal dynamics of the spiking neural network. However, in this embodiment, the pulse emission threshold u is not set. th Set to learnable. Neuron membrane potential The changes come from Therefore, increasing u th and shrink are equivalent, so only learning W can achieve the same result. th It can be learned, but it is easy to affect the stability of the pulse emission rate in the early stage.

[0071] The General Language Understanding Evaluation (GLUE) benchmark includes nine tasks for evaluating natural language understanding models. GLUE is one of the most widely used evaluation standards in natural language processing. The GLUE dataset covers grammatical analysis, sentiment classification, semantic similarity analysis, and natural language inference tasks, focusing on measuring a model's natural language understanding capabilities. The GLUE dataset is divided into the following nine tasks:

[0072] CoLA: Syntax analysis task, which determines whether a short sentence is grammatical. The evaluation indicator is Matthews Correlation Coefficient.

[0073] SST-2: Sentiment classification task, which determines whether a movie review is positive or negative, and the evaluation indicator is accuracy.

[0074] MRPC: Determines whether two statements are semantically equivalent. Positive and negative samples are unbalanced, so the evaluation indicators are F1 value and accuracy.

[0075] QQP: Determines whether two questions are semantically equivalent. Positive and negative samples are also unbalanced. Evaluation indicators are F1 value and accuracy.

[0076] QNLI: Determine whether the given answer can answer the given question.

[0077] RTE: Determine whether two sentences are entailed.

[0078] WNLI: Co-reference resolution task, determining whether the second sentence refers to the correct pronoun in the first sentence.

[0079] MNLI: A three-category task that determines whether the premise and hypothesis are included, contradictory, or cannot be determined. The test is divided into two datasets: matched and mismatched. The matched dataset has the same source as the training set (i.e., similar style), while the mismatched dataset has a different source. The evaluation metric is the accuracy on both datasets.

[0080] STS-B: regression task, which scores the similarity between two sentences. The evaluation indicators are Pearson correlation coefficient and Spearman correlation coefficient.

[0081] Construct an SNN model with l=4 layers (the number of layers l indicates the number of Transformer blocks), hidden layer size d=192 (d indicates the dimension of the output vector of the embedding layer, which is also the dimension of the hidden layer vector in the model), and time step length T=4 (denoted as model A); and an SNN model with l=6 layers, hidden layer size d=384, and time step length T=16 (denoted as model B), and select MiniLMv2 (l=6, d=384) as the ANN model.

[0082] After initializing the parameters in each SNN model based on the principle of maintaining a stable neuronal pulse firing rate, knowledge distillation is first performed on the QQP and MNLI tasks. Knowledge distillation is then further performed on the other seven tasks. According to performance comparison, the SNN model designed in this application reduces power consumption by more than 70% while maintaining more than 98% of the performance of an ANN model of similar scale. This is because the pure pulse Transformer architecture fully exploits the energy efficiency potential of pulse computing, greatly reducing the power consumption of the model; at the same time, the multi-step pulse encoding strategy, the parameter initialization method based on the principle of stable pulse firing rate, and the ANN to SNN knowledge distillation method ensure sufficient performance of the model.

[0083] Table 1 shows the results of each model on various tasks in the GLUE benchmark test:

[0084] Table 1 Results of each model on various tasks of the GLUE benchmark

[0085] Model CoLA SST-2 MRPC QQP QNLI RTE WNLI MNLI STS-B BERTTINY 11.9 83.3 85.2 / 77.5 80.5 / 84.2 80.0 62.5 56.3 69.3 / 69.8 77.9 / 80.6 BERTMINI 32.2 85.3 85.4 / 78.2 83.1 / 87.2 84.5 64.6 56.3 75.3 / 76.4 85.4 / 85.9 MiniLMv2 35.3 90.5 90.1 / 86.3 86.1 / 89.5 90.1 72.2 56.3 82.4 / 83.0 88.1 / 88.0 Model A 19.0 84.4 85.9 / 80.2 81.2 / 85.7 82.2 63.9 60.6 71.7 / 72.5 83.1 / 83.0 Model B 28.1 89.2 89.2 / 84.8 85.2 / 88.9 86.7 70.4 64.8 78.8 / 79.5 86.9 / 86.9

[0086] Table 2 shows the number of model parameters, theoretical computing power consumption, and GLUE scores (averaged across tasks):

[0087] Table 2 Model parameters and GLUE scores

[0088]

[0089] Based on the same inventive concept, the natural language processing method based on the pulse neural network mentioned in the above embodiment can also be used with the help of Figure 5 The architecture of the natural language processing system based on the pulse neural network is implemented as shown in FIG. Figure 5 As shown, the natural language processing system based on the pulse neural network can include a natural language processing model construction module M1, a parameter initialization module M2, an ANN training module M3, a knowledge distillation module M4 and a natural language processing module M5; some modules can also have sub-units for implementing their functions, for example, the knowledge distillation module M4 also includes a Logits distillation loss function determination unit, an attention distillation loss function determination unit and a comprehensive distillation loss function determination unit. Of course, Figure 5 The architecture shown is only exemplary and can be omitted according to actual needs when implementing different functions. Figure 5 One or at least two components of the system shown.

[0090] In an exemplary embodiment, a computer device is provided. The computer device may be a server or a terminal. The internal structure diagram thereof may be as follows: Figure 6 As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, abbreviated as I / O) and a communication interface. The processor, memory and input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and computer program in the non-volatile storage medium. The database of the computer device is used to store natural language text and corresponding natural language processing results. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a natural language processing method based on a pulse neural network is implemented.

[0091] Those skilled in the art will understand that Figure 6 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0092] In an exemplary embodiment, a computer device is further provided, including a memory and a processor. The memory stores a computer program, and the processor implements the steps in the above method embodiments when executing the computer program.

[0093] In an exemplary embodiment, a computer-readable storage medium is provided, storing a computer program. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0094] In an exemplary embodiment, a computer program product is provided, including a computer program. When the computer program is executed by a processor, the steps in the above method embodiments are implemented.

[0095] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0096] Those skilled in the art will understand that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, database or other media used in the embodiments provided in this application may include at least one of non-volatile and volatile memory. Non-volatile memory may include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory may include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM may be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM).

[0097] The databases involved in the various embodiments provided herein may include at least one of a relational database and a non-relational database. Non-relational databases may include, but are not limited to, distributed databases based on blockchains. The processors involved in the various embodiments provided herein may include, but are not limited to, general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic units, data processing logic units based on quantum computing, and the like.

[0098] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0099] This document uses specific examples to illustrate the principles and implementation methods of this application. The description of the above examples is only intended to help understand the method and core concept of this application. At the same time, for those skilled in the art, based on the concept of this application, there may be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting this application.

Claims

1. A natural language processing method based on a spiking neural network, characterized in that: The natural language processing method based on the pulse neural network includes: A natural language processing model is constructed based on a pulse neural network and a Transformer model structure; the natural language processing model includes an embedding layer, a multi-step pulse coding layer, several pulse Transformer layers and a prediction head connected in sequence, and the pulse Transformer layer includes several pulse neurons; the embedding layer is used to convert the natural language text input into the natural language processing model into a sentence vector; the multi-step pulse coding layer is used to convert the sentence vector output by the embedding layer into a pulse sequence; the pulse Transformer layer is used to extract feature information of the pulse sequence; the prediction head is used to predict and output the natural language processing result based on the feature information output by the last layer of the pulse Transformer layer; Initializing the parameters of the natural language processing model based on the principle of maintaining a stable neuronal pulse firing rate; Based on the knowledge distillation algorithm, the knowledge in the artificial neural network model trained on the task set is distilled into the natural language processing model to obtain a trained natural language processing model; the task set includes a number of natural language texts and corresponding natural language processing results; Obtain the natural language text to be recognized, input it into the trained natural language processing model, and obtain the natural language processing result.

2. The natural language processing method based on pulse neural network according to claim 1, characterized in that: The pulse Transformer layer includes a pulse attention module and a feedforward module, and both the pulse attention module and the feedforward module have a pulse residual connection.

3. The natural language processing method based on pulse neural network according to claim 2, characterized in that: The pulse attention module is shown as follows: Where, Q = LIF (XW Q ),K=LIF(XW K ),V=XW V , X is the input pulse sequence, W Q 、W K and W V are all learnable weight parameters, Q is the information to be queried, K is the queried vector, V is the value obtained by the query, d is the size of the pulse attention module, T is the transpose, LIF() is the pulse neuron function, which represents a pulse neuron, and Attention is the output pulse attention.

4. The natural language processing method based on pulse neural network according to claim 3, characterized in that: The pulse residual connection is shown as follows: X'=LIF(X+αSublayer(X)); Where X,X'∈{0,1} s×d are the input and output in pulse form, Sublayer(X) is the sublayer output in floating point form, and α is the scaling factor of the sublayer output.

5. The natural language processing method based on pulse neural network according to claim 4, characterized in that: The spiking neuron performs pulse calculation according to the following formula: in, is the membrane potential of the l-layer neuron at time t, τ is the membrane potential decay constant, W is the synaptic connection weight of the spiking neuron, is the pulse emitted by the neurons in layer l at time t, is a step function, u th is the pulse emission threshold.

6. The natural language processing method based on pulse neural network according to claim 5, characterized in that: The multi-step pulse coding layer performs multi-step pulse coding according to the following formula: Among them, S t ∈{0,1} s×d is the pulse sequence generated by the multi-step pulse coding layer at the tth time step, X BERT is the sentence vector output by the embedding layer, W t and b t are the weights and biases of multi-step pulse coding, respectively.

7. The natural language processing method based on pulse neural network according to claim 6, characterized in that: Initialize the parameters in the natural language processing model according to the following formula: Among them, w ij is the synaptic connection weight between the j-th spiking neuron in the previous hidden layer and the i-th spiking neuron in the current hidden layer, N is the number of spiking neurons in the previous hidden layer, p represents the pulse firing rate of the spiking neurons in the previous hidden layer, k is the constant used to approximate the cumulative density function of the standard normal distribution, and u th is the pulse emission threshold.

8. The natural language processing method based on pulse neural network according to claim 7, characterized in that: Based on the knowledge distillation algorithm, the knowledge in the artificial neural network model trained on the task set is distilled into the natural language processing model to obtain a trained natural language processing model, specifically including: Determine a Logits distillation loss function based on the output distribution of the artificial neural network model and the natural language processing model trained on the task set; Determine the attention distillation loss function based on the mean square error between the artificial neural network model trained on the task set and the attention matrix of the last layer of the natural language processing model; Determine a comprehensive distillation loss function based on the Logits distillation loss function and the attention distillation loss function; With the goal of minimizing the comprehensive distillation loss function, the parameters of the natural language processing model are optimized to obtain a trained natural language processing model.

9. The natural language processing method based on pulse neural network according to claim 8, characterized in that: The Logits distillation loss function is shown below: L logits =Loss(z Teacher / t Teacher ,z Student / t Student ); Among them, L logits is the Logits distillation loss function value, z Teacher is the output distribution of the artificial neural network model trained on the task set, t Teacher is the distillation temperature of the artificial neural network model trained on the task set, z Student is the output distribution of the natural language processing model, t Student is the distillation temperature of the natural language processing model; The attention distillation loss function is shown as follows: L attention =MSE(A Teacher ,A Student ); Among them, L attention is the attention distillation loss function value, T is the number of time steps; The comprehensive distillation loss function is shown as follows: L=(1-n)L logits +ηL attention ; Among them, L is the comprehensive distillation loss function value, and η is the weight of the attention distillation loss function.

10. A natural language processing system based on a spiking neural network, characterized in that: When the natural language processing system based on the pulse neural network is run by a computer, it executes the natural language processing method based on the pulse neural network according to any one of claims 1 to 9.

Images