Natural language processing method and system based on spiking neural network

By adopting a pulse neural network-based method in natural language processing, combining knowledge distillation and multi-step pulse coding strategies, the problem of high computing resources of large language models is solved, and efficient natural language processing on edge devices is achieved.

CN118917318AActive Publication Date: 2024-11-08ZHEJIANG UNIV

Patent Information

Application Number
CN202410950860.5
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-16
Publication Date
2024-11-08
Estimated Expiration
2044-07-16

AI Technical Summary

Technical Problem

Existing large language models are too expensive to compute resources and time in natural language processing, resulting in undeployment in end devices and edge application scenarios with limited computing resources.

Method used

Using a natural language processing method based on pulse neural network, a model including an embedded layer, a multi-step pulse coding layer and a pulse Transformer layer is constructed, combined with a knowledge distillation algorithm, and abandons layer normalization and self-attention mechanisms, and designs multi-step pulse coding. Strategy and parameter initialization method for stabilizing pulse discharge rate make full use of pre-training knowledge of artificial neural network models.

Benefits of technology

It significantly reduces the consumption of computing resources, improves inference efficiency, and reduces power consumption by more than 70%, making natural language processing more feasible on edge devices.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118917318A_ABST
    Figure CN118917318A_ABST
Patent Text Reader

Abstract

The invention discloses a natural language processing method and system based on a spiking neural network, and relates to the technical field of natural language processing, the method is based on the spiking neural network and a Transform structure, and a natural language processing model comprising an embedded layer, a multi-step pulse coding layer, a plurality of pulse Transform layers and a prediction head which are connected in sequence is constructed, the energy efficiency potential of pulse calculation is fully excavated; then, based on the principle of keeping the neuronal pulse emission rate stable, parameters in a natural language processing model are initialized, finally, based on a knowledge distillation algorithm, knowledge in an artificial neural network model trained on a task set is distilled into the natural language processing model, pre-training of an existing artificial neural network model is fully utilized, and therefore the neural network processing efficiency is improved. A large-scale pre-training process does not need to be carried out from zero, the knowledge of the artificial neural network model is effectively distilled to the spiking neural network model, and the consumption of massive computing resources is avoided.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the technical field of natural language processing, and in particular to a natural language processing method and system based on a pulse neural network. Background Art

[0002] Natural Language Processing (NLP) is an important branch in the field of artificial intelligence, involving tasks such as text classification, semantic analysis, machine translation, and dialogue question-answering. These tasks usually involve more advanced semantic understanding and reasoning, and are more complex than other tasks. At present, the field of natural language processing mainly uses large language models (LLM) to complete various tasks. However, such models use a lot of computing resources, consume a lot of time and energy, and such a large training cost is often unacceptable in actual applications.

[0003] With the rise of large pre-trained models, the performance of artificial neural networks (ANN) in natural language processing has reached new heights. However, with the complexity of the task, the structure of the network model has become more and more complex and large, which has brought about an extremely huge computing power overhead. Computational resources and time costs have become issues that cannot be ignored in practical applications. Among them, Transformer is an artificial neural network infrastructure. Its core is its unique self-attention mechanism, which allows the model to simultaneously consider the dependencies between all positions in the input sequence. It is composed of several identical layers stacked together, and each layer contains two sublayers: a multi-head self-attention sublayer and a feedforward fully connected network sublayer. Both sublayers have residual connections and layer normalization to stabilize training and optimize performance. However, due to the large number of floating-point matrix multiplication operations that the Transformer calculation process relies on, the training of large language models requires massive computing resources, usually involving the parallel use of thousands of GPUs or TPUs, which consumes a lot of time and energy. Such a large training cost is often unacceptable. Moreover, such a large-scale model also consumes extremely high power during reasoning, and its reasoning efficiency is low, so it cannot be deployed in terminal devices and edge application scenarios with limited computing resources, which limits its application scope. Therefore, the field of natural language processing urgently needs to find a more efficient implementation method. Summary of the invention

[0004] The purpose of this application is to provide a natural language processing method and system based on a pulse neural network, which can improve the Transformer architecture based on the pulse neural network, fully tap the energy efficiency potential of pulse computing, and save a lot of computing resources.

[0005] To achieve the above objectives, this application provides the following solutions:

[0006] In a first aspect, the present application provides a natural language processing method based on a spiking neural network, comprising the following steps:

[0007] Based on the pulse neural network and Transformer model structure, a natural language processing model is constructed. The natural language processing model includes an embedding layer, a multi-step pulse coding layer, several pulse Transformer layers and a prediction head connected in sequence, and the pulse Transformer layer includes several pulse neurons; the embedding layer is used to convert the natural language text input into the natural language processing model into a sentence vector; the multi-step pulse coding layer is used to convert the sentence vector output by the embedding layer into a pulse sequence; the pulse Transformer layer is used to extract the feature information of the pulse sequence; the prediction head is used to predict and output the natural language processing result according to the feature information output by the last layer of the pulse Transformer layer.

[0008] Based on the principle of keeping the neuron pulse firing rate stable, the parameters in the natural language processing model are initialized.

[0009] Based on the knowledge distillation algorithm, the knowledge in the artificial neural network model trained on the task set is distilled into the natural language processing model to obtain a trained natural language processing model; the task set includes several natural language texts and corresponding natural language processing results.

[0010] The natural language text to be recognized is obtained, and input into the trained natural language processing model to obtain the natural language processing result.

[0011] Optionally, the pulse Transformer layer includes a pulse attention module and a feedforward module, and both the pulse attention module and the feedforward module have a pulse residual connection.

[0012] Optionally, based on a knowledge distillation algorithm, knowledge in the artificial neural network model trained on the task set is distilled into the natural language processing model to obtain a trained natural language processing model, which specifically includes the following steps:

[0013] The Logits distillation loss function is determined according to the output distribution of the artificial neural network model and the natural language processing model trained on the task set.

[0014] The attention distillation loss function is determined according to the mean square error between the artificial neural network model trained on the task set and the last layer attention matrix of the natural language processing model.

[0015] Based on the Logits distillation loss function and the attention distillation loss function, a comprehensive distillation loss function is determined.

[0016] With the goal of minimizing the comprehensive distillation loss function, the parameters of the natural language processing model are optimized to obtain a trained natural language processing model.

[0017] In the second aspect, corresponding to the aforementioned natural language processing method based on pulse neural network, the present invention also provides a natural language processing system based on pulse neural network, and when the natural language processing system based on pulse neural network is run by a computer, it executes the natural language processing method based on pulse neural network as described above.

[0018] According to the specific embodiments provided in this application, this application discloses the following technical effects:

[0019] The present application provides a natural language processing method and system based on a spiking neural network. Based on the spiking neural network and Transformer model structure, a natural language processing model including an embedding layer, a multi-step pulse coding layer, a plurality of pulse Transformer layers and a prediction head connected in sequence is constructed. Based on the principle of maintaining a stable pulse firing rate of neurons, the parameters in the natural language processing model are initialized. Finally, based on a knowledge distillation algorithm, the knowledge in the artificial neural network model trained on a task set is distilled into the natural language processing model. The above scheme of the present application abandons the computing components incompatible with the spiking neural network, such as layer normalization and the original self-attention mechanism in the traditional Transformer network structure, constructs a pure spiking Transformer architecture, fully exploits the energy efficiency potential of spiking computing, and designs a multi-step pulse coding strategy for text data, as well as a parameter initialization method based on the principle of a stable pulse firing rate. In addition, the knowledge obtained by training the artificial neural network model on the task set is effectively transferred to the natural language processing model, making full use of the pre-training of the existing artificial neural network model, without the need to carry out a large-scale pre-training process from scratch, thereby avoiding the consumption of massive computing resources. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0021] Figure 1 This is an application environment diagram of a natural language processing method based on a pulse neural network in one embodiment of the present application.

[0022] Figure 2 A flowchart of a natural language processing method based on a spiking neural network is provided for one embodiment of the present application.

[0023] Figure 3 A schematic diagram of the structure of a natural language processing model in a natural language processing method based on a pulse neural network provided in one embodiment of the present application.

[0024] Figure 4 A detailed flowchart of step S203 in a natural language processing method based on a pulse neural network provided in one embodiment of the present application.

[0025] Figure 5 A schematic diagram of the functional modules of a natural language processing system based on a pulse neural network provided in one embodiment of the present application.

[0026] Figure 6 A schematic diagram of the structure of a computer device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0027] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0028] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, the present application is further described in detail below with reference to the accompanying drawings and specific implementation methods.

[0029] The natural language processing method based on pulse neural network provided in the embodiment of the present application can be applied to Figure 1In the application environment shown. Among them, the terminal 102 communicates with the server 104 through the network. The data storage system can store the data that the server 104 needs to process. The data storage system can be set up separately, integrated on the server 104, or placed on the cloud or other servers. The terminal 102 can send the natural language text to be processed to the server 104. After the server 104 receives the natural language text to be processed, for the natural language text to be processed, the server 104 constructs a natural language processing model based on the pulse neural network and the Transformer model structure; based on the principle of keeping the neuron pulse firing rate stable, the parameters in the natural language processing model are initialized; based on the knowledge distillation algorithm, the knowledge in the artificial neural network model trained on the task set is distilled into the natural language processing model to obtain a trained natural language processing model; the natural language text to be recognized is input into the trained natural language processing model to obtain a natural language processing result. The server 104 can feedback the obtained natural language processing result to the terminal 102. In addition, in some embodiments, the natural language processing method based on the pulse neural network can also be implemented independently by the server 104 or the terminal 102. For example, the terminal 102 can directly process the natural language text to be processed, or the server 104 can obtain the natural language text to be processed from the data storage system and process the natural language text to be processed.

[0030] The terminal 102 may be, but is not limited to, various desktop computers, laptop computers, smart phones, tablet computers, IoT devices, and portable wearable devices. The IoT devices may be smart speakers, smart TVs, smart air conditioners, smart vehicle-mounted devices, etc. The portable wearable devices may be smart watches, smart bracelets, head-mounted devices, etc. The server 104 may be implemented as an independent server or a server cluster consisting of multiple servers, or may be a cloud server.

[0031] In an exemplary embodiment, Figure 2 The flowchart shown in FIG. 1 provides a natural language processing method based on a pulse neural network. The method is executed by a computer device, and can be executed by a computer device such as a terminal or a server alone, or by a terminal and a server together. In the embodiment of the present application, the method is applied to Figure 1 The server 104 in the example is used as an example to illustrate, including the following steps S201 to S204:

[0032] S201. Based on the pulse neural network and Transformer model structure, a natural language processing model is constructed. Figure 3The schematic diagram of the model structure shown in the figure, the natural language processing model includes an embedding layer, a multi-step pulse coding layer, several pulse Transformer layers and a prediction head connected in sequence, and the pulse Transformer layer includes several pulse neurons; the embedding layer is used to convert the natural language text input into the natural language processing model into a sentence vector; the multi-step pulse coding layer is used to convert the sentence vector output by the embedding layer into a pulse sequence; the pulse Transformer layer is used to extract the feature information of the pulse sequence; the prediction head is used to predict and output the natural language processing result according to the feature information output by the last layer of the pulse Transformer layer.

[0033] S202. Based on the principle of maintaining a stable neuron pulse firing rate, initialize the parameters in the natural language processing model.

[0034] S203, based on the knowledge distillation algorithm, distill the knowledge in the artificial neural network model trained on the task set into the natural language processing model to obtain a trained natural language processing model; the task set includes a number of natural language texts and corresponding natural language processing results; Figure 4 In the detailed flowchart shown, step S203 specifically includes the following steps:

[0035] S2031. Determine the Logits distillation loss function according to the output distribution of the artificial neural network model and the natural language processing model trained on the task set. The Logits distillation loss function is shown in the following formula:

[0036] L logits = Loss(z Teacher / t Teacher ,z Student / t Student ).

[0037] Among them, L logits is the value of the Logits distillation loss function, z Teacher is the output distribution of the artificial neural network model trained on the task set, t Teacher is the distillation temperature of the artificial neural network model trained on the task set, z Student is the output distribution of the natural language processing model, t Student Distillation temperature for natural language processing models.

[0038] S2032. Determine the attention distillation loss function based on the mean square error between the artificial neural network model trained on the task set and the last layer attention matrix of the natural language processing model. The attention distillation loss function is shown in the following formula:

[0039] L attention =MSE(A Teacher ,A Student ).

[0040] Among them, L attention is the attention distillation loss function value, T is the number of time steps.

[0041] S2033. Determine a comprehensive distillation loss function based on the Logits distillation loss function and the attention distillation loss function. The comprehensive distillation loss function is shown in the following formula:

[0042] L=(1-η)L logits +ηL attention .

[0043] Among them, L is the value of the comprehensive distillation loss function, and η is the weight of the attention distillation loss function.

[0044] S2034. Optimize the parameters of the natural language processing model with the goal of minimizing the comprehensive distillation loss function to obtain a trained natural language processing model.

[0045] S204: Obtain the natural language text to be recognized, input it into the trained natural language processing model, and obtain the natural language processing result.

[0046] In this embodiment, the pulse Transformer layer includes a pulse attention module and a feedforward module, and both the pulse attention module and the feedforward module have a pulse residual connection.

[0047] Specifically, the pulse attention module is as follows:

[0048] Attention=QK T V / d.

[0049] Where, Q = LIF (XW Q ),K=LIF(XW K ),V=XW V , X is the input pulse sequence, W Q , W K and W V are all learnable weight parameters, Q is the information to be queried, K is the queried vector, V is the value obtained by the query, d is the size of the pulse attention module, T is the transpose, LIF() is the pulse neuron function, which means passing through a pulse neuron, and Attention is the output pulse attention.

[0050] The pulse residual connection is shown as follows:

[0051] X'=LIF(X+αSublayer(X)).

[0052] Where X,X'∈{0,1} s×d are the input and output in pulse form, respectively, and Sublayer(X) is the sublayer output in floating point form (the sublayer can be an attention layer, a feedforward layer, such as Figure 3 As shown), α is the scaling factor of the sub-layer output.

[0053] Spiking neurons perform spike calculations according to the following formula:

[0054]

[0055] in, is the membrane potential of the l-layer neuron at time t, τ is the membrane potential decay constant, W is the synaptic connection weight of the spiking neuron, is the pulse emitted by the neurons in layer l at time t, is a step function, u th is the pulse emission threshold.

[0056] The multi-step pulse coding layer performs multi-step pulse coding according to the following formula:

[0057]

[0058] Among them, S t ∈{0,1} s×d is the pulse sequence generated by the multi-step pulse coding layer at the tth time step, X BERT is the sentence vector output by the embedding layer, W t and b t are the weights and biases of multi-step pulse coding, respectively.

[0059] In this embodiment, the parameters in the natural language processing model are initialized according to the following formula:

[0060]

[0061] Among them, w ij is the synaptic connection weight between the jth spiking neuron in the previous hidden layer and the ith spiking neuron in the current hidden layer, N is the number of spiking neurons in the previous hidden layer, p represents the pulse firing rate of the spiking neurons in the previous hidden layer, k is the constant used to approximate the cumulative density function of the standard normal distribution, and u th is the pulse emission threshold, σ(w ij ) is the standard deviation of the weights.

[0062] In a specific embodiment, the spiking neurons in the natural language processing model constructed in step S201 are all implemented using LIF neurons. LIF (Leaky Integrate and Fire) neurons are the most widely used neuron model in spiking neural networks. Their calculation process is relatively simple, and at the same time they have strong flexibility and are easy to adjust to form different variants. The behavior of LIF neurons is described by the following four processes:

[0063] 1) Leak: When not stimulated, the membrane potential will gradually decrease to the resting potential.

[0064] 2) Integrate: accumulate impulse stimulation from upstream neurons.

[0065] 3) Fire: When the membrane potential reaches the pulse firing threshold, a pulse signal is generated.

[0066] 4) Reset: After the pulse is released, the membrane potential drops back down.

[0067] In different LIF neuron variants, the implementation methods of the above four processes are different. The basic LIF neuron used in this embodiment can be described by the following equation:

[0068]

[0069] in, represents the membrane potential of the l-layer neuron at time t, τ is the membrane potential decay constant, which makes the membrane potential decay exponentially; represents the pulse emitted by the l-layer neuron at time t, W is the synaptic connection weight; the above formula (1) describes the three processes of leakage, integration and reset, and the membrane potential is multiplied by After the pulse is released in the previous step, the membrane potential is reset to 0; the above formula (2) describes the pulse release process, where is a step function, when Greater than the pulse emission threshold U th A pulse signal is generated.

[0070] In the above LIF neuron model, the membrane potential decay constant τ and the pulse firing threshold u th are two adjustable parameters that control the behavior of LIF neurons. In many studies, researchers often use τ and u thSet as a learnable parameter and update it with training. In this embodiment, the membrane potential decay constant τ is set to be learnable. According to formula (1), τ controls the memory ability of neurons for previous stimuli. Neurons in different layers and modules learn their own values, which can make neuronal behavior more heterogeneous, thus helping to improve the spatiotemporal dynamics of the spiking neural network. However, in this embodiment, the pulse emission threshold u th Set to learnable. Neuron membrane potential The changes come from Therefore, increasing u th and Zoom out are equivalent, so only learning W can achieve the same result. th It can be learned, but it is easy to affect the stability of the pulse emission rate in the early stage.

[0071] General Language Understanding Evaluation (GLUE) is a benchmark that contains 9 tasks for evaluating natural language understanding models. GLUE is one of the most common evaluation standards in the field of natural language processing. The GLUE dataset covers grammatical analysis, sentiment classification, semantic similarity analysis, and natural language inference tasks, focusing on measuring a model's natural language understanding capabilities. The GLUE dataset is divided into the following 9 tasks:

[0072] CoLA: Syntax analysis task, which determines whether a short sentence is grammatical. The evaluation indicator is Matthews Correlation Coefficient.

[0073] SST-2: Sentiment classification task, judging whether the movie review is positive or negative, and the evaluation indicator is accuracy.

[0074] MRPC: Determines whether two statements are semantically equivalent. Positive and negative samples are unbalanced, so the evaluation indicators are F1 value and accuracy.

[0075] QQP: Determine whether two questions are semantically equivalent. Positive and negative samples are also unbalanced. The evaluation indicators are F1 value and accuracy.

[0076] QNLI: Determine whether the given answer can answer the given question.

[0077] RTE: Determine whether two sentences are entailed.

[0078] WNLI: Co-reference resolution task, determining whether the second sentence refers correctly to the pronoun in the first sentence.

[0079] MNLI: A three-category task to determine whether the premise and hypothesis are included, contradictory, or cannot be determined. The test is divided into two data sets: matched and mismatched. The matched set has the same source as the training set (i.e., similar style), while the mismatched set has a different source from the training set. The evaluation indicator is the accuracy on the two data sets.

[0080] STS-B: Regression task, which scores the similarity between two sentences. The evaluation indicators are Pearson correlation coefficient and Spearman correlation coefficient.

[0081] Construct an SNN model with l=4 layers (the number of layers l indicates the number of Transformer blocks), hidden layer size d=192 (d indicates the dimension of the output vector of the embedding layer, which is also the dimension of the hidden layer vector in the model), and time step length T=4 (denoted as model A); and an SNN model with l=6 layers, hidden layer size d=384, and time step length T=16 (denoted as model B), and select MiniLMv2 (l=6, d=384) as the ANN model.

[0082] After initializing the parameters in each SNN model based on the principle of maintaining a stable neuron pulse firing rate, knowledge distillation is first performed on the QQP and MNLI tasks. Then knowledge distillation is further performed on the other 7 tasks. According to the performance comparison, the SNN model designed in this application reduces power consumption by more than 70% while maintaining more than 98% of the performance of an ANN model of similar scale. This is because the pure pulse Transformer architecture fully exploits the energy efficiency potential of pulse computing, greatly reducing the power consumption of the model; at the same time, the multi-step pulse coding strategy, the parameter initialization method based on the principle of stable pulse firing rate, and the ANN to SNN knowledge distillation method ensure sufficient performance of the model.

[0083] Table 1 shows the results of each model in various tasks in the GLUE benchmark test:

[0084] Table 1 Results of each model on various tasks of the GLUE benchmark

[0085] Model CoLA SST-2 MRPC QQP QNLI RTE WNLI MNLI STS-B BERTTINY 11.9 83.3 85.2 / 77.5 80.5 / 84.2 80.0 62.5 56.3 69.3 / 69.8 77.9 / 80.6 BERTMINI 32.2 85.3 85.4 / 78.2 83.1 / 87.2 84.5 64.6 56.3 75.3 / 76.4 85.4 / 85.9 MiniLMv2 35.3 90.5 90.1 / 86.3 86.1 / 89.5 90.1 72.2 56.3 82.4 / 83.0 88.1 / 88.0 Model A 19.0 84.4 85.9 / 80.2 81.2 / 85.7 82.2 63.9 60.6 71.7 / 72.5 83.1 / 83.0 Model B 28.1 89.2 89.2 / 84.8 85.2 / 88.9 86.7 70.4 64.8 78.8 / 79.5 86.9 / 86.9

[0086] Table 2 gives the parameters of each model, theoretical computing power consumption and GLUE score (averaged from various tasks):

[0087] Table 2 Model parameters and GLUE scores

[0088]

[0089] Based on the same inventive concept, the natural language processing method based on the pulse neural network mentioned in the above embodiment can also be used Figure 5 The architecture of the natural language processing system based on the pulse neural network is shown in FIG. Figure 5 As shown, the natural language processing system based on the pulse neural network may include a natural language processing model building module M1, a parameter initialization module M2, an ANN training module M3, a knowledge distillation module M4 and a natural language processing module M5; some modules may also have sub-units for realizing their functions, for example, the knowledge distillation module M4 also includes a Logits distillation loss function determination unit, an attention distillation loss function determination unit and a comprehensive distillation loss function determination unit. Of course, Figure 5 The architecture shown is only exemplary and can be omitted according to actual needs when implementing different functions. Figure 5 One or at least two components of the system shown.

[0090] In an exemplary embodiment, a computer device is provided. The computer device may be a server or a terminal. The internal structure diagram thereof may be as follows: Figure 6 As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. Among them, the processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store natural language texts and corresponding natural language processing results. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a natural language processing method based on a pulse neural network is implemented.

[0091] Those skilled in the art will understand that Figure 6 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0092] In an exemplary embodiment, a computer device is further provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above-mentioned method embodiments when executing the computer program.

[0093] In an exemplary embodiment, a computer-readable storage medium is provided, storing a computer program, and when the computer program is executed by a processor, the steps in the above method embodiments are implemented.

[0094] In an exemplary embodiment, a computer program product is provided, including a computer program, and when the computer program is executed by a processor, the steps in the above method embodiments are implemented.

[0095] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0096] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM may be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM).

[0097] The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. The non-relational database may include a distributed database based on blockchain, etc., but is not limited thereto. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but is not limited thereto.

[0098] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0099] This article uses specific examples to illustrate the principles and implementation methods of this application. The description of the above embodiments is only used to help understand the method and core ideas of this application. At the same time, for those skilled in the art, according to the ideas of this application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting this application.

Claims

1. A natural language processing method based on a spiking neural network, characterized in that: The natural language processing method based on the pulse neural network includes: A natural language processing model is constructed based on a pulse neural network and a Transformer model structure; the natural language processing model includes an embedding layer, a multi-step pulse coding layer, a plurality of pulse Transformer layers and a prediction head connected in sequence, and the pulse Transformer layer includes a plurality of pulse neurons; the embedding layer is used to convert the natural language text input into the natural language processing model into a sentence vector; the multi-step pulse coding layer is used to convert the sentence vector output by the embedding layer into a pulse sequence; the pulse Transformer layer is used to extract feature information of the pulse sequence; the prediction head is used to predict and output a natural language processing result according to the feature information output by the last layer of the pulse Transformer layer; Initializing the parameters in the natural language processing model based on the principle of maintaining a stable neuron pulse firing rate; Based on the knowledge distillation algorithm, the knowledge in the artificial neural network model trained on the task set is distilled into the natural language processing model to obtain a trained natural language processing model; the task set includes a number of natural language texts and corresponding natural language processing results; The natural language text to be recognized is obtained, and input into the trained natural language processing model to obtain the natural language processing result.

2. The natural language processing method based on pulse neural network according to claim 1, characterized in that: The pulse Transformer layer includes a pulse attention module and a feedforward module, and both the pulse attention module and the feedforward module have a pulse residual connection.

3. The natural language processing method based on pulse neural network according to claim 2, characterized in that: The pulse attention module is shown as follows: Where, Q = LIF (XW Q ),K=LIF(XW K ),V=XW V , X is the input pulse sequence, W Q , W K and W V are all learnable weight parameters, Q is the information to be queried, K is the queried vector, V is the value obtained by the query, d is the size of the pulse attention module, T is the transpose, LIF() is the pulse neuron function, which means passing through a pulse neuron, and Attention is the output pulse attention.

4. The natural language processing method based on pulse neural network according to claim 3 is characterized in that: The pulse residual connection is shown as follows: X' = ​​LIF(X + αSublayer(X)); Where X,X'∈{0,1} s×d They are input and output in pulse form, Sublayer(X) is the sublayer output in floating point form, and α is the scaling factor of the sublayer output.

5. The natural language processing method based on pulse neural network according to claim 4 is characterized in that: The spiking neuron performs spiking calculations according to the following formula: in, is the membrane potential of the l-layer neuron at time t, τ is the membrane potential decay constant, W is the synaptic connection weight of the spiking neuron, is the pulse emitted by the neurons in layer l at time t, is a step function, u th is the pulse emission threshold.

6. The natural language processing method based on pulse neural network according to claim 5, characterized in that: The multi-step pulse coding layer performs multi-step pulse coding according to the following formula: Among them, S t ∈{0,1} s×d is the pulse sequence generated by the multi-step pulse coding layer at the tth time step, X BERT is the sentence vector output by the embedding layer, W t and b t are the weights and biases of multi-step pulse coding, respectively.

7. The natural language processing method based on pulse neural network according to claim 6, characterized in that: Initialize the parameters in the natural language processing model according to the following formula: Among them, w ij is the synaptic connection weight between the jth spiking neuron in the previous hidden layer and the ith spiking neuron in the current hidden layer, N is the number of spiking neurons in the previous hidden layer, p represents the pulse firing rate of the spiking neurons in the previous hidden layer, k is the constant used to approximate the cumulative density function of the standard normal distribution, and u th is the pulse emission threshold.

8. The natural language processing method based on pulse neural network according to claim 7, characterized in that: Based on the knowledge distillation algorithm, the knowledge in the artificial neural network model trained on the task set is distilled into the natural language processing model to obtain a trained natural language processing model, which specifically includes: Determine a Logits distillation loss function according to the output distribution of the artificial neural network model and the natural language processing model trained on the task set; Determining an attention distillation loss function according to a mean square error between an artificial neural network model trained on a task set and an attention matrix of a last layer of the natural language processing model; Determine a comprehensive distillation loss function based on the Logits distillation loss function and the attention distillation loss function; With the goal of minimizing the comprehensive distillation loss function, the parameters of the natural language processing model are optimized to obtain a trained natural language processing model.

9. The natural language processing method based on pulse neural network according to claim 8, characterized in that: The Logits distillation loss function is as follows: L logits =Loss(z Teacher / t Teacher ,z Student / t Student ); Among them, L logits is the value of the Logits distillation loss function, z Teacher is the output distribution of the artificial neural network model trained on the task set, t Teacher is the distillation temperature of the artificial neural network model trained on the task set, z Student is the output distribution of the natural language processing model, t Student Distillation temperature for natural language processing models; The attention distillation loss function is shown as follows: L attention =MSE(A Teacher ,A Student ); Among them, L attention is the attention distillation loss function value, T is the number of time steps; The comprehensive distillation loss function is shown as follows: L=(1-η)L logits +ηL attention ; Among them, L is the value of the comprehensive distillation loss function, and η is the weight of the attention distillation loss function.

10. A natural language processing system based on a spiking neural network, characterized in that: When the natural language processing system based on the pulse neural network is run by a computer, it executes the natural language processing method based on the pulse neural network as described in any one of claims 1 to 9.

Citation Information

Patent Citations

  • Text processing method and device

    CN114548046A

  • Method and system for training a neural network model using gradual knowledge distillation

    WO2022051855A1

  • KR20230134809A

Cited By

  • Natural language processing method and device based on hybrid neural network model, equipment and medium

    CN120409454A

  • Action method and device based on pulse body model

    CN121798646A

  • Motion Method and Device Based on Pulse-Based Large-Scale Body Model

    CN121798646B