Pulse language model training method, text classification method and device

Abstract

The present invention provides a pulse language model training method, a text classification method and a device, which can be applied to the fields of data processing and computer technology. The pulse language model training method comprises: pre-training a pulse language model according to a predetermined corpus and a pre-trained predetermined teacher model to obtain a first pulse language model, wherein each layer structure included in the pulse language model includes M modules, each module includes a binary conversion unit, a pulse calculation unit and a normalization unit in sequence, the predetermined corpus includes a plurality of first training sentences, and M is a positive integer; fine-tuning a predetermined teacher model according to a predetermined training set to obtain a fine-tuning teacher model, wherein the predetermined training set includes a plurality of second training sentences; fine-tuning the first pulse language model according to the predetermined training set and the fine-tuning teacher model to obtain a target pulse language model.

Description

Technical Field

[0001] The present invention relates to the field of data processing and computer technology, and more specifically, to a pulse language model training method, a text classification method and a device. Background Art

[0002] With the development of artificial intelligence, the demand for computing power and resources of artificial neural networks has increased significantly. For example, the energy consumption of large language models in the training and reasoning stages has reached an incalculable level, which is the best example of this escalating demand. Spiking Neural Networks (SNNs) are called the third generation of neural networks. They have attracted widespread attention due to their specific high biological credibility, event-driven characteristics, and low power consumption when running on neuromorphic hardware. In related technologies, in order to achieve low-energy artificial intelligence, methods for peak computing based on neuromorphic hardware have been widely studied.

[0003] In the process of implementing the concept of the present invention, it is found that there are at least the following problems in the related art: the current pulse neural network with transformer as the core structure cannot be truly deployed in the neuromorphic chip. In the research of related pulse neural networks, a new pulse neural network model needs to be trained for each task, and the parameter adjustment efficiency of the model is very low. Summary of the invention

[0004] In view of this, the present invention provides a pulse language model training method, a text classification method and a device.

[0005] According to one aspect of the present invention, a pulse language model training method is provided, the method comprising:

[0006] Pre-training the pulse language model according to a predetermined corpus and a pre-trained predetermined teacher model to obtain a first pulse language model, wherein each layer structure included in the pulse language model includes M modules, each module includes a binary conversion unit, a pulse calculation unit and a normalization unit in sequence, the predetermined corpus includes a plurality of first training sentences, and M is a positive integer;

[0007] Fine-tune the predetermined teacher model according to a predetermined training set to obtain a fine-tuned teacher model, wherein the predetermined training set includes a plurality of second training sentences;

[0008] According to the predetermined training set and the fine-tuning teacher model, the first pulse language model is fine-tuned to obtain a target pulse language model.

[0009] According to another aspect of the present invention, a text classification method is provided, the method comprising:

[0010] The target sentence is classified using the target pulse language model obtained by the pulse language model training method to obtain a target classification result, wherein each layer structure included in the target pulse language model includes M modules, each module includes a binary conversion unit, a pulse calculation unit and a normalization unit in sequence, the predetermined corpus includes multiple first training sentences, and M is a positive integer.

[0011] According to another aspect of the present invention, a text classification device is provided, the device comprising:

[0012] A module is obtained, which is used to classify the target sentence using the target pulse language model obtained by the pulse language model training method to obtain a target classification result, wherein each layer structure included in the target pulse language model includes M modules, each module includes a binary conversion unit, a pulse calculation unit and a normalization unit in sequence, the predetermined corpus includes multiple first training sentences, and M is a positive integer.

[0013] According to the pulse language model training method provided by the present invention, since each layer structure included in the pulse language model includes M modules, each module includes a binary conversion unit, a pulse calculation unit and a normalization unit in sequence, so that the input of the pulse calculation unit in each module of the pulse language model is a binary number, thereby ensuring that the pulse language model can perform full pulse calculation, and the pulse language model can be truly deployed in a neuromorphic chip. By pre-training the pulse language model according to a predetermined corpus and a pre-trained predetermined teacher model, a first pulse language model is obtained, and general knowledge from a predetermined teacher model is quickly imparted to the pulse language model, and this knowledge is mainly related to sentence-level understanding. Then, according to a predetermined training set, the predetermined teacher model is fine-tuned to obtain a fine-tuned teacher model, and according to the predetermined training set and the fine-tuned teacher model, the first pulse language model is fine-tuned to obtain a target pulse language model, so that the task knowledge related to the feature classification task from the fine-tuned teacher model is quickly imparted to the first pulse language model, and the target pulse language model that has learned the task knowledge related to the feature classification task is quickly obtained. Moreover, for different classification tasks, there is no need to pre-train the pulse language model. You only need to use a predetermined training set to fine-tune the predetermined teacher model and the first pulse language model to obtain the target pulse language model, which greatly improves the training efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] The above and other objects, features and advantages of the present invention will become more apparent through the following description of the embodiments of the present invention with reference to the accompanying drawings, in which:

[0015] Figure 1 An application scenario diagram of a pulse language model training method and a text classification method according to an embodiment of the present invention is shown;

[0016] Figure 2 A flow chart of a pulse language model training method according to an embodiment of the present invention is shown;

[0017] Figure 3 A schematic diagram of a pulse language model used in a pulse language model training method according to an embodiment of the present invention is shown;

[0018] Figure 4 A schematic diagram showing a multi-task pulse adapter module included in a pulse language model according to an embodiment of the present invention;

[0019] Figure 5 A flowchart of a text classification method according to an embodiment of the present invention is shown;

[0020] Fig. 6A The target pulse language model obtained by the pulse language model training method according to an embodiment of the present invention is shown, in the process of classifying the first sentiment classification sentence, at the last time step of the attention map;

[0021] Figure 6B shows an attention map at the last time step of a target pulse language model obtained by the pulse language model training method according to an embodiment of the present invention, in the process of classifying a second sentiment classification sentence; and

[0022] Figure 7 A structural block diagram of a text classification device according to an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0023] Below, embodiments of the present invention will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the present invention. In the following detailed description, for ease of explanation, many specific details are set forth to provide a comprehensive understanding of embodiments of the present invention. However, it is apparent that one or more embodiments may also be implemented without these specific details. In addition, in the following description, descriptions of known structures and technologies are omitted to avoid unnecessary confusion of concepts of the present invention.

[0024] The terms used herein are only for describing specific embodiments and are not intended to limit the present invention. The terms "comprise", "include", etc. used herein indicate the existence of the features, steps, operations and / or components, but do not exclude the existence or addition of one or more other features, steps, operations or components.

[0025] All terms (including technical and scientific terms) used herein have the meanings commonly understood by those skilled in the art unless otherwise defined. It should be noted that the terms used herein should be interpreted as having a meaning consistent with the context of this specification and should not be interpreted in an idealized or overly rigid manner.

[0026] When using expressions such as "at least one of A, B, and C, etc.", they should generally be interpreted according to the meaning of the expression commonly understood by those skilled in the art (for example, "a system having at least one of A, B, and C" should include but is not limited to a system having A alone, B alone, C alone, A and B, A and C, B and C, and / or A, B, C, etc.).

[0027] In the technical solution of the present invention, the user information (including but not limited to user personal information, user image information, user device information, such as location information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved are all information and data authorized by the user or fully authorized by all parties, and the collection, storage, use, processing, transmission, provision, disclosure and application of the relevant data comply with relevant laws, regulations and standards, take necessary confidentiality measures, do not violate public order and good morals, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0028] Spiking neural networks use binary pulse signals to transmit and process information, which is usually weaker than traditional artificial neural networks in terms of effect. Therefore, designing a spiking neural network architecture with good performance is very valuable for research. For example, in the field of computer vision, due to the excellent performance of the transformer model on artificial neural networks, related technologies introduce the transformer architecture into spiking neural networks for image classification tasks.

[0029] The pulse neural network model used in the related art has the problem of non-full pulses in the calculation process. This problem mainly occurs in the model with the transformer as the core structure. The following will explain this problem based on formula (1) and formula (2).

[0030] (1)

[0031] (2)

[0032] in, Characterization The output of the residual connection block, Characterization The output of the residual connection block, Characterization The output of the residual connection block, Linear represents the pulse calculation unit, and BN represents the normalization unit. Characterization Binary conversion units, Characterization Binary conversion unit.

[0033] From formula (1), we can see that It can be binary pulse data or non-binary pulse data. For non-binary pulse data, The first When a residual connection block is connected, the pulse calculation unit Linear performs non-pulse calculation.

[0034] The mechanism of the spiking neural network is to use binary pulse signals to transmit and process information, using AC operations instead of MAC operations. It is precisely because of this that neuromorphic hardware does not support non-peak operations. Therefore, since the output data of the residual connection block of the spiking neural network in the related art contains non-binary data, the spiking neural network in the related art performs non-full pulse calculations, and the neuromorphic hardware does not support non-full pulse calculations. The current spiking neural network with transformer as the core structure cannot be truly deployed on neuromorphic chips.

[0035] Since the internal operations of the pulse neural network are not differentiable, it is impossible to use the traditional back-propagation training algorithm like the artificial neural network. Related technology, when training the pulse neural network, the following two common training methods are usually used: conversion from artificial neural network to pulse neural network and direct training through proxy gradient. Among them, the conversion-based method uses pulse neurons to replace the ReLU activation unit in the artificial neural network, and converts the artificial neural network to a pulse neural network through operations such as normalization and threshold balancing. However, the conversion is only an approximation and this operation requires the simulation of a large number of time steps, which not only leads to a decrease in the accuracy of the pulse neural network but also brings the problem of long conversion time. The direct training method through the proxy gradient expands the pulse neural network on the simulated time step, and trains the pulse neural network through the back-propagation algorithm over time. Compared with the conversion method, the direct training method has greatly improved the accuracy.

[0036] Related research focuses more on developing various spiking neural network model architectures to adapt to various fields. These spiking neural network models use traditional fine-tuning techniques throughout the training phase. However, an obvious disadvantage is that when faced with many downstream tasks, how to improve the parameter adjustment efficiency of spiking neural models is a common problem.

[0037] Research on related spiking neural networks usually requires training a new spiking neural network model for each task, which results in low parameter adjustment efficiency of the model. Since efficient parameter fine-tuning has been proven to improve the parameter efficiency of the model in artificial neural networks, it is a reasonable solution to transfer the methods that can improve the parameter adjustment efficiency in artificial neural networks to spiking neural networks. However, spiking neural networks are sparsely activated internally and use pulse signals to represent and transmit information, which is different from the working mechanism of artificial neural networks. Therefore, how to apply the efficient parameter fine-tuning methods in artificial neural networks to spiking neural networks is an urgent problem to be solved.

[0038] In order to at least partially solve the technical problems existing in the related art, the embodiments of the present invention provide a pulse language model training method, a text classification method and a device, which can be applied to the fields of data processing and computer technology.

[0039] An embodiment of the present invention provides a pulse language model training method, comprising: pre-training a pulse language model according to a predetermined corpus and a pre-trained predetermined teacher model to obtain a first pulse language model, wherein each layer structure included in the pulse language model includes M modules, each module includes a binary conversion unit, a pulse calculation unit and a normalization unit in sequence, the predetermined corpus includes a plurality of first training sentences, and M is a positive integer; fine-tuning a predetermined teacher model according to a predetermined training set to obtain a fine-tuned teacher model, wherein the predetermined training set includes a plurality of second training sentences; fine-tuning the first pulse language model according to the predetermined training set and the fine-tuning teacher model to obtain a target pulse language model.

[0040] Figure 1 An application scenario diagram of a pulse language model training method and a text classification method according to an embodiment of the present invention is shown.

[0041] like Figure 1 As shown, the application scenario 100 according to this embodiment may include a first terminal device 101, a second terminal device 102, a third terminal device 103, a network 104, and a server 105. The network 104 is used to provide a medium for a communication link between the first terminal device 101, the second terminal device 102, the third terminal device 103, and the server 105. The network 104 may include various connection types, such as wired, wireless communication links, or optical fiber cables, etc.

[0042] The user can use the first terminal device 101, the second terminal device 102, and the third terminal device 103 to interact with the server 105 through the network 104 to receive or send messages, etc. Various communication client applications can be installed on the first terminal device 101, the second terminal device 102, and the third terminal device 103, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social platform software, etc. (only for example).

[0043] The first terminal device 101, the second terminal device 102, and the third terminal device 103 may be various electronic devices having display screens and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, desktop computers, and the like.

[0044] The server 105 may be a server that provides various services, such as a background management server (only as an example) that provides support for websites browsed by users using the first terminal device 101, the second terminal device 102, and the third terminal device 103. The background management server may analyze and process the received data such as user requests, and feed back the processing results (such as web pages, information, or data obtained or generated according to user requests) to the terminal device.

[0045] It should be noted that the pulse language model training method and the text classification method provided in the embodiment of the present invention can generally be executed by the server 105. The pulse language model training method and the text classification method provided in the embodiment of the present invention can also be executed by a server or server cluster that is different from the server 105 and can communicate with the first terminal device 101, the second terminal device 102, the third terminal device 103 and / or the server 105.

[0046] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0047] The following will be based on Figure 1 The scenario described is used to describe the pulse language model training method and text classification method of the embodiments of the present invention in detail.

[0048] Figure 2 A flow chart of a pulse language model training method according to an embodiment of the present invention is shown.

[0049] like Figure 2 As shown, the pulse language model training method includes operations S201 to S203.

[0050] In operation S201, a pulse language model is pre-trained according to a predetermined corpus and a pre-trained predetermined teacher model to obtain a first pulse language model, wherein each layer structure included in the pulse language model includes M modules, each module includes a binary conversion unit, a pulse calculation unit and a normalization unit in sequence, the predetermined corpus includes multiple first training sentences, and M is a positive integer.

[0051] According to an embodiment of the present invention, a binary conversion unit can be used to perform binary conversion on the feature data of an input binary unit to obtain a binary feature, so as to ensure that the eigenvalues ​​included in the binary feature are all in binary form. The pulse calculation unit can be used to perform linear mapping on the binary feature to obtain a mapping feature. Since the eigenvalues ​​included in the binary feature are all in binary form, the input data of the input pulse calculation unit are all in binary form. Therefore, regardless of whether the weight parameters of the pulse calculation unit as a linear layer are in binary form, it can be ensured that the matrix multiplication operation is a peak operation, thereby ensuring the full pulse mode of the entire model. The normalization unit is used to normalize the mapping feature to obtain a normalized feature.

[0052] The process of processing input feature data by each module of each layer structure included in the pulse language model provided by the embodiment of the present invention is shown in formulas (3) and (4).

[0053] (3)

[0054] (4)

[0055] It can be seen from formula (3) and formula (4) that each module included in each layer structure of the pulse language model provided by the embodiment of the present invention includes: a binary conversion unit SN, a pulse calculation unit Linear and a normalization unit BN in sequence, so as to place the binary conversion unit SN (i.e., a unit that realizes the function of a pulse neuron) before the pulse calculation unit Linear (i.e., a linear layer), and ensure that in the process of processing feature data in each module, the feature data entering the pulse calculation unit are all activated by pulses in advance and are all in binary form. Therefore, regardless of whether the weight parameters of the linear layer are in binary form, it can be ensured that the matrix multiplication operation is a peak operation, thereby ensuring that the calculation process of the entire pulse language model is in full pulse mode. The pulse language model can be truly deployed in a neuromorphic chip.

[0056] According to an embodiment of the present invention, the predetermined corpus can be selected according to actual conditions, and is not limited here. For example, the predetermined corpus can be Chinese Wikipedia in the Chinese corpus, or Bookcorpus in the English corpus, or a subset of Chinese Wikipedia or Bookcorpus.

[0057] According to an embodiment of the present invention, the predetermined corpus representation may enable the pulse language model to learn a training set of general knowledge.

[0058] According to an embodiment of the present invention, the predetermined teacher model can be selected according to the actual situation, which is not limited here. For example, the predetermined teacher model can be a text classification model based on the self-attention mechanism Transformer or a model for text classification using a convolutional neural network. For example, the predetermined teacher model can be a BERT model or a TextCNN model.

[0059] According to the embodiment of the present invention, M can be selected according to actual conditions and is not limited here. For example, M can be 3, 4 or 5.

[0060] According to an embodiment of the present invention, the pulse language model may include multiple layer structures. For example, the pulse language model may include 24 layer structures, 50 layer structures, 64 layer structures, 100 layer structures, 200 layer structures, 300 layer structures or 384 layer structures.

[0061] According to an embodiment of the present invention, when pre-training the pulse language model, each first training sentence included in the predetermined corpus can be input into the predetermined teacher model and the pulse language model to obtain a first classification result output by the pulse language model and a second classification result output by the predetermined teacher model. Then, the pulse language model parameters are updated according to the first classification result and the second classification result to obtain the first pulse language model.

[0062] In operation S202, fine-tuning training is performed on a predetermined teacher model according to a predetermined training set to obtain a fine-tuned teacher model, wherein the predetermined training set includes a plurality of second training sentences.

[0063] According to an embodiment of the present invention, the predetermined training set representation can enable the first pulse language model and the predetermined teacher model to learn a training set of task knowledge corresponding to a specific text classification task.

[0064] According to an embodiment of the present invention, the predetermined training set can be selected according to actual conditions, and is not limited here. For example, the predetermined training set can be a user review dataset corresponding to various fields. For example, the predetermined training set can be a movie review dataset, a takeaway review dataset, or a hotel review dataset.

[0065] According to an embodiment of the present invention, the method for fine-tuning the predetermined teacher model can be selected according to actual conditions and is not limited here.

[0066] In operation S203, the first pulse language model is fine-tuned according to a predetermined training set and a fine-tuned teacher model to obtain a target pulse language model.

[0067] For example, when fine-tuning the first pulse language model, each second training sentence included in the predetermined training set can be input into the fine-tuning teacher model and the first pulse language model to obtain a third classification result output by the first pulse language model and a fourth classification result output by the fine-tuning teacher model. Then, the model parameters of the first pulse language model are fine-tuned according to the third classification result and the fourth classification result to obtain the target pulse language model.

[0068] According to the pulse language model training method provided by the present invention, since each layer structure included in the pulse language model includes M modules, each module includes a binary conversion unit, a pulse calculation unit and a normalization unit in sequence, so that the input of the pulse calculation unit in each module of the pulse language model is a binary number, thereby ensuring that the pulse language model can perform full pulse calculation, and the pulse language model can be truly deployed in a neuromorphic chip. By pre-training the pulse language model according to a predetermined corpus and a pre-trained predetermined teacher model, a first pulse language model is obtained, and general knowledge from a predetermined teacher model is quickly imparted to the pulse language model, and this knowledge is mainly related to sentence-level understanding. Then, according to a predetermined training set, the predetermined teacher model is fine-tuned to obtain a fine-tuned teacher model, and according to the predetermined training set and the fine-tuned teacher model, the first pulse language model is fine-tuned to obtain a target pulse language model, so that the task knowledge related to the feature classification task from the fine-tuned teacher model is quickly imparted to the first pulse language model, and the target pulse language model that has learned the task knowledge related to the feature classification task is quickly obtained. Moreover, for different classification tasks, there is no need to pre-train the pulse language model. You only need to use a predetermined training set to fine-tune the predetermined teacher model and the first pulse language model to obtain the target pulse language model, which greatly improves the training efficiency.

[0069] According to an embodiment of the present invention, a predetermined corpus can be first divided into N first training data sets, and then the pulse language model can be trained for multiple rounds using the N first training data sets. The pulse language model is trained once in each round using each first training data set, so that the pulse language model is trained N times in each round and the model parameters are updated N times.

[0070] According to an embodiment of the present invention, Figure 2 The operation S201 shown, pre-training the pulse language model according to the predetermined corpus and the pre-trained predetermined teacher model to obtain the first pulse language model, may include the following operations:

[0071] The pulse language model is iteratively trained as follows using N first training data sets corresponding to the predetermined corpus until the number of training rounds reaches a first predetermined number of rounds, thereby obtaining a first pulse language model:

[0072] In the i-th round of training, the pulse language model is iteratively trained as follows until the number of iterations is equal to N, and the i-th first pulse language model is obtained, where i is an integer greater than or equal to 1 and less than the first predetermined number of rounds:

[0073] For each first training sentence included in the j-th first training data set among the N first training data sets, the first training sentence is input into the pulse language model and the predetermined teacher model, the pulse language model outputs a first classification result, and the predetermined teacher model outputs a second classification result, wherein j is an integer greater than or equal to 1 and less than or equal to N;

[0074] Update the model parameters of the pulse language model according to the first classification results, the second classification results and the labels corresponding to the first training sentences included in the j-th first training data set;

[0075] Increment j, return the operation of inputting the first training sentence into the pulse language model and the predetermined teacher model, the pulse language model outputting the first classification result, and the predetermined teacher model outputting the second classification result for each first training sentence included in the j-th first training data set among the N first training data sets;

[0076] The i-th first pulse language model is determined as the pulse language model of the i+1th round, and the pulse language model is trained for the i+1th round.

[0077] According to an embodiment of the present invention, N and the first predetermined number of rounds can be selected according to actual conditions and are not limited here. For example, N can be 10, 25, 35, 50 or 100, etc. The first predetermined number of rounds can be 2, 5, 10, 35 or 50, etc.

[0078] According to an embodiment of the present invention, the number of first training sentences included in each first training data set can be selected according to actual conditions and is not limited here. For example, the number of first training sentences included in each first training data set can be 24, 50 or 100, etc.

[0079] According to an embodiment of the present invention, the pulse language model is iteratively trained as follows using N first training data sets corresponding to a predetermined corpus until the number of training rounds reaches a first predetermined number of rounds, thereby obtaining a first pulse language model: the pulse language model is iteratively trained as follows in the i-th round of training until the number of iterations is equal to N, thereby obtaining the i-th first pulse language model, and then the i-th first pulse language model is determined as the pulse language model of the i+1th round, and the pulse language model is trained for the i+1th round, thereby performing multiple rounds of training on the pulse language model according to the predetermined corpus, and updating the model parameters of the pulse language model multiple times in each round of training, thereby improving the prediction accuracy of the obtained first pulse language model.

[0080] Figure 3 A schematic diagram of a pulse language model used in a pulse language model training method according to an embodiment of the present invention is shown.

[0081] like Figure 3 As shown, the pulse language model 300 may include S layer structures. The S layer structures may be, for example, S encoders 310. The pulse language model 300 may be set to different numbers of layers. Generally speaking, the more layers there are, the larger the number of parameters of the model is, and the better the effect is. S may be, for example, 64.

[0082] Each layer structure includes M modules, where M can be 3, for example.

[0083] The M modules may include: a pulse self-attention module 311, a pulse adapter module 312 and a fully connected module 313.

[0084] According to an embodiment of the present invention, Figure 3 The pulse language model 300 in is denoted as ESpikingBERT and can be used Figure 2 The pulse language model training method shown in the figure trains ESpikingBERT. The entire training process is divided into two stages: pre-training stage and parameter efficient fine-tuning stage. In the pre-training stage, the pre-trained BERT can be used as the teacher model and ESpikingBERT as the student model, where the BERT model as the teacher is trained on a large-scale corpus. The main goal of this stage is to impart general knowledge from BERT to ESpikingBERT, which is mainly related to sentence-level understanding.

[0085] According to an embodiment of the present invention, for each first training sentence included in the j-th first training data set among the N first training data sets, the first training sentence is input into the pulse language model 300 and the predetermined teacher model, the pulse language model 300 outputs a first classification result, and the predetermined teacher model outputs a second classification result, including: when the first training sentence is input into the pulse language model 300,

[0086] Quantizing the first training sentence using the pulse language model 300 to obtain a quantized feature;

[0087] The quantized features are iteratively processed as follows using each layer structure included in the pulse language model 300 to obtain the target quantized features: for the s-th layer structure,

[0088] The pulse self-attention module 311 is used to process the quantized features after the s-1th layer structure is updated to obtain the self-attention features, where s is a positive integer. When s is 1, the quantized features after the s-1th layer structure is updated are the quantized features obtained by quantizing the first training sentence using the pulse language model 300;

[0089] Processing the first fused feature using the pulse adapter module 312 to obtain a pulse feature, wherein the first fused feature represents a feature obtained by fusing the self-attention feature and the quantization feature;

[0090] The second fused feature is fully connected by using the fully connected module 313 to obtain the quantized feature after the s-th layer structure is updated, wherein the second fused feature represents the feature obtained after the pulse feature and the first fused feature are fused;

[0091] The pulse language model 300 is used to classify the first training sentence according to the target quantitative feature to obtain a first classification result.

[0092] exist Figure 3 In the example, the first training sentence may be a word, and the quantitative feature may be a word embedding 301 .

[0093] exist Figure 3 In the embodiment, the second fused feature can be fully connected using the fully connected module 313 to obtain the fully connected quantized feature, and then the fully connected quantized feature and the first fused feature can be fused using the pulse language model 300 to obtain the quantized feature after the s-th layer structure is updated.

[0094] The target quantitative feature may be input into the classification head 320 included in the pulse language model 300, and the classification head 320 may be used to classify the first training sentence to obtain a first classification result.

[0095] like Figure 3 As shown, the pulse self-attention module 311 may sequentially include a binary conversion unit 3111 and a pulse self-attention layer 3112. The pulse self-attention layer 3112 may sequentially include a pulse calculation unit and a normalization unit.

[0096] The pulse adapter module 312 may include a first binary conversion unit 3121, a first pulse calculation unit 3122, a first normalization unit 3123, a second binary conversion unit 3124, a second pulse calculation unit 3125, and a second normalization unit 3126 in sequence.

[0097] The first pulse calculation unit 3122 may be a dimensionality reduction feedforward layer, and the second pulse calculation unit 3125 may be a dimensionality increase feedforward layer. The first normalization unit 3123 and the second normalization unit 3126 may both be batch normalization layers.

[0098] The fully connected module 313 may include a third binary conversion unit 3131 , a third pulse calculation unit 3132 , and a third normalization unit 3133 in sequence.

[0099] Among them, the third pulse calculation unit 3132 can be a feedforward layer, and the third normalization unit 3133 can be a batch normalization layer.

[0100] like Figure 3 It can be seen that inside each module of the pulse language model 300, the pulse neuron (i.e., binary conversion unit) is placed before the linear layer (i.e., pulse calculation unit), ensuring that in the process of feature data flow, the data entering the linear layer are all pulse activated in advance. Inside the linear layer, since the input data is in binary form, regardless of whether the weight parameters of the linear layer are in binary form, it can be guaranteed that the matrix multiplication operation is a peak operation, thereby ensuring that the calculation process of the entire pulse language model 300 is in full pulse mode. The pulse language model 300 can be truly deployed in a neuromorphic chip.

[0101] According to an embodiment of the present invention, the pulse adapter module 312 is used to process the first fusion feature to obtain the pulse feature including:

[0102] Using the first binary conversion unit 3121 to perform binary conversion on the first fusion feature to obtain a first binary feature;

[0103] Performing dimensionality reduction mapping on the first binary feature by using the first pulse calculation unit 3122 to obtain a dimensionality reduction feature;

[0104] The dimension reduction feature is normalized by the first normalization unit 3123 to obtain a first normalized feature;

[0105] Using the second binary conversion unit 3124 to perform binary conversion on the first normalized feature to obtain a second binary feature;

[0106] Performing dimension-up mapping on the second binary feature by using the second pulse calculation unit 3125 to obtain a dimension-up mapping feature;

[0107] The second normalization unit 3126 is used to normalize the increased-dimensional features to obtain pulse features.

[0108] According to an embodiment of the present invention, the second fusion feature is fully connected using the fully connected module 313 to obtain the quantized feature after the s-th layer structure is updated, including:

[0109] Using the binary conversion unit included in the fully connected module 313, that is, the third binary conversion unit 3131, the second fusion feature is binary converted to obtain a third binary feature;

[0110] Using the pulse calculation unit included in the fully connected module 313, that is, the third pulse calculation unit 3132, to perform full connection mapping on the third binary feature to obtain a fully connected feature;

[0111] Using the normalization unit included in the fully connected module 313, that is, the third normalization unit 3133, the fully connected features are normalized to obtain normalized features;

[0112] After fusing the normalized features and the first fusion features using the s-th layer structure, the updated quantitative features of the s-th layer structure are obtained.

[0113] According to an embodiment of the present invention, updating the model parameters of the pulse language model according to the first classification results, the second classification results, and the labels corresponding to the first training sentences included in the j-th first training data set includes:

[0114] Calculate feature distillation loss based on the first classification result, the second classification result and the label;

[0115] Calculate the masked language modeling loss based on the first classification result and label;

[0116] The feature distillation loss and the masked language modeling loss are weighted summed to obtain the pre-training loss value;

[0117] Update the model parameters of the pulse language model according to the pre-training loss value.

[0118] For example, according to formula (5), the feature distillation loss can be calculated based on the first classification result, the second classification result and the label; the masked language modeling loss can be calculated based on the first classification result and the label; the feature distillation loss and the masked language modeling loss can be weighted summed to obtain the pre-training loss value. .

[0119] (5)

[0120] in, represents the mean squared loss between the output features of the sth layer of the student model and the teacher model, Characterizes distillation loss, Representation Mask Language Modeling Loss, characterizes a first predetermined hyperparameter, Characterize a second predetermined hyperparameter.

[0121] According to an embodiment of the present invention, and You can choose according to the actual situation, and there is no limitation here. For example, and Both can be 1.0.

[0122] According to an embodiment of the present invention, in the stage of efficient parameter fine-tuning, BERT fine-tuned in a specific downstream task can be used as a teacher model, and ESpikingBERT after the pre-training stage can be used as a student model. In this stage, our goal is to let ESpikingBERT learn task knowledge on a specific task.

[0123] According to an embodiment of the present invention, Figure 2 Operation S203 shown, fine-tuning the first pulse language model according to the predetermined training set and the fine-tuning teacher model to obtain a target pulse language model, may include the following operations:

[0124] The first pulse language model is iteratively trained using L second training data sets corresponding to the predetermined training set until the number of training rounds reaches a second predetermined number of rounds, thereby obtaining a target pulse language model:

[0125] In the kth round of training, the first pulse language model is iteratively trained as follows until the number of iterations is equal to L, and the kth target pulse language model is obtained, where k is an integer greater than or equal to 1 and less than the second predetermined number of rounds:

[0126] For each second training sentence included in the p-th second training data set among the L second training data sets, the second training sentence is input into the first pulse language model and the fine-tuning teacher model, the first pulse language model outputs a third classification result, and the fine-tuning teacher model outputs a fourth classification result, wherein p is an integer greater than or equal to 1 and less than or equal to L;

[0127] Fine-tune the model parameters of the first pulse language model according to the third classification results, the fourth classification results, and the labels corresponding to each second training sentence included in the p-th second training data set;

[0128] Increment p, return the operation of inputting the second training sentence into the first pulse language model and the fine-tuning teacher model, the first pulse language model outputting the third classification result, and the fine-tuning teacher model outputting the fourth classification result for each second training sentence included in the p-th second training data set in the L second training data sets;

[0129] The kth target pulse language model is determined as the first pulse language model of the k+1th round, and the first pulse language model is trained for the k+1th round.

[0130] According to an embodiment of the present invention, L and the second predetermined number of rounds may be selected according to actual conditions and are not limited herein. For example, L may be 10, 25, 35, 50 or 100, etc. The second predetermined number of rounds may be 2, 5, 10, 35 or 50, etc.

[0131] According to an embodiment of the present invention, the number of second training sentences included in each second training data set can be selected according to actual conditions and is not limited here. For example, the number of second training sentences included in each second training data set can be 24, 50 or 100, etc.

[0132] According to an embodiment of the present invention, the following training is iteratively performed on the first pulse language model using L second training data sets corresponding to a predetermined training set until the number of training rounds reaches a second predetermined number of rounds, thereby obtaining a target pulse language model: in the kth round of training, the following training is iteratively performed on the first pulse language model until the number of iterations is equal to L, thereby obtaining the kth target pulse language model, and then the kth target pulse language model is determined as the first pulse language model for the k+1th round, and the first pulse language model is subjected to the k+1th round of training, thereby performing multiple rounds of training on the first pulse language model according to the predetermined training set, and fine-tuning the model parameters of the first pulse language model multiple times in each round of training, thereby improving the prediction accuracy of the obtained target pulse language model and improving the training speed.

[0133] According to an embodiment of the present invention, fine-tuning the model parameters of the first pulse language model according to the third classification results, the fourth classification results, and the labels corresponding to each second training sentence included in the pth second training data set includes:

[0134] According to the third classification results, the fourth classification results and the labels, the model embedding layer feature distillation loss, the hidden layer feature distillation loss and the prediction probability distillation loss are calculated;

[0135] Calculate the cross entropy loss based on the third classification result and label;

[0136] The model embedding layer feature distillation loss, hidden layer feature distillation loss, prediction probability distillation loss and cross entropy loss are weighted summed to obtain the fine-tuning loss value;

[0137] According to the fine-tuning loss value, model parameters of the first pulse language model are fine-tuned.

[0138] For example, according to formula (6), the model embedding layer feature distillation loss, hidden layer feature distillation loss and prediction probability distillation loss can be calculated based on the third classification result, the fourth classification result and the label; the cross entropy loss can be calculated based on the third classification result and the label; the model embedding layer feature distillation loss, the hidden layer feature distillation loss, the prediction probability distillation loss and the cross entropy loss are weighted summed to obtain the fine-tuning loss value. .

[0139] (6)

[0140] in, Characterize the model embedding layer feature distillation loss, Characterize the hidden layer feature distillation loss, Characterize the prediction probability distillation loss, Characterize the cross entropy loss, Characterizes the first coefficient, Characterizes the second coefficient, Characterize the third hyperparameter coefficient, Characterizes the fourth hyperparameter coefficient.

[0141] According to an embodiment of the present invention, You can choose according to the actual situation, and there is no limitation here. For example, Both can be 1.0.

[0142] According to an embodiment of the present invention, Figure 3 The pulse language model 300 in FIG. 1 may include a plurality of pulse adapter modules 312 to form a multi-task pulse adapter module. By replacing the multi-task pulse adapter module Figure 3 The pulse adapter module 312 in the multi-task pulse language model can be used for multi-task processing, and the replaced pulse language model can be named ESpikingBERT-MSA. Figure 4 shown.

[0143] Figure 4 A schematic diagram of a multi-task pulse adapter module included in a pulse language model according to an embodiment of the present invention is shown.

[0144] Figure 4 The multi-task pulse adapter module 412 in can enable the ESpikingBERT-MSA model to share information across tasks in a multi-task scenario.

[0145] like Figure 4 As shown, Figure 3 The multiple spike adapter modules 312 in the ESpikingBERT-MSA are combined together through the attention mechanism in the adapter fusion module 414 to form a multi-task spike adapter module 412. The multi-task spike adapter module 412 enables the entire ESpikingBERT-MSA model to perform multi-task learning.

[0146] When performing multi-task learning, the corresponding pulse adapter module 312 may be trained for each downstream task, and then the trained pulse adapter modules 312 may be extracted from the corresponding trained pulse language model 300. Finally, through the attention mechanism in the adapter fusion module 414, multiple trained pulse adapter modules 312 are combined to obtain a trained multi-task pulse adapter module 412.

[0147] Figure 5A flowchart of a text classification method according to an embodiment of the present invention is shown.

[0148] like Figure 5 As shown, the text classification method includes operation S501.

[0149] In operation S501, a target sentence is classified using a target pulse language model obtained by a pulse language model training method to obtain a target classification result, wherein each layer structure included in the target pulse language model includes M modules, each module includes a binary conversion unit, a pulse calculation unit and a normalization unit in sequence, the predetermined corpus includes multiple first training sentences, and M is a positive integer.

[0150] According to an embodiment of the present invention, the target sentence represents a sentence corresponding to a specific text classification task. The target sentence may be, for example, a movie review sentence, a takeaway review sentence, or a hotel review sentence.

[0151] According to an embodiment of the present invention, Figure 5 The target pulse language model used in Figure 2 The target pulse language model is obtained by training the pulse language model training method in . The target pulse language model structure can be Figure 3 The model structure in Figure 3 The model structure is obtained after the pulse adapter module 312 included in the target pulse language model is replaced by the pulse adapter module 312.

[0152] It should be noted that, unless it is explicitly stated that there is a sequence of execution between different operations shown in the flowchart in the embodiments of the present invention, or different operations have a sequence of execution in technical implementation, otherwise, the execution order of multiple operations may not be particular, and multiple operations may also be executed simultaneously.

[0153] The following will demonstrate the classification results of the target sentence by the target pulse language model trained by the pulse language model training method provided in the embodiment of the present invention based on actual experimental results.

[0154] In the experiment, during the pre-training phase of the model, the predetermined corpus was a large-scale text corpus. For Chinese, Chinese Wikipedia was used. For English, a subset of Bookcorpus was selected. For downstream task fine-tuning, 6 text classification data were used, including 4 English datasets MR, SST-2, Subj, and SST-5, and 2 Chinese datasets ChnSenti and Waimai. In the multi-task experiment, in order to expand the number of datasets, two additional English datasets Airline and AGNews were added. All the above data are open source data.

[0155] In terms of models, we used the open-source BERT-base-cased and Chinese-BERT-wwm-base as the teacher models for the English and Chinese datasets, respectively, and implemented them with the help of the HuggingFace transformers library. The SpikingBERT implementation uses the SpikingJelly framework. During the training phase, we used four NVIDIA A100-PCIE GPUs. Each dataset was trained for two rounds.

[0156] For the baseline model, selection is made according to two fields. In the field of artificial neural networks, the lightweight and efficient TextCNN model and the widely used BERT model are selected for comparison. In the field of spiking neural networks, SpikeBERT, which performs best on text classification tasks, and the SNN-TextCNN model are selected for comparison. For the method proposed in this invention, two settings are tested: a fixed spiking adapter size and a task-specific optimal spiking adapter size.

[0157] Table 1 shows the experimental results of classifying six Chinese and English data sets using the target pulse language model obtained by the pulse language model training method provided by an embodiment of the present invention when classifying a single task, that is, the classification accuracy results.

[0158] In Table 1, Avg represents the average value, Ours (64) represents the ESpikingBERT including a 64-layer structure obtained by the present invention, and Ours (24-384) represents the optimal model structure in the ESpikingBERT including a 24-384-layer structure obtained by the present invention.

[0159] Table 1

[0160]

[0161] The results in Table 1 show that compared with the pulse neural network baseline model, the target pulse language model (ESpikingBERT) obtained by the pulse language model training method of the embodiment of the present invention achieves state-of-the-art performance on six datasets. It is 4.5% higher than SNN-TextCNN on average and 1.01% higher than SpikeBERT on average. However, ESpikingBERT only needs to fine-tune about 18.37% of the parameters. In addition, the performance on the Chinese datasets (ChnSenti and Waimai) is equally outstanding.

[0162] In the experiment, the cross-task information sharing capability of ESpikingBERT was further studied. For example, the classification results of multiple tasks of the ESpikingBERT-MSA model were obtained when the pulse-pulse adapter module in ESpikingBERT was replaced with a multi-task pulse adapter module including 64 pulse-pulse adapter modules. And the classification results of multiple tasks of the ESpikingBERT-MSA model were obtained when the pulse-pulse adapter module in ESpikingBERT was replaced with a multi-task pulse adapter module including the optimal number of pulse-pulse adapter modules between 24-384. The experimental results are shown in Table 2. Table 2 shows the experimental results of classifying six English data sets when classifying multiple tasks using the target pulse language model obtained by the pulse language model training method provided in an embodiment of the present invention, that is, the classification accuracy results.

[0163] Table 2

[0164]

[0165] Table 2 shows that ESpikingBERT-MSA outperforms ESpikingBERT on both adapter size configurations. This suggests that ESpikingBERT-MSA can achieve good cross-task information sharing, thereby improving single-task performance through multi-task learning in text processing. It is worth noting that this enhancement is task-specific, showing improvements in MR, SST-2, and Airline tasks.

[0166] In order to further demonstrate the effectiveness of the target pulse language model obtained by the pulse language model training method provided by the embodiment of the present invention through visualization, the attention map of ESpikingBERT at the last time step is shown in FIG6 .

[0167] Fig. 6A The target pulse language model obtained by the pulse language model training method according to an embodiment of the present invention is shown in the attention map at the last time step during the classification of the first sentiment classification sentence. Figure 6B The target pulse language model obtained by the pulse language model training method according to an embodiment of the present invention is shown in the attention map at the last time step during the classification of the second sentiment classification sentence.

[0168] exist Fig. 6A There are some obvious highlights in the positions of words with positive attributes, especially "wonderful" and "vivid", which have higher brightness compared with other words, indicating that they have received higher attention. Figure 6BThere are some obvious highlights in the positions of words with negative attributes. For example, the word "fake", which is associated with negative attributes, has a higher brightness compared to other words, indicating that it has received the most attention. It is worth noting that Fig. 6A and Figure 6B These highly-regarded words in the text play a key role in determining the overall sentiment of the text, whether positive or negative. Therefore, after efficient parameter fine-tuning, the ESpikingBERT model can well capture these words that are critical to evaluating classification semantics. This proves that the ESpikingBERT model obtained by the pulse language model training method of the embodiment of the present invention can handle text classification tasks well.

[0169] Based on the above text classification method, the present invention also provides a text classification device. Figure 7 The device is described in detail.

[0170] Figure 7 A structural block diagram of a text classification device according to an embodiment of the present invention is shown.

[0171] like Figure 7 As shown, the text classification device 700 of this embodiment includes an obtaining module 710.

[0172] A module 710 is obtained, which is used to classify the target sentence using the target pulse language model obtained by the pulse language model training method to obtain a target classification result, wherein each layer structure included in the target pulse language model includes M modules, each module includes a binary conversion unit, a pulse calculation unit and a normalization unit in sequence, the predetermined corpus includes multiple first training sentences, and M is a positive integer.

[0173] It should be noted that the text classification device part in the embodiment of the present invention corresponds to the text classification method part in the embodiment of the present invention. For the specific description corresponding to the text classification device part, reference can be made to the text classification method part, which will not be repeated here.

[0174] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present invention. In this regard, each box in the flow chart or block diagram can represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0175] It will be appreciated by those skilled in the art that the features described in the various embodiments of the present invention may be combined and / or combined in various ways, even if such combinations or combinations are not explicitly described in the present invention. In particular, without departing from the spirit and teachings of the present invention, the features described in the various embodiments of the present invention may be combined and / or combined in various ways. All of these combinations and / or combinations fall within the scope of the present invention.

[0176] The embodiments of the present invention have been described above. However, these embodiments are only for the purpose of illustration, and are not intended to limit the scope of the present invention. Although each embodiment has been described above, this does not mean that the measures in each embodiment cannot be advantageously used in combination. The scope of the present invention is limited by the attached embodiments and their equivalents. Without departing from the scope of the present invention, those skilled in the art may make various substitutions and modifications, which should all fall within the scope of the present invention.

Claims

1. A pulse language model training method, characterized in that: The method comprises: Pre-training a pulse language model according to a predetermined corpus and a pre-trained predetermined teacher model to obtain a first pulse language model, wherein each layer structure included in the pulse language model includes M modules, each module includes a binary conversion unit, a pulse calculation unit and a normalization unit in sequence, the predetermined corpus includes a plurality of first training sentences, and M is a positive integer; According to a predetermined training set, fine-tuning the predetermined teacher model to obtain a fine-tuned teacher model, wherein the predetermined training set includes a plurality of second training sentences; Fine-tune the first pulse language model according to the predetermined training set and the fine-tuning teacher model to obtain a target pulse language model; The step of pre-training the pulse language model according to the predetermined corpus and the pre-trained predetermined teacher model to obtain the first pulse language model comprises: The pulse language model is iteratively trained as follows using N first training data sets corresponding to the predetermined corpus until the number of training rounds reaches a first predetermined number of rounds, thereby obtaining the first pulse language model: In the i-th round of training, the pulse language model is iteratively trained as follows until the number of iterations is equal to N, and the i-th first pulse language model is obtained, where i is an integer greater than or equal to 1 and less than the first predetermined number of rounds: For each first training sentence included in the j-th first training data set among the N first training data sets, inputting the first training sentence into the pulse language model and the predetermined teacher model, the pulse language model outputs a first classification result, and the predetermined teacher model outputs a second classification result, wherein j is an integer greater than or equal to 1 and less than or equal to N; updating the model parameters of the pulse language model according to the first classification results, the second classification results and the labels corresponding to the first training sentences included in the j-th first training data set; Incrementing j, returning, for each first training sentence included in the j-th first training data set among the N first training data sets, inputting the first training sentence into the pulse language model and the predetermined teacher model, the pulse language model outputting a first classification result, and the predetermined teacher model outputting a second classification result; Determine the i-th first pulse language model as the pulse language model of the i+1th round, and perform the i+1th round of training on the pulse language model; The M modules include: a pulse self-attention module, a pulse adapter module and a fully connected module, and for each first training sentence included in the jth first training data set in the N first training data sets, the first training sentence is input into the pulse language model and the predetermined teacher model, the pulse language model outputs a first classification result, and the predetermined teacher model outputs a second classification result, including: when the first training sentence is input into the pulse language model, quantizing the first training sentence using the pulse language model to obtain a quantized feature; The quantized features are iteratively processed as follows using each layer structure included in the pulse language model to obtain a target quantized feature: for the s-th layer structure, Using the pulse self-attention module to process the quantized features after the s-1th layer structure is updated, to obtain a self-attention feature, wherein s is a positive integer, and when s is 1, the quantized features after the s-1th layer structure is updated are the quantized features obtained by quantizing the first training sentence using the pulse language model; Processing the first fused feature using the pulse adapter module to obtain a pulse feature, wherein the first fused feature represents a feature obtained by fusing the self-attention feature and the quantization feature; Using the fully connected module to fully connect the second fused feature, to obtain the quantized feature after the s-th layer structure is updated, wherein the second fused feature represents the feature obtained after the pulse feature and the first fused feature are fused; Using the pulse language model, classifying the first training sentence according to the target quantitative feature to obtain the first classification result; The pulse adapter module includes a first binary conversion unit, a second binary conversion unit, a first pulse calculation unit, a second pulse calculation unit, a first normalization unit and a second normalization unit. The pulse adapter module is used to process the first fusion feature to obtain the pulse feature, which includes: Using a first binary conversion unit to perform binary conversion on the first fusion feature to obtain a first binary feature; Using the first pulse calculation unit to perform dimensionality reduction mapping on the first binary feature to obtain a dimensionality reduction feature; Normalizing the dimension reduction feature by using the first normalization unit to obtain a first normalized feature; Using the second binary conversion unit to perform binary conversion on the first normalized feature to obtain a second binary feature; Using the second pulse calculation unit to perform dimension-up mapping on the second binary feature to obtain a dimension-up feature; Using the second normalization unit to normalize the dimension-increased feature to obtain the pulse feature; The method of using the fully connected module to fully connect the second fusion feature to obtain the quantized feature after the s-th layer structure is updated includes: Using the binary conversion unit included in the fully connected module to perform binary conversion on the second fusion feature to obtain a third binary feature; Performing full connection mapping on the third binary feature using the pulse calculation unit included in the full connection module to obtain a full connection feature; Normalizing the fully connected features using a normalization unit included in the fully connected module to obtain normalized features; After the normalized feature and the first fused feature are fused using the s-th layer structure, the updated quantized feature of the s-th layer structure is obtained.

2. The method according to claim 1, characterized in that The updating of the model parameters of the pulse language model according to the first classification results, the second classification results and the labels corresponding to the first training sentences included in the j-th first training data set comprises: Calculating feature distillation loss according to the first classification result, the second classification result, and the label; Calculating a masked language modeling loss based on the first classification result and the label; Performing a weighted summation on the feature distillation loss and the masked language modeling loss to obtain a pre-training loss value; According to the pre-training loss value, the model parameters of the pulse language model are updated.

3. The method according to claim 1, characterized in that The step of fine-tuning the first pulse language model according to the predetermined training set and the fine-tuning teacher model to obtain a target pulse language model comprises: The first pulse language model is iteratively trained using L second training data sets corresponding to the predetermined training set until the number of training rounds reaches a second predetermined number of rounds, thereby obtaining the target pulse language model: In the kth round of training, the first pulse language model is iteratively trained as follows until the number of iterations is equal to L, and a kth target pulse language model is obtained, where k is an integer greater than or equal to 1 and less than the second predetermined number of rounds: For each second training sentence included in the pth second training data set in the L second training data sets, the second training sentence is input into the first pulse language model and the fine-tuning teacher model, the first pulse language model outputs a third classification result, and the fine-tuning teacher model outputs a fourth classification result, wherein p is an integer greater than or equal to 1 and less than or equal to L. Fine-tune the model parameters of the first pulse language model according to the third classification results, the fourth classification results, and the labels corresponding to the second training sentences included in the p-th second training data set; Incrementing p, returning an operation of, for each second training sentence included in the pth second training data set among the L second training data sets, inputting the second training sentence into the first pulse language model and the fine-tuning teacher model, the first pulse language model outputting a third classification result, and the fine-tuning teacher model outputting a fourth classification result; The kth target pulse language model is determined as the first pulse language model of the k+1th round, and the first pulse language model is trained for the k+1th round.

4. The method according to claim 3, characterized in that The step of fine-tuning the model parameters of the first pulse language model according to the third classification results, the fourth classification results, and the labels corresponding to the second training sentences included in the pth second training data set comprises: Calculate the model embedding layer feature distillation loss, the hidden layer feature distillation loss and the prediction probability distillation loss according to the third classification result, the fourth classification result and the label; Calculating a cross entropy loss according to the third classification result and the label; Performing a weighted summation on the model embedding layer feature distillation loss, the hidden layer feature distillation loss, the prediction probability distillation loss, and the cross entropy loss to obtain a fine-tuning loss value; Fine-tune model parameters of the first pulse language model according to the fine-tuning loss value.

5. A text classification method, characterized in that: The method comprises: The target sentence is classified using the target pulse language model obtained by the pulse language model training method according to any one of claims 1 to 4 to obtain a target classification result, wherein each layer structure included in the target pulse language model includes M modules, each module includes a binary conversion unit, a pulse calculation unit and a normalization unit in sequence, the predetermined corpus includes multiple first training sentences, and M is a positive integer.

6. A text classification device, characterized in that: The device comprises: A module is obtained, which is used to classify a target sentence using a target pulse language model obtained by the pulse language model training method according to any one of claims 1 to 4 to obtain a target classification result, wherein each layer structure included in the target pulse language model includes M modules, each module includes a binary conversion unit, a pulse calculation unit and a normalization unit in sequence, the predetermined corpus includes multiple first training sentences, and M is a positive integer.

Images