Natural language processing method and device based on hybrid neural network model, equipment and medium

Abstract

The invention discloses a natural language processing method and device based on a hybrid neural network model, equipment and a medium, and belongs to the technical field of computers, and the method comprises the steps: obtaining a to-be-processed natural language text; inputting a natural language text to be processed into the hybrid neural network model to obtain a natural language processing result output by the hybrid neural network model; wherein the hybrid neural network model is constructed on the basis of a pulse neural network model and a BERT neural network model. The hybrid neural network model is constructed on the basis of the spiking neural network model and the BERT neural network model, the natural language processing task is completed through the hybrid neural network model, and the high precision advantage of the artificial neural network model and the low power consumption calculation advantage of the spiking neural network model are effectively combined. The efficiency and accuracy of natural language processing tasks are improved, and powerful support is provided for development of the large-scale artificial intelligence model industry.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular, to a natural language processing method, apparatus, device, and medium based on a hybrid neural network model. Background Art

[0002] Currently, with the rapid development of artificial intelligence technology, artificial intelligence models have been successfully applied to the branch of natural language processing (NLP), showing good application prospects.

[0003] In related technologies, traditional artificial neural network (ANN) models such as BERT and GPT are usually adopted in the NLP field. However, in practical applications, it is found that traditional ANN models have problems such as high computational overhead and energy consumption, and the consumption of computing resources has become a major bottleneck.

[0004] In summary, the technical problems existing in related technologies need to be improved. Summary of the Invention

[0005] Embodiments of this application provide a natural language processing method, apparatus, device, and medium based on a hybrid neural network model, which can effectively combine the high-precision advantages of an artificial neural network model and the low-power consumption computing advantages of a spiking neural network model, improve the efficiency and accuracy of natural language processing tasks, and provide strong support for the development of the large-scale artificial intelligence model industry.

[0006] On the one hand, embodiments of this application provide a natural language processing method based on a hybrid neural network model. The method includes the following steps:

[0007] Obtain a natural language text to be processed;

[0008] Input the natural language text to be processed into the hybrid neural network model to obtain a natural language processing result output by the hybrid neural network model;

[0009] Wherein, the hybrid neural network model is constructed based on a spiking neural network model and a BERT neural network model.

[0010] Optionally, the hybrid neural network model includes an embedding layer, a plurality of encoding feature extraction modules, and a prediction head connected in sequence;

[0011] The encoding feature extraction module includes a periodic integrate-and-fire neuron, a periodic pulse attention module, and a hybrid unit-driven attention modulation module.

[0012] Optionally, the periodic integrate-and-fire neuron design has a periodic modulation input current mechanism and a periodic reset voltage mechanism;

[0013] Among them, the periodic modulation input current mechanism includes periodically modulating the input current of the periodic integrate-and-fire neuron; the periodic reset voltage mechanism includes periodically resetting the output voltage of the periodic integrate-and-fire neuron.

[0014] Optionally, the periodic pulse attention module is designed with a pulse processing mechanism, an adaptive convolution mechanism, and a periodic feature fusion mechanism;

[0015] Among them, the pulse processing mechanism includes calling the periodic integrate-and-fire neuron to convert the feature sequence into a pulse sequence; the adaptive convolution mechanism includes adjusting the convolution kernel size according to the number of heads and the sequence length in the multi-head attention mechanism to complete one-dimensional convolution; the periodic feature fusion mechanism includes performing periodic feature fusion on the original input feature and the feature output by the adaptive convolution.

[0016] Optionally, the hybrid unit-driven attention modulation module is designed with a dual-path hybrid coding mechanism and a joint loss function mechanism;

[0017] Among them, the dual-path hybrid coding mechanism includes a BERT neural network model path and a spiking neural network model path, and feature extraction and weighted fusion are respectively performed on the key matrix in the multi-head attention mechanism through the two paths; the joint loss function mechanism includes introducing a joint loss function, and the joint loss function includes a task loss and a regularization term.

[0018] Optionally, the encoded feature extraction module has a residual connection.

[0019] Optionally, the hybrid neural network model is trained based on the following steps:

[0020] Obtain a plurality of historical natural language texts, and obtain natural language processing results corresponding to the plurality of historical natural language texts;

[0021] Use each of the historical natural language texts as a sample, and use the natural language processing result corresponding to each of the historical natural language texts as the sample label corresponding to the sample to construct a training data set;

[0022] Use the training data set to pre-train the hybrid neural network model.

[0023] On the other hand, an embodiment of the present application provides a natural language processing device based on a hybrid neural network model. The device includes:

[0024] A text acquisition module, configured to acquire a natural language text to be processed;

[0025] A text processing module, configured to input the natural language text to be processed into a hybrid neural network model to obtain a natural language processing result output by the hybrid neural network model;

[0026] Wherein, the hybrid neural network model is constructed based on a spiking neural network model and a BERT neural network model.

[0027] On the other hand, an embodiment of the present application provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and when the processor executes the computer program, the above-mentioned natural language processing method based on the hybrid neural network model is implemented.

[0028] On the other hand, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program, and when the computer program is executed by a processor, the above-mentioned natural language processing method based on the hybrid neural network model is implemented.

[0029] In the embodiment of the present application, a hybrid neural network model is constructed based on a spiking neural network model and a BERT neural network model, and then the natural language processing task is completed through the hybrid neural network model, effectively combining the high-precision advantages of the artificial neural network model and the low-power consumption calculation advantages of the spiking neural network model, improving the efficiency and accuracy of the natural language processing task, and providing strong support for the development of the large-scale artificial intelligence model industry. Description of the Drawings

[0030] Figure 1 is a schematic diagram of the implementation environment of a natural language processing method based on a hybrid neural network model provided by an embodiment of the present application;

[0031] Figure 2 is a schematic flowchart of a natural language processing method based on a hybrid neural network model provided by an embodiment of the present application;

[0032] Figure 3 is a schematic diagram of the architecture of a hybrid neural network model provided by an embodiment of the present application;

[0033] Figure 4 is a schematic flowchart of data processing of a periodic pulse attention module provided by an embodiment of the present application;

[0034] Figure 5 is a schematic flowchart of data processing of a hybrid unit-driven attention modulation module provided by an embodiment of the present application;

[0035] Figure 6It is a schematic structural diagram of a natural language processing device based on a hybrid neural network model provided by an embodiment of the present application;

[0036] Figure 7 It is a schematic hardware structure diagram of an electronic device provided by an embodiment of the present application. Detailed implementation manners

[0037] In order to make the objectives, technical solutions and advantages of the present application more clear and understandable, the present application will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application. When the following description refers to the accompanying drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The implementation manners described in the following exemplary embodiments do not represent all implementation manners consistent with the embodiments of the present application. They are only examples of devices and methods consistent with some aspects of the embodiments of the present application detailed in the appended claims.

[0038] It can be understood that the terms "first", "second", etc. used in the present application can be used in this article to describe various concepts, but unless otherwise specified, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another. For example, without departing from the scope of the embodiments of the present application, the first information can also be referred to as the second information. Similarly, the second information can also be referred to as the first information. Depending on the context, the words "if", "when" used herein can be interpreted as "when...", "when...", or "in response to determining".

[0039] The terms "at least one", "multiple", "each", "any one", etc. used in the present application, at least one includes one, two or more than two, multiple includes two or more than two, each refers to each of the corresponding multiple, and any one refers to any one of the multiple.

[0040] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the technical field to which the present application belongs. The terms used herein are only for the purpose of describing the embodiments of the present application and are not intended to limit the present application.

[0041] Currently, with the rapid development of artificial intelligence technology, artificial intelligence models have been successfully applied to the branch of natural language processing, showing good application prospects.

[0042] In related technologies, traditional artificial neural network models such as BERT, GPT, etc. are usually adopted in the NLP field. However, in practical applications, it is found that traditional ANN models have problems such as high computational overhead and energy consumption, and the consumption of computing resources has become a major bottleneck.

[0043] In view of this, in the embodiments of the present application, a natural language processing method, apparatus, device, and medium based on a hybrid neural network model are provided. By using a spiking neural network model and a BERT neural network model as the basis to construct a hybrid neural network model, and then completing natural language processing tasks through the hybrid neural network model, the high-precision advantages of artificial neural network models and the low-power consumption calculation advantages of spiking neural network models are effectively combined, improving the efficiency and accuracy of natural language processing tasks, and providing strong support for the development of the large-scale artificial intelligence model industry.

[0044] Next, with reference to the accompanying drawings, the specific implementation manners of the embodiments of the present application will be described in detail. First, a natural language processing method based on a hybrid neural network model provided in the embodiments of the present application will be described with reference to the accompanying drawings.

[0045] Please refer to Figure 1 , Figure 1 which is a schematic diagram of the implementation environment of a natural language processing method based on a hybrid neural network model provided in the embodiments of the present application. In this implementation environment, the main software and hardware entities involved include a terminal processor 110 and a server 120.

[0046] Specifically, a control program related to the natural language processing method based on the hybrid neural network model can be installed in the terminal processor 110, and the server 120 is the background server for this control program. The terminal processor 110 and the background server 120 are communicatively connected. The natural language processing method provided in the embodiments of the present application can be executed on the side of the terminal processor 110.

[0047] The server 120 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms.

[0048] In addition, the server 120 can also be a node server in a blockchain network.

[0049] A communication connection can be established between the terminal processor 110 and the server 120 through a wireless network. The wireless network uses standard communication technologies and / or protocols. The network can be set to the Internet or any other network, such as including but not limited to any combination of a Local Area Network (LAN), a Metropolitan Area Network (MAN), a Wide Area Network (WAN), a mobile, or a wireless network, a private network, or a virtual private network. Moreover, between these software and hardware entities, the same communication connection method can be adopted, or different communication connection methods can be adopted. This application does not make specific restrictions on this.

[0050] Of course, it can be understood that Figure 1 the implementation environment in Figure 1 is only some optional application scenarios in the natural language processing method based on the hybrid neural network model provided in the embodiments of this application. The actual application is not fixed to

[0051] As Figure 2 shown, Figure 2 is a schematic flowchart of a natural language processing method based on a hybrid neural network model provided in the embodiments of this application, specifically including but not limited to steps 100 to 200.

[0052] Step 100: Obtain the natural language text to be processed.

[0053] In the embodiments of this application, the natural language text to be processed can be obtained by responding to the natural language text input by the user, and can come from various data sources such as documents, chat conversations, and social media.

[0054] Step 200: Input the natural language text to be processed into the hybrid neural network model to obtain the natural language processing result output by the hybrid neural network model;

[0055] Among them, the hybrid neural network model is constructed based on a spiking neural network model and a BERT neural network model.

[0056] In the embodiments of this application, the obtained natural language text data to be processed is input into the hybrid neural network model. The hybrid neural network model performs feature extraction and understanding, and finally outputs the processed natural language processing result, such as the output of tasks such as text classification, sentiment analysis, and question answering.

[0057] In practical applications, the hybrid neural network model is constructed based on the spiking neural network model and the BERT neural network model. It can not only effectively utilize the powerful language understanding ability of the BERT model to capture semantic information in the text, but also utilize the event-driven sparsity and binary activation characteristics of the spiking neural network (SNN) model inspired by biology, reducing energy consumption while ensuring computational efficiency.

[0058] Therefore, the hybrid neural network model provided in this application combines the spiking neural network model and the BERT neural network model, leveraging the high-precision advantages of the artificial neural network model and the low-power computing advantages of the spiking neural network model to improve the performance and efficiency of the hybrid neural network model in natural language processing tasks, providing strong support for the development of the large-scale artificial intelligence model industry. For example, it has significant advantages when deployed in low-resource environments (such as edge computing devices).

[0059] Specifically, as an optional implementation manner, the hybrid neural network model includes an embedding layer, multiple encoding feature extraction modules, and a prediction head connected in sequence;

[0060] The encoding feature extraction module includes a periodic integrate-and-fire neuron, a periodic pulse attention module, and a hybrid unit-driven attention modulation module.

[0061] In the embodiments of this application, please refer to Figure 3 , Figure 3 which is a schematic diagram of the architecture of a hybrid neural network model provided by the embodiments of this application. The hybrid neural network model mainly includes an embedding layer, multiple encoding feature extraction modules, and a prediction head.

[0062] It can be understood that the encoding feature extraction module, as the basic structure for encoding and extracting feature information in the hybrid neural network model, can stack multiple encoding feature extraction modules between the embedding layer and the prediction head to form the hybrid neural network model provided in this application. In addition, what is shown in Figure 3 is the structure of the Nth encoding feature extraction module, which is used to explain the specific structure of the encoding feature extraction module and is not intended to limit that the hybrid neural network model provided in this application can only include one encoding feature extraction module.

[0063] Such as Figure 3As shown in the figure, the encoding feature extraction module includes a Periodic Integrate-and-Fire (PIF) neuron, a Periodic Spiking Attention Module (PSAM), and a Hybrid Unit-driven Attention Modulation (HUAM) module.

[0064] Among them, the natural language text (such as a sentence) to be processed input into the hybrid neural network model is first converted into a matrix form through an embedding layer, which can be composed of a word vector dimension, a position encoding dimension, and an inter-sentence dimension.

[0065] Further, the converted feature matrix will be used as the initial data and input into the encoding feature extraction module, and feature extraction is performed using three paths. The first path is to perform individual feature extraction using the Periodic Integrate-and-Fire (PIF) neuron provided in this application. The second path is to perform feature extraction using the Hybrid Unit-driven Attention Modulation (HUAM) module provided in this application. The third path is to perform feature extraction using a linear transformation layer (Linear). After feature extraction is completed, data conversion is required to facilitate subsequent data calculations.

[0066] Further, the output results after feature extraction and data conversion using PIF and HUAM will pass through a QK T matrix multiplication operation, and then be input into a scaling module (SCALE) and the Periodic Spiking Attention Module (PSAM) provided in this application for feature extraction and data conversion. Then, the results will be input into a normalization module (softmax) to calculate scores and generate attention weights. Finally, the results will be combined with the features extracted by the linear transformation layer (Linear) (as Figure 3 shown in ), and input into a Multilayer Perceptron (MLP) for further feature extraction.

[0067] Finally, the above feature extraction process is continuously repeated through multiple encoding feature extraction modules that are set, and finally a weight is output and input into a prediction head for task processing to obtain the natural language processing result output by the hybrid neural network model.

[0068] As an optional implementation manner, the encoding feature extraction module has a residual connection.

[0069] In the embodiments of the present application, a residual connection may be provided in each encoding feature extraction module. By adding the output attention weight to the input feature, a structure of "Attention + Input" in the Transformer is formed, as Figure 3 shown. A total of two residual connections are provided, as Figure 3 shown in the two , and residual connections are respectively made to the input feature input to the encoding feature extraction module and the input feature input to the MLP.

[0070] Specifically, as an optional implementation manner, the periodic integrate-and-fire neuron is designed with a periodic modulation input current mechanism and a periodic reset voltage mechanism;

[0071] Among them, the periodic modulation input current mechanism includes periodically modulating the input current of the periodic integrate-and-fire neuron; the periodic reset voltage mechanism includes periodically resetting the output voltage of the periodic integrate-and-fire neuron.

[0072] In the embodiments of the present application, the existing integrate-and-fire (IF) neuron model usually fires a pulse after the voltage accumulates to a threshold. This mechanism overly focuses on the instantaneous cumulative effect of the voltage and ignores the periodic characteristics of neuron firing, making it difficult to effectively transmit continuous and fine-grained important information. Especially in tasks that require high-precision information transmission (such as natural language processing NLP), this defect will cause a large amount of information loss, thereby affecting the performance of the neural network model.

[0073] Taking the existing IF neuron as an example, its voltage accumulation process is shown in the following formula (1):

[0074] V(t) = V(t - Δt) + I(t) × Δt (1)

[0075] Among them, V(t) represents the membrane potential of the neuron at time step t; I(t) represents the input current at time step t; Δt represents the time step.

[0076] When V(t) ≥ Vth (Vth is a preset voltage threshold), the neuron fires a pulse and resets the membrane potential to a specific value through hard reset or soft reset. However, in some cases, if the input current I(t) is small, the voltage may not reach the threshold Vth within multiple time steps, resulting in the inability to fire a pulse, which leads to information loss. Especially when this occurs in the shallow layer of the deep network, it will seriously affect the feature extraction ability of the subsequent network and ultimately damage the performance of the entire network.

[0077] Therefore, the present application provides a periodically integrating and firing neuron PIF, and a periodically modulating input current mechanism and a periodically resetting voltage mechanism are designed in the PIF. Among them, the periodically modulating input current mechanism includes periodically modulating the input current of the PIF; the periodically resetting voltage mechanism includes periodically resetting the output voltage of the PIF.

[0078] Specifically, the present application periodically modulates the input current by introducing a differentiable transformation function (such as the tangent function) to simulate the periodic behavior of biological neurons and suppress the influence of extreme current values on voltage accumulation.

[0079] In practical applications, the voltage accumulation process of the PIF provided by the present application is shown in the following formula (2):

[0080] V(t) = V(t - Δt) + tanh(β × I(t)) × Δt (2)

[0081] Where β is a scaling factor used to adjust the nonlinear amplitude of the tangent function and can be set according to the specific scenario usage requirements.

[0082] That is to say, the present application scales the input current to the interval [-1, 1] by using the tangent function, which can effectively weaken the influence of extreme inputs.

[0083] Furthermore, the present application also replaces the existing hard reset and soft reset methods by designing a periodically resetting voltage mechanism, enabling the neuron to dynamically adjust the pulse firing state according to the period within each time step, increasing the pulse firing frequency, and improving the stability of neural network information transmission.

[0084] In practical applications, the PIF voltage reset of the present application can be shown in the following formulas (3) and (4):

[0085]

[0086] Where one time step corresponds to one period, A is the period amplitude and can be set according to the specific scenario usage requirements; T is the period length.

[0087] That is to say, when the membrane potential V(t) is negative, it indicates the trough of the cycle, and the PIF will not trigger the emission of pulses. The voltage at the next time step will be guided to the peak position of the cycle. By adding the voltage V(t-1) at the previous moment to the periodic modulation term, the voltage of the neuron can be lifted and adjusted towards the peak of the cycle. In this way, even if the input is small, the periodic mechanism can be used to gradually accumulate to a higher potential to prepare for the possible pulse emission next time; when the membrane potential V(t) is positive and exceeds the voltage threshold Vth, the PIF will be triggered to emit pulses, and the voltage at the next time step will be guided to the trough position of the cycle. By adding the current voltage V(t) to a periodic modulation term, the voltage can be rapidly decreased and directed to the trough of the cycle.

[0088] Therefore, through the periodic voltage reset mechanism, the present application raises the voltage through a sine function at the trough and reduces the voltage through a cosine function after the emission, so that the voltage changes along a preset periodic trajectory at each time step, simulating the periodic excitation characteristics of neurons, making the pulses more evenly distributed in the time dimension, and effectively avoiding the information loss caused by the inability of small currents to emit pulses.

[0089] It can be understood that by introducing the PIF and designing a periodic modulation input current mechanism and a periodic voltage reset mechanism, the present application can not only dynamically suppress the interference of extreme inputs to the network during the voltage accumulation stage, but also avoid information loss through the periodic mechanism during the voltage reset stage.

[0090] Therefore, the hybrid network model of the present application can improve the information processing ability of the SNN in NLP tasks by introducing periodic integrate-and-fire neurons, enabling it to achieve more efficient information transmission in fewer time steps and reducing the information loss problem caused by low reflectivity.

[0091] Specifically, as an optional implementation manner, the periodic pulse attention module is designed with a pulse processing mechanism, an adaptive convolution mechanism, and a periodic feature fusion mechanism;

[0092] Among them, the pulse processing mechanism includes calling the periodic integrate-and-fire neurons to convert the feature sequence into a pulse sequence; the adaptive convolution mechanism includes adjusting the convolution kernel size according to the number of heads in the multi-head attention mechanism and the sequence length to complete one-dimensional convolution; the periodic feature fusion mechanism includes performing periodic feature fusion on the original input features and the features output by the adaptive convolution.

[0093] In the embodiments of the present application, the existing BERT model is based on the Transformer model as the model backbone, and Transformer achieves the global information modeling ability in natural language processing tasks by virtue of its multi-head self-attention mechanism. However, its information transmission process mainly depends on the following three coding dimensions: word vector dimension: capturing the semantic features of the word itself; position encoding dimension: enabling the model to have sequence perception ability by adding absolute or relative position encoding information; inter-sentence dimension: achieving comprehensive modeling of inter-sentence information.

[0094] However, in the actual training process, the neural network model requires not only the initial global position information but also the dynamic capture of the change relationship between contexts. Due to the discrete characteristics of the SNN network, the information captured in the pulse conversion of the Transformer structure will be damaged to a certain extent and cannot be flexibly adjusted according to the change of weights during the training process, resulting in its insufficient performance in capturing dynamic context information. This deficiency will reduce the network's modeling ability for long-distance dependence relationships and affect the overall performance.

[0095] That is to say, when combining the existing SNN network model with the ANN network model, due to the deficiencies of the ANN network in long-distance dependence modeling, information transmission stability, and precision optimization after pulse conversion, there will be a problem of insufficient precision in natural language processing tasks.

[0096] Therefore, the present application introduces a periodic pulse attention module PSAM, and a pulse conversion mechanism, an adaptive convolution mechanism, and a periodic feature fusion mechanism are designed in the PSAM.

[0097] Among them, the pulse conversion mechanism includes calling the periodic integrate-and-fire neuron PIF to convert the feature sequence into a pulse sequence. The PSAM module uses the PIF neuron to convert the serialized intermediate result into a pulse, which can combine the sparse excitation characteristics of the PIF neuron similar to biological nerves, thereby being able to effectively model dynamic signals.

[0098] Furthermore, to achieve dynamic modeling, the PSAM module is designed with an adaptive convolution mechanism, including adjusting the convolution kernel size according to the number of heads in the multi-head attention mechanism and the sequence length, so as to complete one-dimensional convolution. During the convolution process, the size of the convolution kernel can be dynamically adjusted according to the number of heads in the Transformer multi-head attention mechanism and the length of the input feature sequence, which can be specifically expressed as , where α is a scaling factor used to control the scaling of the convolution kernel; h is the number of heads in the multi-head attention mechanism; N is the length of the input feature sequence; represents the rounding operation.

[0099] In addition, the present application also introduces a periodic feature fusion mechanism for the PSAM module, including performing periodic feature fusion on the original input features and the features output by the adaptive convolution, and further enhancing the network's perception ability of dynamic relationships by utilizing the periodic features of the SNN. By repeating the excitation and decay of pulses, the feature fusion can be shown as the following formula (5):

[0100] X PSAM = Fusion(X conv , X orig ) (5)

[0101] Wherein, X PSAM represents the feature result output after weighted fusion; X conv is the feature result output after completing one-dimensional convolution in the adaptive convolution mechanism; X orig is the original input feature; Fusion(.) is the feature fusion operation, which can be weighted summation or concatenation.

[0102] Therefore, through the pulse processing mechanism, the present application can capture the dynamic weight changes during the training process in real time, enhance the perception ability of context information, and then through the adaptive convolution mechanism and the periodic feature fusion mechanism, it can flexibly adjust the receptive field of the convolution kernel, effectively solve the problem of insufficient long-distance dependence modeling ability in the pulsed Transformer structure, and reduce the computational complexity through the sparse pulse mechanism of PIF, enhancing the robustness of the model to noise.

[0103] In practical applications, please refer to Figure 4 , Figure 4 which is a schematic flow chart of data processing of a periodic pulse attention module provided by an embodiment of the present application.

[0104] First, the multi-head attention mechanism of the Transformer will generate a series of intermediate feature representations, input these feature data into the PSAM module, and perform serialization processing, that is, convert the two-dimensional feature matrix into a one-dimensional sequence, so as to complete the serialization processing for subsequent operations. Among them, the two-dimensional feature matrix can connect all the feature values in units of rows to form a long vector, so as to be converted into a one-dimensional sequence.

[0105] Furthermore, through the pulse processing mechanism introduced by the present application, the one-dimensional sequence is pulse-processed, and a sequence represented by a pulse signal is obtained according to the pulse outputs of all time steps.

[0106] Furthermore, after the pulse processing is completed, the output pulse signal sequence can pass through the adaptive convolution mechanism introduced by the present application to complete the adaptive convolution.

[0107] Exemplarily, the sequence data can be first subjected to data form conversion, the sequence data is split and stacked into a square shape, and then data splitting is performed according to the behavior unit and according to the column unit. Among them, the horizontal data operation is row merging, and the vertical data operation is column merging.

[0108] The split data is convolved and the activation function is called. Finally, the convolution structures of the horizontal data operation and the vertical data operation are merged, and the periodic feature fusion mechanism is called to perform periodic feature fusion on the original input features and the features output by the adaptive convolution, and finally the feature result is output.

[0109] Specifically, as an optional implementation manner, the hybrid unit-driven attention modulation module is designed with a dual-path hybrid coding mechanism and a joint loss function mechanism;

[0110] Among them, the dual-path hybrid coding mechanism includes a BERT neural network model path and a spiking neural network model path, and the key matrix in the multi-head attention mechanism is respectively feature-extracted and weighted-fused through the two paths; the joint loss function mechanism includes introducing a joint loss function, and the joint loss function includes a task loss and a regularization term.

[0111] In the embodiments of the present application, in the Transformer architecture, the attention mechanism realizes feature interaction by generating query (Q), key (K), and value (V) matrices. However, if only the Q matrix is pulsed (sparse coding based on SNN), and the coding method of the K matrix is inconsistent, a series of problems will be caused. First, the pulsed Q matrix presents significant binary characteristics (0 or 1), while the continuous K matrix retains high precision but lacks sparsity, which leads to an imbalance in feature matching. For example, two similar eigenvalue (such as a difference of 0.1) in the Q matrix may be mapped to the same pulse state, and then when calculating the similarity, the subtle differences are weakened, affecting the accurate allocation of attention weights. Second, if the K matrix is completely pulsed, although the calculation efficiency can be improved, the semantic continuity as the "key" will be lost, resulting in a decline in feature expression ability. Especially in language tasks that require high-precision modeling of long-distance dependencies, the model performance will be significantly damaged. Therefore, the inconsistent coding methods of the Q matrix and the K matrix will affect the effect of the attention mechanism and cannot balance accuracy and efficiency.

[0112] Therefore, the present application introduces a hybrid unit-driven attention modulation module HUAM, and is designed with a dual-path hybrid coding mechanism and a joint loss function mechanism.

[0113] Among them, the dual-path hybrid coding mechanism includes a BERT neural network model path and a spiking neural network model path, and the key matrix in the multi-head attention mechanism is respectively feature-extracted and weighted-fused through the two paths.

[0114] Specifically, HUAM decomposes the generation of the K matrix into two parallel feature extraction paths. One is a path using a traditional ANN network (such as a BERT neural network model) to capture fine-grained semantic information and ensure high-precision modeling of key attributes. The other is a path of an SNN network (such as the PIF provided in this application), which enhances feature discrimination through sparse pulse coding, suppresses noise interference, and improves computational efficiency.

[0115] Furthermore, the output results of the two paths are fused by weighting, and the process of weighted fusion can be shown as the following formula (6):

[0116] K hybrid =α orig ×K ANN +α spiking ×K SNN (6)

[0117] Wherein, K hybrid represents the weighted fusion output result of the HUAM module; K ANN represents the output result of the BERT neural network model path; K SNN represents the output result of the spiking neural network model path; α orig is the coefficient of the BERT neural network model path; α spiking is the coefficient of the spiking neural network model path.

[0118] In practical applications, the outputs of the two paths are weighted and fused through dynamic mixing coefficients α orig and α spiking . Among them, α orig , α spiking ∈[0,1], and can be dynamically optimized by the loss function to achieve task-driven adaptive adjustment. For example, when high-precision semantic modeling is required (such as in language tasks), α orig approaches 1 to activate and enhance the BERT neural network model path; when efficient noise suppression is required (such as in dynamic perception tasks), α spiking approaches 1 to activate and enhance the spiking neural network model path.

[0119] Therefore, in order to further balance model accuracy and computational efficiency, the HUAM module also designs a joint loss function mechanism, and the joint loss function includes a task loss and a regularization term.

[0120] Specifically, the task loss can adopt the cross-entropy loss L task, ensuring the accuracy of attention weight allocation and supporting end-to-end optimization of long-distance dependencies. Specifically, the cross-entropy loss maximizes the similarity between predictions and true labels, prompting the model to learn how to allocate attention weights. Especially for tasks that require long-distance dependencies, it helps enhance the model's ability in long-distance information transmission.

[0121] The regularization term can be designed as a dual constraint term. To balance the contributions of the BERT neural network model path and the spiking neural network model path, a hybrid regularization term can be introduced.

[0122] Among them, the hybrid regularization term can include two main constraint terms. The first is weight coefficient conservation. By constraining the coefficients α orig and α spiking , ensuring that the sum of the two coefficients approaches 1, that is, |α spiking +α orig -1|. The HUAM module balances the contributions of the BERT neural network model path and the spiking neural network model path through the dynamically generated coefficients α orig and α spiking . It is driven by a periodic neuron activation function and can adaptively adjust according to the temporal characteristics of the input features.

[0123] Furthermore, the second is the smoothing constraint. By introducing a regularization term with a coefficient μ to control the changes of the coefficients α orig and α spiking , enabling the model to smoothly adjust these coefficients, preventing them from experiencing drastic fluctuations during training, which helps improve the stability of the model training process and avoid unstable convergence of the model caused by drastic changes in the coefficients.

[0124] Therefore, the regularization term loss function of this application can be expressed as the following formula (7):

[0125]

[0126] Among them, L reg is the regularization term loss function; μ is the adjustment coefficient factor, which can be set according to the specific usage scenario requirements.

[0127] Furthermore, the joint loss function of this application can be expressed as the following formula (8):

[0128] L HUAM =L task +ρL reg (8)

[0129] Among them, L HUAM represents the joint loss function of the HUAM module; L taskDenote the task loss composed of cross-entropy loss; L reg is the regularization term loss function; ρ is the adjustment coefficient factor of the regularization term loss function, which can be set according to the specific usage scenario requirements.

[0130] Therefore, by introducing the joint loss function mechanism, the HUAM module can simultaneously optimize the accuracy (through the task loss) and the model stability (through the regularization term), achieve Pareto Optimal between accuracy and efficiency, provide the network model architecture with an adaptive feature modulation ability, and can flexibly adjust the path contribution under different computing requirements.

[0131] In practical applications, please refer to Figure 5 , Figure 5 is a schematic flowchart of the data processing of a hybrid unit-driven attention modulation module provided by an embodiment of the present application.

[0132] First, the text data input to the encoding feature extraction module will complete data embedding through the embedding layer, and generate query (Q), key (K), and value (V) matrices through the attention mechanism in the Transformer architecture.

[0133] Furthermore, extract the K matrix among them, and perform feature extraction through two paths, namely the BERT neural network model path and the spiking neural network model path respectively, then perform weighted fusion through dynamic hybrid weights, and calculate the attention score, and input it to the feed-forward network to complete task prediction.

[0134] Furthermore, the hybrid loss function can also be calculated according to the prediction result, where the hybrid loss function mainly includes a task loss function and a regularization term loss function.

[0135] Finally, by combining the task loss function and the regularization term loss function, the hybrid loss function is obtained, and then the model parameters are optimized and dynamically adjusted through the hybrid loss function.

[0136] Therefore, the present application introduces the HUAM module in the hybrid neural network model, realizes cross-paradigm information fusion in the self-attention mechanism by adaptively adjusting the contribution ratio of the BERT neural network model path and the spiking neural network model path, thereby enhancing the model's ability to model dynamic context and fine-grained information.

[0137] It can be understood that the hybrid neural network model provided by the present application can achieve more efficient information transmission in fewer time steps and reduce the problem of information loss by introducing the periodic integrate-and-fire neuron PIF. By introducing the PSAM module, the accuracy problem of the BERT neural network model when dealing with long-distance dependencies after pulse conversion is solved, effectively improving the ability to capture context information in NLP tasks. By introducing the HUAM module, cross-paradigm information fusion is achieved in the self-attention mechanism by adaptively adjusting the contribution ratio of the BERT neural network model path and the spiking neural network model path, thereby enhancing the model's ability to model dynamic context and fine-grained information.

[0138] That is to say, while solving the problem of insufficient accuracy of the spiking neural network model, the hybrid neural network model of the present application also significantly reduces energy consumption and performs excellently in multiple benchmark tests in the field of natural language processing. It not only maintains the same accuracy as the traditional ANN model but also greatly reduces the energy consumption of the model, contributing to the development and wide application of large-scale artificial intelligence models and providing strong support for the development of the large-scale artificial intelligence model industry.

[0139] Exemplarily, the General Language Understanding Evaluation (GLUE) is one of the most common evaluation criteria in the field of natural language processing. According to the existing data, through experiments and simulations, on the public dataset, the performance indicators of the hybrid neural network model provided by the present application are shown in Table 1 below. Table 1 is the evaluation table of model performance indicators.

[0140] Table 1 Evaluation Table of Model Performance Indicators

[0141] Model Theoretical Calculation Power Consumption GLUE Score Existing ANN Model 51.51 82.9 Hybrid Neural Network Model 26.51 83.5

[0142] Therefore, it can be determined that the hybrid neural network model provided by the present application not only achieves an accuracy similar to that of the ANN model but also reduces the energy consumption by approximately 1 times compared to the ANN model with the same accuracy.

[0143] Specifically, as an optional implementation manner, the hybrid neural network model is trained based on the following steps:

[0144] Obtain a plurality of historical natural language texts and obtain the natural language processing results corresponding to the plurality of historical natural language texts;

[0145] Take each of the historical natural language texts as a sample and take the natural language processing result corresponding to each historical natural language text as the sample label corresponding to the sample to construct a training dataset;

[0146] Pre-train the hybrid neural network model using the training data set.

[0147] In the embodiments of the present application, multiple historical natural language texts and the natural language processing results of the multiple historical natural language texts during the processing of historical NLP tasks can be obtained. Then, taking any one of the historical natural language texts as a sample and the natural language processing result of the any one historical natural language text as the sample label corresponding to the sample, a set of training samples is formed. By obtaining multiple training samples, a training data set is constructed. Then, the training data set is input into the hybrid neural network model, and the model parameters in the hybrid neural network model are adjusted according to each output result of the hybrid neural network model, and finally the pre-training process of the hybrid neural network model is completed.

[0148] Among them, it can be considered that the pre-training process of the hybrid neural network model is completed after reaching the preset number of pre-training times; it can also be considered that the pre-training process of the hybrid neural network model is completed when the training output result of the hybrid neural network model converges.

[0149] In practical applications, after constructing the training data set, it can also be divided into a training set, a validation set, and a test set. The training set accounts for 80% of the training data set, the validation set accounts for 10% of the training data set, and the test set accounts for 10% of the training data set. The training set is used for model training, and the validation set and the test set are used for model validation.

[0150] Please refer to Figure 6 , Figure 6 FIG. is a schematic structural diagram of a natural language processing device based on a hybrid neural network model provided by an embodiment of the present application. An embodiment of the present application also provides a natural language processing device based on a hybrid neural network model, which can implement the above natural language processing method based on a hybrid neural network model. The device includes:

[0151] A text acquisition module 610, configured to acquire a natural language text to be processed;

[0152] A text processing module 620, configured to input the natural language text to be processed into the hybrid neural network model to obtain a natural language processing result output by the hybrid neural network model;

[0153] Among them, the hybrid neural network model is constructed based on a spiking neural network model and a BERT neural network model.

[0154] It can be understood that the content in the above method embodiments is applicable to the device embodiments of the present application. The functions specifically implemented by the device embodiments of the present application are the same as those in the above method embodiments, and the beneficial effects achieved are also the same as those in the above method embodiments.

[0155] Please refer to Figure 7 , Figure 7 which is a schematic diagram of the hardware structure of an electronic device provided by an embodiment of the present application. The electronic device includes:

[0156] A processor 701, which can be implemented in ways such as a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided by the embodiments of the present application;

[0157] A memory 702, which can be implemented in forms such as a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 702 can store an operating system and other application programs. When implementing the technical solutions provided by the embodiments of this specification through software or firmware, the relevant program codes are stored in the memory 702 and are called by the processor 701 to execute the natural language processing method based on the hybrid neural network model of the embodiments of the present application;

[0158] An input / output interface 703, which is used to implement information input and output;

[0159] A communication interface 704, which is used to implement communication interaction between this device and other devices, and can implement communication through wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.);

[0160] A bus 705, which transmits information between various components of the device (such as the processor 701, the memory 702, the input / output interface 703, and the communication interface 704);

[0161] Among them, the processor 701, the memory 702, the input / output interface 703, and the communication interface 704 are communicatively connected to each other inside the device through the bus 705.

[0162] The embodiments of the present application also provide a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the above-mentioned natural language processing method based on the hybrid neural network model is implemented.

[0163] It can be understood that the content in the above method embodiments is applicable to the present storage medium embodiment. The functions specifically implemented by the present storage medium embodiment are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those of the above method embodiments.

[0164] As a non-transitory computer-readable storage medium, the memory can be used to store non-transitory software programs and non-transitory computer-executable programs. In addition, the memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one magnetic disk storage device, a flash memory device, or other non-transitory solid-state storage devices. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor through a network. Examples of the above networks include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0165] A natural language processing method, apparatus, device, and medium based on a hybrid neural network model provided by an embodiment of the present application constructs a hybrid neural network model by using a spiking neural network model and a BERT neural network model as a basis, and then completes natural language processing tasks through the hybrid neural network model, effectively combining the high-precision advantages of artificial neural network models with the low-power consumption calculation advantages of spiking neural network models, improving the efficiency and accuracy of natural language processing tasks, and providing strong support for the development of the large-scale artificial intelligence model industry.

[0166] The embodiments described in the embodiments of the present application are for more clearly illustrating the technical solutions of the embodiments of the present application, and do not constitute a limitation on the technical solutions provided by the embodiments of the present application. Those skilled in the art will know that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of the present application are equally applicable to similar technical problems.

[0167] Those skilled in the art can understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than those shown, or combine certain steps, or different steps.

[0168] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separated, that is, they may be located in one place, or may be distributed to multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0169] Those of ordinary skill in the art can understand that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, and appropriate combinations thereof.

[0170] In the description of this application and the above-mentioned drawings, terms such as "first", "second", "third", "fourth", etc. (if any) are used to distinguish similar objects and do not necessarily describe a specific order or sequence. It should be understood that the data used in this way can be interchanged under appropriate circumstances so that the embodiments of this application described here can be implemented in an order other than those illustrated or described here. In addition, the terms "comprising" and "having" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that comprises a series of steps or units does not necessarily have to be limited to those steps or units clearly listed, but may include other steps or units not clearly listed or inherent to these processes, methods, products, or devices.

[0171] It should be understood that in this application, "at least one (item)" means one or more, and "a plurality" means two or more. "And / or" is used to describe the association relationship of associated objects and indicates that three relationships can exist. For example, "A and / or B" can mean: only A exists, only B exists, and both A and B exist at the same time. Among them, A and B can be singular or plural. The character " / " generally means that the associated objects before and after are in an "or" relationship. "At least one (one) of the following" or a similar expression means any combination of these items, including any combination of single items (ones) or plural items (ones). For example, at least one (one) of a, b, or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0172] In several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the above-mentioned division of units is only a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed coupling or direct coupling or communication connection to each other can be through some interfaces, and the indirect coupling or communication connection of devices or units can be in electrical, mechanical, or other forms.

[0173] The units described above as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they can be located in one place or distributed to multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0174] In addition, in each embodiment of the present application, the functional units can be integrated into one processing unit, or each unit can exist physically alone, or two or more units can be integrated into one unit. The above integrated unit can be implemented in the form of hardware or in the form of a software functional unit.

[0175] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes multiple instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods in each embodiment of the present application. The foregoing storage medium includes: various media that can store programs such as USB flash drives, mobile hard disks, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical discs.

[0176] The preferred embodiments of the embodiments of the present application have been described above with reference to the accompanying drawings. However, this does not limit the scope of the rights of the embodiments of the present application. Any modification, equivalent replacement, and improvement made by those skilled in the art without departing from the scope and essence of the embodiments of the present application shall fall within the scope of the rights of the embodiments of the present application.

Claims

1. A natural language processing method based on a hybrid neural network model, characterized in that, The method includes the following steps: Obtain the natural language text to be processed; Input the natural language text to be processed into the hybrid neural network model to obtain the natural language processing result output by the hybrid neural network model; Among them, the hybrid neural network model is constructed based on the spiking neural network model and the BERT neural network model.

2. The natural language processing method based on a hybrid neural network model according to claim 1, characterized in that The hybrid neural network model includes an embedding layer, multiple encoding feature extraction modules, and a prediction head connected in sequence; The encoding feature extraction module includes a periodic integrate-and-fire neuron, a periodic pulse attention module, and a hybrid unit-driven attention modulation module.

3. The natural language processing method based on the hybrid neural network model according to claim 2, characterized in that The periodic integrate-and-fire neuron is designed with a periodic modulation input current mechanism and a periodic reset voltage mechanism; Among them, the periodic modulation input current mechanism includes periodically modulating the input current of the periodic integrate-and-fire neuron; the periodic reset voltage mechanism includes periodically resetting the output voltage of the periodic integrate-and-fire neuron.

4. The natural language processing method based on a hybrid neural network model according to claim 2, characterized in that The periodic pulse attention module is designed with a pulse processing mechanism, an adaptive convolution mechanism, and a periodic feature fusion mechanism; Among them, the pulse processing mechanism includes calling the periodic integrate-and-fire neuron to convert the feature sequence into a pulse sequence; the adaptive convolution mechanism includes adjusting the convolution kernel size according to the number of heads in the multi-head attention mechanism and the sequence length to complete one-dimensional convolution; the periodic feature fusion mechanism includes performing periodic feature fusion on the original input feature and the feature output by the adaptive convolution.

5. The natural language processing method based on a hybrid neural network model according to claim 2, characterized in that The hybrid unit-driven attention modulation module is designed with a dual-path hybrid encoding mechanism and a joint loss function mechanism; Among them, the dual-path hybrid encoding mechanism includes a BERT neural network model path and a spiking neural network model path, and feature extraction and weighted fusion are performed on the key matrix in the multi-head attention mechanism through the two paths respectively; the joint loss function mechanism includes introducing a joint loss function, and the joint loss function includes a task loss and a regularization term.

6. The natural language processing method based on the hybrid neural network model according to claim 2, characterized in that The encoding feature extraction module has a residual connection.

7. The natural language processing method based on a hybrid neural network model according to claim 1, characterized in that The hybrid neural network model is obtained through training based on the following steps: Obtain multiple historical natural language texts, and obtain the natural language processing results corresponding to the multiple historical natural language texts; Use each of the historical natural language texts as a sample, and use the natural language processing result corresponding to each of the historical natural language texts as the sample label corresponding to the sample to construct a training data set; Use the training data set to pre-train the hybrid neural network model.

8. A natural language processing device based on a hybrid neural network model, characterized in that, The device includes: A text acquisition module for acquiring the natural language text to be processed; A text processing module for inputting the natural language text to be processed into the hybrid neural network model to obtain the natural language processing result output by the hybrid neural network model; Among them, the hybrid neural network model is constructed based on the spiking neural network model and the BERT neural network model.

9. An electronic device, characterized in that, The electronic device includes a memory and a processor. The memory stores a computer program, and when the processor executes the computer program, it implements the natural language processing method based on the hybrid neural network model according to any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the natural language processing method based on the hybrid neural network model according to any one of claims 1 to 7.

Images