Pulse transformer self-attention time domain interaction enhancement method and device

Abstract

The application discloses a pulse transformer self-attention time domain interaction enhancement method and device, a terminal and a medium. The scheme provided by the application introduces a designed temporal processing unit (TPU) into a pulse self-attention (SSA) unit in a Spikformer model to form a Spatio-Temporal Self-Attention (STSA). The TPU processes the pulse features at the current time t and the pulse features at the previous time t-1, performs a dilated convolution on the t-1 pulse features to increase the spatial feature capturing capability, and fuses the pulse feature information at different time steps, so as to realize information interaction of the pulse data features across the time steps. The STSA further performs spatial attention interaction with the time feature information, realizes attention interaction in time and space, realizes modeling of the pulse attention in the time and space dimensions, and improves the performance of the pulse neural network in processing time characteristic data.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a method and device for enhancing pulse transformer self-attention time domain interaction. Background Art

[0002] Spiking neural networks (SNNs), a third-generation neural network inspired by biological neural systems, offer significant advantages in low power consumption and real-time performance due to their event-driven and sparse coding characteristics. However, due to the discontinuous and non-differentiable nature of SNNs, which transmit spike information in discrete time steps, their training and design are relatively complex, and their performance is difficult to match that of traditional deep neural networks.

[0003] As a deep neural network model based on self-attention mechanism, Transformer has made breakthrough progress in fields such as natural language processing and computer vision. In recent years, researchers have combined Transformer with SNN and proposed the Spkiformerr model. The model structure is as follows: Figure 1 As shown. The existing pulse Transformer is modified based on this, where Spiking Self Attention (SSA) is the pulse form of self-attention, and its calculation process is Figure 2 As shown in , it can be seen that attention is calculated separately for each time step, and there is no interaction across time steps, that is, there is no attention interaction in the time domain, only interaction in the QKV spatial domain. Existing pulse transformers all have this problem, which makes the model lack of information interaction on the characteristics of multi-time-step pulse data, limiting its performance in processing data with time features. Summary of the Invention

[0004] The present application provides a method and device for enhancing the time domain interaction of pulse transformer self-attention, which is used to solve the technical problem of low model performance of the existing pulse transformer model when processing dynamic data.

[0005] To solve the above technical problems, the first aspect of the present application provides a method for enhancing temporal interaction of pulse transformer self-attention, including:

[0006] Based on the SSA unit in the Spikformer model, obtaining the pulse characteristic parameters input to the SSA unit;

[0007] According to the pulse characteristic parameters at time t and the pulse characteristic parameters at time t-1, a dilated convolution process is performed on the pulse characteristic parameters at time t-1, and then the processed pulse characteristic parameters at time t-1 are weighted fused with the number of pulse acquisitions at time t, and then remapped back to the pulse characteristics through the pulse neuron layer to obtain the fused pulse characteristic parameters;

[0008] The weighted fused feature parameters are input into the pulse neuron layer and remapped into pulse features, so as to be output to the spatial self-attention operation unit in the pulse transformer model after being processed by the pulse neuron layer.

[0009] Optionally, the pulse characteristic parameters include: a Query characteristic component, a Key characteristic component and a Value characteristic component of the pulse neuron current value.

[0010] Optionally, according to the pulse characteristic parameters at time t and the pulse characteristic parameters at time t-1, performing dilated convolution processing on the time t-1, and then performing weighted fusion of the pulse characteristic parameters at time t-1 after the processing with the time t, to obtain the fused pulse characteristic parameters specifically includes:

[0011] Extracting the Query feature component and the Value feature component at time t and time t-1 according to the pulse feature parameters at time t and the pulse feature parameters at time t-1;

[0012] According to different feature component types, the Query feature component and the Value feature component at time t-1 are respectively subjected to dilated convolution processing, and the processed feature component at time t-1 is fused with the feature component of the same type at time t, and then remapped back to the pulse feature through the pulse neuron layer to obtain the fused Query feature component and Value feature component.

[0013] Optionally, fusing the processed pulse characteristic parameters at time t-1 with the time t to obtain the fused pulse characteristic parameters specifically includes:

[0014] The pulse characteristic parameters at the processed time t-1 and the time t are weightedly fused by element-wise addition logic to obtain the fused pulse characteristic parameters.

[0015] The second aspect of the present application provides a pulse transformer self-attention time domain interaction enhancement device, comprising:

[0016] A pulse feature acquisition unit, configured to acquire pulse feature parameters input to the SSA unit based on the SSA unit in the Spikformer model;

[0017] A time processing unit is used to perform a dilated convolution process on the pulse characteristic parameters at time t-1 according to the pulse characteristic parameters at time t and the pulse characteristic parameters at time t-1, and then perform a weighted fusion of the processed pulse characteristic parameters at time t-1 and the number of pulse acquisitions at time t, and then remap them back to the pulse characteristics through the pulse neuron layer to obtain the fused pulse characteristic parameters;

[0018] A pulse feature output unit is used to input the pulse feature parameters into the pulse neuron layer to realize pulse mapping, so as to output them to the spatial self-attention operation unit in the pulse transformer model after being processed by the pulse neuron layer.

[0019] Optionally, the pulse characteristic parameters include: a Query characteristic component, a Key characteristic component and a Value characteristic component of the pulse neuron current value.

[0020] Optionally, the time processing unit is specifically configured to:

[0021] Extracting the Query feature component and the Value feature component at time t and time t-1 according to the pulse feature parameters at time t and the pulse feature parameters at time t-1;

[0022] According to different feature component types, the Query feature component and the Value feature component at time t-1 are respectively subjected to dilated convolution processing, and the processed feature component at time t-1 is fused with the feature component of the same type at time t, and then remapped back to the pulse feature through the pulse neuron layer to obtain the fused Query feature component and Value feature component.

[0023] Optionally, the time processing unit fuses the pulse characteristic parameters of the processed time t-1 with the time t to obtain the fused pulse characteristic parameters, specifically comprising:

[0024] The pulse characteristic parameters at the processed time t-1 and the time t are weightedly fused by element-wise addition logic to obtain the fused pulse characteristic parameters.

[0025] A third aspect of the present application provides a computer terminal, comprising: a memory and a processor;

[0026] The memory is used to store program code, and the program code is used to implement a pulse transfomer self-attention time domain interaction enhancement method as provided in the first aspect of the present application;

[0027] The processor is configured to read and execute the program code.

[0028] The fourth aspect of the present application provides a computer-readable storage medium, in which program code is stored, and the program code is used to be read and executed by a processor to implement a pulse transformer self-attention time domain interaction enhancement method as provided in the first aspect of the present application.

[0029] It can be seen from the above technical solutions that this application has the following advantages:

[0030] The solution provided in this application is based on the SSA unit in the Spikformer model, obtains the pulse feature parameters output by the SSA unit, and performs void convolution processing on the t-1 moment according to the pulse feature parameters at time t and the pulse feature parameters at time t-1 to increase the spatial feature capture capability and fuse the pulse feature information of different time steps. At this time, the attention already has temporal features and contains information on features across time steps. Spatial interaction is then performed to achieve information interaction of pulse data features across time steps, and also improve the performance of the spiking neural network in processing dynamic data. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0032] Figure 1 This is the model structure diagram of the Spikformer model.

[0033] Figure 2 Schematic diagram of the operational logic of the Spiking Self-Attention (SSA) unit of the existing Spiking Transformer model.

[0034] Figure 3 A flowchart of an embodiment of a pulse transformer self-attention time domain interaction enhancement method provided in this application.

[0035] Figure 4 This is a logic block diagram of the TPU unit in the pulse transformer self-attention time domain interaction enhancement method provided in this application.

[0036] Figure 5 This application provides a logic block diagram of a spatiotemporal spike self-attention (STSA) unit based on the spike transformer self-attention temporal domain interaction enhancement method.

[0037] Figure 6This is a structural diagram of an embodiment of a pulse transformer self-attention time domain interaction enhancement device provided in this application.

[0038] Figure 7 This is a structural diagram of an embodiment of a pulse transformer self-attention time domain interaction enhancement device provided in this application. DETAILED DESCRIPTION

[0039] It should be noted that the spiking neural network (SNN) uses discrete pulse sequences in calculation and information transmission. In SNN, the core function of the spiking neuron is to convert the continuous input signal into a pulse signal. Current research covers a variety of spiking neuron models, such as the Integrate-and-Fire (IF) model, the Hodgkin-Huxley model, the Izhikevich model, and the Leaky Integrate-and-Fire (LIF) model. Among them, the LIF neuron has become the most widely used model because it achieves a better balance between computational efficiency and biological similarity. This embodiment also uses LIF neurons as the basis for neuron modeling. The dynamic description of the LIF neuron is as follows:

[0040]

[0041] S[t]=Θ(H[t]-V th ) (2)

[0042] V[t]=H[t](1-S[t])+V reset S[t] (3)

[0043] Where τ is the membrane time constant and X[t] is the input current at time step t. When the membrane potential H[t] exceeds the excitation threshold V th When , the spiking neuron will trigger a pulse S[t]. Θ(v) is the Heaviside step function, which is equal to 1 when v ≥ 0 and equal to 0 otherwise. V[t] represents the membrane potential after the triggering event. If no pulse is generated, it is equal to H[t], otherwise it is equal to the reset potential V reset .

[0044] The data flow of Spiking Self Attention (SSA) in the figure is as follows:

[0045] Q=SN Q (BN(XW Q )),K=SN k (BN(XW k )),V=SN v (BN(XW v )),X∈RT×N×D (4)

[0046] SSA'(Q,K,V)=SN(QK T V*s) (5)

[0047] SSA(Q,K,V)=SN(BN(Linear(SSA′(Q,K,V))) (6)

[0048] Assuming that the current value entering the pulse neuron at time t is I[t[, it can be expressed as:

[0049] I[t]=Q[t]K T [t]V[t]*s (7)

[0050] Where X represents Figure 1 In the Spike Feature, W represents the weight in the first column of the Linear layer in the SSA unit, BN represents batch normalization, and SN represents the spike neuron layer. T×N×D Indicates the size of the Spike Feature sequence, T represents the time step, D represents the embedding dimension, and N is calculated based on the actual size. Q, K, and V correspond to the Query (Q), Key (K), and Value (V) in the transformer respectively. s is the scaling factor. K T represents the transpose of K, QK T V represents matrix multiplication calculation.

[0051] Substituting (7) into (3), the membrane potential update equation after entering the LIF neuron is:

[0052]

[0053] From the above formula (5), we can see that at time t, the core calculation of SSA only realizes the interaction of Q, K, and V features in the spatial domain. Formula (9) shows that the dynamic update of membrane potential V[t] only depends on the membrane potential [t-1] at the previous moment and the input current I[t] at the current moment, where τ and V reset are all constants. Therefore, existing temporal feature processing is performed solely through dynamic changes in membrane potential, without explicitly incorporating temporal information interactions into attention calculation mechanisms. This design limitation prevents SSA from fully capturing temporal dependencies, limiting the Spiking Transformer's temporal modeling capabilities and performance on complex time series tasks.

[0054] In view of this, an embodiment of the present application provides a pulse transformer self-attention time domain interaction enhancement method and device, which is used to solve the technical problem of low model performance of the existing pulse transformer model when processing dynamic data.

[0055] In order to make the purpose, features, and advantages of the invention of this application more obvious and easy to understand, the technical solutions in the embodiments of this application will be clearly and completely described below in conjunction with the drawings in the embodiments of this application. Obviously, the embodiments described below are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0056] First, a detailed description of an embodiment of a pulse transformer self-attention time domain interaction enhancement method provided by this application is as follows:

[0057] See also Figure 3 , this application provides a pulse transformer self-attention time domain interaction enhancement method, including:

[0058] Step 101: Based on the SSA unit in the Spikformer model, obtain the pulse characteristic parameters input to the SSA unit;

[0059] Step 102: Based on the pulse characteristic parameters at time t and the pulse characteristic parameters at time t-1, a dilated convolution process is performed on the pulse characteristic parameters at time t-1, and then the processed pulse characteristic parameters at time t-1 are weighted fused with the number of pulse acquisitions at time t. The weighted fusion is then remapped back to the pulse characteristic through the spiking neuron layer to obtain the fused pulse characteristic parameters.

[0060] Step 103: Input the pulse feature parameters into the pulse neuron layer so as to be output to the spatial self-attention operation unit in the pulse transformer model after being processed by the pulse neuron layer.

[0061] It should be noted that the method provided in this embodiment Figure 1 The improvement scheme proposed based on the Spikformer model shown in the figure is specifically an improvement scheme proposed for the pulse self-attention SSA unit in the pulse transformer model. Among them, Spikformer is a specific network structure of the pulse transformer model. The self-attention operation of this type of network all adopts the SSA method in Spikformer. The specific implementation example of the method is as follows:

[0062] First, based on the SSA unit in the Spikformer model, the pulse feature parameters input to the SSA unit are obtained, and the query feature component, key feature component, and value feature component can be separated through the BN layer and convolutional network layer with different weights;

[0063] like Figure 4 As shown, after obtaining the Query feature component, Key feature component and Value feature component, feature processing is performed through time processing logic, which can be represented by Figure 4 The TPU unit in the algorithm is executed to extract feature information of different time steps from the existing self-attention mechanism. Different time steps are multiplied by different attention scores and represented by the gating unit β, which can realize the proportional fusion of feature information of existing attention at different time channels.

[0064] TPU can process the pulse characteristic information at time t and t-1 through different channels respectively, and the pulse characteristic information at time t-1 is processed through Spike t-1 The channel is processed by dilated convolution. DConv is used to increase the spatial feature capture capability of this time step. Finally, the output is controlled by the attention gate unit β. The information at time t is transmitted through Spike t The channel passes through the gating unit 1-β, and finally the pulse features of t and t-1 are weighted and fused. This embodiment uses element-wise addition, and β is set as a hyperparameter. In this way, this embodiment realizes the interaction of pulse features across time steps. At the same time, due to the cross-time information fusion, non-pulse calculations are inevitably involved. To ensure pure event-driven characteristics, this embodiment adds a spiking neuron layer after element-wise addition. The TPU processing data flow is as follows:

[0065]

[0066] Where, Spike t Spike represents the pulse sequence at time T=t. t-1 represents the pulse sequence at time T=t-1, DConv represents dilated convolution, β represents the weight of controlling the output information (greater than or equal to 0, less than or equal to 1), and SN represents the spiking neuron layer.

[0067] Through the pulse characteristics of the TPU, cross-time step interaction is achieved, and time feature information is obtained. It can be seen from the data flow that this embodiment realizes information interaction of multiple time steps. This embodiment inserts the TPU into the existing self-attention unit to construct a spatiotemporal self-attention unit (STSA), introduces a time processing unit (TPU) into the self-attention mechanism, and fuses the pulse feature information of different time steps. At this time, the attention already has time features and contains information about cross-time step features. Spatial interaction is then carried out to realize the fusion interaction of spatiotemporal information. With the help of the TPU in this embodiment, spatiotemporal self-attention is successfully realized.

[0068] More specifically, step 102 includes the following steps:

[0069] Extracting the Query feature component and the Value feature component at time t and time t-1 according to the pulse feature parameters at time t and the pulse feature parameters at time t-1;

[0070] According to different feature component types, the Query feature component and the Value feature component of the pulse feature parameter at time t-1 are subjected to dilated convolution processing respectively, and then the processed feature component at time t-1 is fused with the feature component of the same type at time t, and then remapped back to the pulse feature through the pulse neuron layer to obtain the fused Query feature component and Value feature component.

[0071] It should be noted that, in the TPU ablation experiment of this embodiment, when the K channel also uses TPU, redundant pulse information will be generated, affecting the network's effective processing of features, and the performance will be reduced compared to the optimal method.

[0072] More specifically, regarding the TPU ablation experiment mentioned in this example: To verify the impact of TPUs on network performance in STSA, ablation experiments were conducted on both dynamic and static datasets. To control for variables, this example uniformly adopted the Spiking ST-former network structure. In STSA, this example compared the use of different numbers of TPUs, including using only one TPU, two TPUs, and three TPUs.

[0073] Results show that using the TPU alone in Q slightly improves network performance, while the most significant improvement is achieved when TPUs are used in both Q and V. At four time steps, compared to STSA without TPU, the accuracy on CIFAR10 and CIFAR100 improves by 0.82% and 1.69%, respectively. The effect is even more pronounced on dynamic datasets. With the same 16 time steps, CIFAR10-DVS improves by 2.79%, while DVS128 Gesture improves by 0.85%. On the N-CALTECH101 dataset, using STSA attention improves accuracy by 3.40% when the time step is 10. These results demonstrate that time step interaction significantly improves self-attention performance on dynamic datasets.

[0074] However, when this embodiment attempts to fuse different time-step information in QKV at the same time, the experimental results show a different trend. Compared with TPU processing only QV, TPU processes QKV at the same time. Although TPU is used to interact with information at different time steps in the three channels of QKV, the result did not achieve the best effect, but the performance declined. On CIFAR100, the accuracy of TPU processing QKV at the same time is even worse than that of using TPU to process Q interaction alone, with an accuracy drop of 0.27%. This embodiment analyzes that when TPU processes QV, K has actually interacted with all time-step information of Q and V. When the three channels of QKV interact with time information at the same time, and then perform spatial interaction of attention, redundant pulse information may be generated, affecting the network's effective processing of features, resulting in performance degradation. Therefore, if Figure 5 As shown, the best effect is achieved when only the Q channel and V channel are processed by the TPU at the same time, while the K channel is directly connected to the attention operation unit.

[0075] The above is a detailed description of a pulse transformer self-attention time domain interaction enhancement method provided by this application. The following is a detailed description of an embodiment of a pulse transformer self-attention time domain interaction enhancement device provided by this application.

[0076] See also Figure 6 , an embodiment of the present application provides a pulse transformer self-attention time domain interaction enhancement device, comprising:

[0077] The pulse characteristic acquisition unit 201 is used to acquire the pulse characteristic parameters input to the SSA unit based on the SSA unit in the Spikformer model;

[0078] The time processing unit 202 is used to perform a dilated convolution process on the pulse characteristic parameters at time t-1 based on the pulse characteristic parameters at time t and the pulse characteristic parameters at time t-1, and then perform a weighted fusion of the processed pulse characteristic parameters at time t-1 and the number of pulse acquisitions at time t, and then remap them back to the pulse characteristics through the spiking neuron layer to obtain the fused pulse characteristic parameters;

[0079] The pulse feature output unit 203 is used to input the pulse feature parameters into the pulse neuron layer so as to output them to the spatiotemporal self-attention unit in the pulse transformer model after being processed by the pulse neuron layer.

[0080] Furthermore, the pulse feature parameters include: a Query feature component, a Key feature component, and a Value feature component of the pulse neuron current value.

[0081] Furthermore, the time processing unit 202 is specifically configured to:

[0082] Extracting the Query feature component and the Value feature component at time t and time t-1 according to the pulse feature parameters at time t and the pulse feature parameters at time t-1;

[0083] According to different feature component types, the Query feature component and the Value feature component at time t-1 are subjected to dilated convolution processing respectively. The processed feature component at time t-1 is fused with the feature component of the same type at time t, and then remapped back to the pulse feature through the pulse neuron layer to obtain the fused Query feature component and Value feature component.

[0084] Furthermore, the time processing unit fuses the pulse characteristic parameters at time t-1 and time t to obtain the fused pulse characteristic parameters, specifically including:

[0085] Through element-wise addition logic, the pulse characteristic parameters at time t-1 and time t after processing are weighted fused to obtain the fused pulse characteristic parameters.

[0086] In addition, based on the above embodiments, the present application also provides a detailed description of an embodiment of a computer terminal and a computer-readable storage medium, which are as follows:

[0087] like Figure 7 As shown, an embodiment of the present application provides a computer terminal, the main components of the terminal include: a memory 33 and a processor 31, the memory 33 and the processor 31 can be connected via a communication bus 34;

[0088] The memory 33 is used to store program codes, and the program codes are used to implement a pulse transformer self-attention time domain interaction enhancement method as provided in the above embodiment;

[0089] The processor 31 is used to read and execute program codes.

[0090] The fourth aspect of the present application provides a computer-readable storage medium, which stores program code, and the program code is used to be read and executed by a processor to implement a pulse transformer self-attention time domain interaction enhancement method as provided in the above embodiment.

[0091] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described devices and units can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0092] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of the units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0093] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequential order. It should be understood that the numbers used in this way can be interchanged where appropriate, so that the embodiments of the present application described herein can, for example, be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0094] It should be understood that in this application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following items" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0095] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0096] In addition, the functional units in the various embodiments of the present invention may be integrated into a single processing unit, each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0097] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0098] As described above, the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the above embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the above embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A pulse transformer self-attention time domain interaction enhancement method, characterized by: include: Based on the SSA unit in the Spikformer model, obtaining the pulse characteristic parameters input to the SSA unit; According to the pulse characteristic parameters at time t and the pulse characteristic parameters at time t-1, a dilated convolution process is performed on the pulse characteristic parameters at time t-1, and then the processed pulse characteristic parameters at time t-1 are weighted fused with the pulse characteristic parameters at time t, and then remapped back to the pulse characteristics through the pulse neuron layer to obtain the fused pulse characteristic parameters; Inputting the weighted fused feature parameters into a spiking neuron layer and remapping them into spiking features, so as to output them to a spatial self-attention operation unit in the spiking transformer model after being processed by the spiking neuron layer; Wherein, the pulse characteristic parameters include: Query characteristic component, Key characteristic component and Value characteristic component of the pulse neuron current value; According to the pulse characteristic parameters at time t and the pulse characteristic parameters at time t-1, a dilated convolution process is performed on the time t-1, and then the processed pulse characteristic parameters at time t-1 and the time t are weightedly fused to obtain the fused pulse characteristic parameters. Specifically, the fused pulse characteristic parameters include: Extracting the Query feature component and the Value feature component at time t and time t-1 according to the pulse feature parameters at time t and the pulse feature parameters at time t-1; According to different feature component types, the Query feature component and the Value feature component at time t-1 are respectively subjected to dilated convolution processing, and the processed feature component at time t-1 is weightedly fused with the feature component of the same type at time t, and then remapped back to the pulse feature through the pulse neuron layer to obtain the fused Query feature component and Value feature component; The fusion of the pulse characteristic parameters at time t-1 after processing and at time t to obtain the fused pulse characteristic parameters specifically includes: The pulse characteristic parameters at the processed time t-1 and the time t are weightedly fused by element-wise addition logic to obtain the fused pulse characteristic parameters.

2. A pulse Transformer self-attention time domain interaction enhancement device, characterized in that: include: A pulse feature acquisition unit, configured to acquire pulse feature parameters input to the SSA unit based on the SSA unit in the Spikformer model; a time processing unit for performing a dilated convolution process on the pulse characteristic parameters at time t-1 according to the pulse characteristic parameters at time t and the pulse characteristic parameters at time t-1, performing a weighted fusion of the processed pulse characteristic parameters at time t-1 and the pulse characteristic parameters at time t, and then remapping the pulse characteristic parameters back to the pulse characteristics through the pulse neuron layer to obtain the fused pulse characteristic parameters; A pulse feature output unit, configured to input the pulse feature parameters into a pulse neuron layer to implement pulse mapping, so as to output the pulse feature parameters to a spatial self-attention operation unit in the pulse transformer model after being processed by the pulse neuron layer; Wherein, the pulse characteristic parameters include: Query characteristic component, Key characteristic component and Value characteristic component of the pulse neuron current value; The time processing unit is specifically used for: Extracting the Query feature component and the Value feature component at time t and time t-1 according to the pulse feature parameters at time t and the pulse feature parameters at time t-1; According to different feature component types, the Query feature component and the Value feature component at time t-1 are respectively subjected to dilated convolution processing, and the processed feature component at time t-1 is weightedly fused with the feature component of the same type at time t, and then remapped back to the pulse feature through the pulse neuron layer to obtain the fused Query feature component and Value feature component; The time processing unit fuses the pulse characteristic parameters of the processed time t-1 with the time t to obtain the fused pulse characteristic parameters, specifically comprising: The pulse characteristic parameters at the processed time t-1 and the time t are weightedly fused by element-wise addition logic to obtain the fused pulse characteristic parameters.

3. A computer terminal, characterized in that: include: memory and processor; The memory is used to store program code, and the program code is used to implement the pulse tranformer self-attention time domain interaction enhancement method according to claim 1; The processor is configured to read and execute the program code.

4. A computer-readable storage medium, characterized in that The computer-readable storage medium stores program code, which is used to be read and executed by a processor to implement a pulse transformer self-attention time domain interaction enhancement method as described in claim 1.

Images