Oral image classification method based on enhanced pulse transformer

Abstract

The application provides an oral image classification method based on an enhanced pulse Transformer, comprising: obtaining an original oral image and a trained model, the model comprising a double-pooling residual channel attention input patch embedding module, a pulse position embedding layer, an alternately connected parallel cyclic pulse attention module and a double-pooling residual channel attention multilayer perception module, and a classification head. After the original oral image is extracted to obtain a structured pulse feature sequence by the input patch embedding module, position coding information is superimposed by the pulse position embedding layer, and the structured pulse feature sequence is alternately input into the parallel cyclic pulse attention module and the double-pooling residual channel attention multilayer perception module to extract a final high-level pulse feature, and finally, a lesion category label is obtained by classification of the classification head. The application alleviates the feature loss problem in the oral image classification task, improves the classification accuracy, effectively compresses the model parameter quantity, and reduces the demand of model edge deployment on storage and memory resources.

Description

Technical Field

[0001] This application belongs to the field of oral image classification technology, specifically relating to an oral image classification method based on an enhanced pulse Transformer. Background Technology

[0002] With the widespread adoption of edge intelligent devices, DVS (Dynamic Vision Sensors), and miniature embedded vision terminals, low-power, low-latency, and lightweight image classification algorithms have become an industry necessity. Traditional image classification methods mainly rely on ANN (Artificial Neural Network), CNN (Convolutional Neural Network), and standard Transformer models. These methods rely on intensive floating-point matrix multiplication and addition operations for feature extraction and inference, resulting in high computational cost and power consumption, making them difficult to implement on small terminal devices with limited battery power and computing power. Meanwhile, the sparse event stream data generated by neuromorphic vision sensors (such as dynamic vision sensors) is difficult for traditional visual classification models to process efficiently, necessitating a lightweight image classification solution that combines low power consumption with strong spatiotemporal feature representation capabilities.

[0003] In existing image classification technologies, SNN (Spiking Neural Network) simulates the pulse firing mechanism of biological neurons and adopts event-driven sparse computing. It only generates pulse output when the input membrane potential exceeds the threshold. It naturally has the advantages of low power consumption, adaptability to neuromorphic hardware, and good processing of temporal event signals, and has become the mainstream research direction for edge visual classification tasks. Spikformer (Zhou Z, Zhu Y, He C, et al. Spikformer: When spiking neural network meets transformer[J]. arXiv preprint arXiv:2209.15425, 2022.) is the first spiking visual classification model that combines spiking neurons with the Transformer self-attention mechanism. It uses LIF (Leaky Integrate-and-Fire) spiking neurons to replace the activation function of the standard Transformer. It transforms the query, key, and value matrix into a 0 / 1 binary spiking sequence through SSA (Spiking Multi-head Self-Attention), eliminates Softmax floating-point operations, and reduces computational energy consumption by relying on spiking sparsity. It uses the SPS (Spiking Patch Splitting) module to complete image segmentation, convolutional downsampling, and spiking position embedding, and builds a multi-layer spiking self-attention encoder to achieve image classification. It is suitable for static images and the DVS neuromorphic dataset. In addition, the SE-ResNet (Hu J, Shen L, Sun G. Squeeze-and-excitation networks[C] / / Proceedings of the IEEE conference on computer vision and pattern recognition. 2018: 7132-7141.) channel attention network squeezes global channel features through global average pooling, models the inter-channel dependencies using double fully connected layers, outputs weight-calibrated feature maps, and is paired with residual connections to alleviate the gradient vanishing problem in deep networks. It has been widely used to improve the channel feature representation capabilities of various image classification networks.The Circulant_Linear lightweight linear layer (Tang Y, Han K, Xu C, et al. Augmentedshortcuts for vision transformers[J]. Advances in Neural Information Processing Systems, 2021, 34: 15316-15327.) utilizes the periodicity and symmetry of the circulant matrix to split the fully connected weight matrix, and generates the weight matrix cyclically with a one-dimensional parameter vector. This can significantly reduce the number of parameters in the fully connected layer, providing an effective technical path for the lightweight deployment of image classification models.

[0004] However, the existing image classification methods still have the following drawbacks. First, Spikformer's SPS feature embedding module uses global max pooling for downsampling, retaining only the strongest local response features and losing global average statistical information. This results in the filtering of weak discriminative features, severely limiting recognition accuracy for image classification tasks with low contrast, low illumination, or small differences in target granularity. Simultaneously, the traditional SE module uses only a single average pooling, capturing only global statistical features while ignoring local peak discriminative features. When applied to pulse image classification networks, it cannot adapt to the feature attenuation problem caused by the pulse threshold, and its channel feature calibration capability is insufficient. Second, existing pulse Transformer image classification models require long time steps (usually T≥4) to complete temporal information encoding, resulting in redundant model parameters, increased difficulty in deploying to edge devices, and high memory consumption. Shortening the time step leads to insufficient extraction of temporal features, causing a sharp drop in classification accuracy, making it difficult to meet the stringent accuracy requirements of edge visual classification scenarios. Third, lightweighting methods for conventional image classification models (such as reducing the number of network layers and compressing the number of channels) significantly reduce classification accuracy, making it difficult to balance lightweighting with recognition performance. Furthermore, existing pulse Transformer image classification models rely solely on single-stream pulse self-attention for feature extraction, resulting in a single feature dimension and redundant parameters. They require deep network structures and long temporal strides to ensure classification accuracy, limiting their deployment on resource-constrained devices. Fourth, existing Spikformer models do not incorporate channel attention mechanisms in the feature embedding, encoder, and MLP (Multilayer Perceptron) stages. They apply the same firing standard to all feature channels using a uniform LIF pulse threshold, failing to distinguish between critical feature channels and noisy channels. When faced with image inputs containing complex backgrounds, noise interference, or varying illumination, the model struggles to adaptively enhance effective features and suppress irrelevant information, resulting in poor classification robustness and insufficient generalization ability. Summary of the Invention

[0005] To address the aforementioned problems in the existing technology, this application provides a method for classifying oral cavity images based on an enhanced pulse Transformer. The technical problem to be solved by this application is achieved through the following technical solution: A method for classifying oral cavity images based on an enhanced pulse Transformer includes: S1. Acquire raw oral cavity images from an oral cavity detector and obtain a trained enhanced pulse Transformer model; wherein, the trained enhanced pulse Transformer model includes a dual-pooling residual channel attention input patch embedding module, a pulse position embedding layer, N parallel cyclic pulse attention modules, N dual-pooling residual channel attention multilayer perceptron modules, and a classification head; wherein, the parallel cyclic pulse attention modules and the dual-pooling residual channel attention multilayer perceptron modules are alternately connected; S2, the original oral cavity image is input into the trained enhanced pulse Transformer model to extract a structured pulse feature sequence from the original oral cavity image using the dual-pooling residual channel attention input patch embedding module. Position encoding information is superimposed onto the structured pulse feature sequence using the pulse position embedding layer. Then, the final high-level pulse features are alternately extracted from the structured pulse feature sequence carrying position encoding information using the N parallel cyclic pulse attention modules and the N dual-pooling residual channel attention multilayer perceptron modules. The classification head is used to classify the final high-level pulse features to obtain the lesion category label corresponding to the original oral cavity image.

[0006] Compared with the prior art, this application has the following advantages: (1) This application designs a dual-pooling residual channel attention module, which integrates global average pooling and global max pooling strategies in parallel. This expands the network's perception of features from a single global average response to a multi-level expression of global-local features, overcoming the feature loss problem caused by a single pooling method. It generates more discriminative channel weights and significantly enhances the adaptive calibration capability for complex features. At the same time, this application embeds this module into the entire link such as feature embedding and feature transformation. It recalibrates each channel one by one through the channel weight vector, emphasizing key feature channels and suppressing noise channels. This effectively improves the defect of existing pulse Transformers that cannot distinguish key features from noise due to a uniform pulse threshold, and enhances the model's classification robustness in low-contrast, complex background and noisy interference scenarios.

[0007] (2) This application designs a parallel cyclic pulse attention module, which uses three branches—identity mapping, pulse self-attention, and cyclic linear transformation—to extract features in parallel and fuse them element by element. The cyclic linear transformation branch relies on the weight sharing mechanism of the cyclic matrix to generate a weight matrix with a one-dimensional parameter vector as the cyclic basis, which greatly reduces the number of parameters in the fully connected layer and provides additional feature expression dimensions with extremely low parameter quantity. Combined with the identity mapping branch, it retains the original information, makes up for the problem of insufficient pulse information under short time step size, and enables the model to fully extract discriminative features without relying on long time step iteration, which greatly reduces the time step size and inference delay required by the model and achieves coordinated and controllable parameter quantity and accuracy.

[0008] (3) This application adopts a structure in which N parallel cyclic pulse attention modules and N dual-pooling residual channel attention multilayer perceptron modules are alternately connected, so that the network can repeatedly perform feature interaction and nonlinear transformation at different levels, deepen the feature expression step by step, fully explore the spatial and temporal information of the image, and finally extract more discriminative high-level pulse features, providing a flexible and lightweight control scheme for resource-constrained deployment scenarios.

[0009] (4) This application relies on LIF leak integral firing spiking neurons to achieve event-driven sparse computation. Each module in the enhanced spiking Transformer model uses binary spiking sequences as processing objects, without relying on floating-point matrix multiplication and addition operations, and naturally has the advantages of low power consumption and adaptability to neuromorphic hardware. The designed spiking self-attention branch uses the non-negativity of spiking sequences to directly measure feature similarity, omitting Softmax floating-point operations, which further reduces computational overhead during this operation. Through the close connection of each module, this application forms a complete technical closed loop from feature extraction, feature interaction to feature transformation and classification decision. The innovative modules are optimized collaboratively, which significantly improves the accuracy of oral image lesion classification under the premise of effectively controlling the number of parameters, and provides a complete, efficient and deployable technical solution for oral image-assisted diagnosis and a wider range of visual classification tasks.

[0010] The present application will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0011] Figure 1 This is a schematic flowchart of an oral image classification method based on an enhanced pulse Transformer provided in an embodiment of this application; Figure 2 This is a schematic diagram of the structure of the enhanced pulse Transformer model provided in the embodiments of this application; Figure 3 This is a schematic diagram of the structure of the dual-pooling residual channel attention input map embedding module provided in the embodiments of this application; Figure 4 This is a schematic diagram of the structure of the parallel cyclic pulse attention module provided in the embodiments of this application; Figure 5 This is a schematic diagram of the structure of the pulse self-attention module provided in the embodiments of this application; Figure 6 This is a schematic diagram of the structure of the dual-pooling residual channel attention multilayer perceptron module provided in the embodiments of this application; Figure 7 This is a schematic diagram of the internal structure of the dual-pooling residual channel attention module provided in an embodiment of this application; Figure 8 This is a comparison graph of the change curves of loss function value and accuracy in the ablation experiment provided in the embodiments of this application. Detailed Implementation

[0012] The present application will be described in further detail below with reference to specific embodiments, but the implementation of the present application is not limited thereto.

[0013] This application provides an oral image classification method based on an enhanced pulse Transformer. By fusing a dual-pooling residual channel attention mechanism with a recurrent linear lightweight structure, it achieves accurate classification of lesion types in oral images. The complete process is as follows: the original oral image is input to a dual-pooling residual channel attention input patch embedding module. Local initial features are extracted and channel weights are calibrated through four cascaded embedding sub-modules. Average pooling downsampling is used to output a structured pulse feature sequence. This structured pulse feature sequence is input to a pulse position embedding layer to superimpose position encoding information, resulting in a structured pulse feature sequence carrying position encoding information. This structured pulse feature sequence carrying position encoding information is input to N alternately connected parallel recurrent pulse attention modules and a dual-pooling residual channel attention multilayer perceptron module. After multiple feature interactions and nonlinear transformations, the final high-level pulse features are output. Finally, the classification head outputs the predicted probabilities of each category to obtain the lesion category label corresponding to the oral image.

[0014] like Figure 1 As shown, the oral cavity image classification method based on enhanced pulse Transformer provided in this application includes: S1: Acquire raw oral cavity images from the oral cavity detector and obtain a trained enhanced pulse Transformer model.

[0015] In this step, raw oral images to be classified are acquired using an oral imaging device. This device includes, but is not limited to, an oral endoscope and a pathological slide scanner. The raw oral images include at least one of the following: clinical white light images, fluorescence images, endoscopic images, and pathological slide microscopic images. Clinical white light images are photographs of the oral cavity taken under conventional white light, used to observe the macroscopic morphological characteristics of lesions, including their color, boundaries, and surface texture. Fluorescence images utilize the principle that normal and diseased tissues emit different colors of fluorescence under specific lighting to enhance the visualization of suspicious areas. Pathological slide microscopic images are high-resolution images taken under a microscope after biopsy samples have been prepared into pathological slides; they allow direct observation of the microscopic structure of cells and tissues and are an important basis for determining the benign or malignant nature of lesions.

[0016] Simultaneously, obtain the trained enhanced pulse Transformer model, for reference. Figure 2 The enhanced pulse Transformer model includes a dual-pooling residual channel attention input patch embedding module, a pulse position embedding layer, N parallel cyclic pulse attention modules, N dual-pooling residual channel attention multilayer perceptron modules, and a classification head; wherein the parallel cyclic pulse attention modules and the dual-pooling residual channel attention multilayer perceptron modules are alternately connected.

[0017] The enhanced spiking Transformer model in this application relies on LIF (Leaky Integrate-and-Fire) spiking neurons to achieve event-driven sparse computation. LIF neurons simulate the spiking mechanism of biological neurons, generating spiking output only when the input signal accumulates to exceed a threshold, naturally possessing the advantages of low power consumption and compatibility with neuromorphic hardware.

[0018] S2, the original oral cavity image is input into the trained enhanced pulse Transformer model, so that the enhanced pulse Transformer model uses the dual-pooling residual channel attention input patch embedding module to extract structured pulse feature sequences from the original oral cavity image, uses the pulse position embedding layer to superimpose position encoding information into the structured pulse feature sequences, uses N alternately connected parallel cyclic pulse attention modules and dual-pooling residual channel attention multilayer perceptron modules to alternately extract the final high-level pulse features from the structured pulse feature sequences carrying position encoding information, and uses the classification head to classify the final high-level pulse features to obtain the lesion category label corresponding to the original oral cavity image.

[0019] In this step, the raw oral cavity image acquired by S1 is input into the trained enhanced pulse Transformer model. The enhanced pulse Transformer model processes the raw oral cavity image according to the following process: (a) The DPSER-IPE (DualPool SE-Resnet Input Patches Embedding) module extracts structured pulse feature sequences.

[0020] (ii) SPE (Spike Position Embedding) superimposes position encoding information onto the structured pulse feature sequence, and outputs a structured pulse feature sequence carrying position encoding information; (iii) The PCSA (Parallel Circulant Spiking Attention) module is used to extract self-attention features; (iv) The DPSER-MLP (DualPool SE-Resnet MLP) module is used to extract impulse features, and the last DPSER-MLP module outputs the final high-level impulse features. (v) The classification head aggregates the final high-level pulse features, maps them to the classification category dimension through a linear layer, outputs the predicted probability of each category through the Softmax activation function, and takes the category with the highest probability value as the lesion category label corresponding to the oral cavity image.

[0021] The specific implementation methods of each module are described in detail below.

[0022] The DPSER-IPE module aims to transform the spatial pixel information of the input image into a structured sequence representation, while adaptively enhancing the saliency of key features through a dual-pooling channel attention mechanism. The DPSER-IPE module contains multiple cascaded embedding sub-modules.

[0023] refer to Figure 3The dual-pooling residual channel attention input patch embedding module comprises four cascaded embedding sub-modules, with the output of the previous embedding sub-module serving as the input of the next embedding sub-module. Each embedding sub-module includes a two-dimensional convolutional layer, a first batch normalization layer, a first dual-pooling residual channel attention module, and a first spiking neural network, with average pooling layers only present in the third and fourth embedding sub-modules. Each of the N parallel cyclic spiking attention modules includes three parallel first branches and a first fusion layer, with the outputs of the three first branches connected to the input of the first fusion layer. Each of the N dual-pooling residual channel attention multilayer perceptron modules includes three parallel second branches and a second fusion layer, with the outputs of the three second branches connected to the input of the second fusion layer.

[0024] exist Figure 3 In this architecture, the DPSER-IPE module comprises four cascaded embedding sub-modules. The number of output channels in each sub-module expands sequentially along the cascade direction to D / 8, D / 4, D / 2, and D, respectively. The output of the preceding sub-module serves as the input to the following sub-module. Each sub-module includes a Conv2d (Convolution 2D) layer, a first BN (Batch Normalization) layer, a first DPSER (DualPool SE-ResNet) layer, and a first spiking neural network. Only the third and fourth sub-modules contain an AP (Average Pooling) layer. The average pooling layer is used to compress the feature map space size of the corresponding sub-module's output to half its original size.

[0025] The specific processing procedure for each of the embedded sub-modules is as follows: The two-dimensional convolutional layer uses a 3×3 kernel size and a stride of 1 to perform convolution operations on the input features to extract local features, resulting in convolutional features. The input features of the first embedding submodule are the original oral cavity image, while the input features of the remaining embedding submodules are the calibrated features output from the previous embedding submodule.

[0026] The first batch normalization layer normalizes the convolutional features to stabilize the training process and accelerate model convergence, resulting in normalized features. The dual-pooling residual channel attention module recalibrates the channels of the normalized features to obtain calibrated features. The first spiking neural network inputs the calibrated features into the built-in LIF leaky integral firing spiking neurons, converting them into a binary pulse time sequence. The average pooling layers in the third and fourth embedding sub-modules are used to perform average pooling downsampling on the feature maps output by the corresponding embedding sub-modules in the spatial dimension, compressing the spatial size to half of the original size; For the embedded submodules with average pooling layers, the average pooling layer performs average pooling downsampling on the calibrated feature map in the spatial dimension, compressing the spatial size to half of its original size, thus reducing computational complexity while preserving global feature statistics. This application replaces the max pooling used for downsampling in the original Spikformer (Spiking Transformer) with an average pooling layer because average pooling can more comprehensively preserve feature information, reduce dependence on local extrema, and avoid feature loss or bias that may be caused by max pooling. After processing by the average pooling layers of the third and fourth embedded submodules, the spatial size of the feature map has been progressively compressed. The dimension of the structured pulse feature sequence output by the DPSER-IPE module is... ,in For time step, The number of image blocks. The feature embedding dimension.

[0027] like Figure 4 As shown, the three first branches set in parallel in the N parallel cyclic pulse attention modules are the first cyclic linear transformation branch, the first identity mapping branch, and the pulse self-attention branch, respectively. The first cyclic linear transformation branch in the current parallel cyclic pulse attention module is composed of a first cyclic linear layer, used to perform a linear transformation on the current input feature to obtain the current first linear feature; the first identity mapping branch is used to directly output the current input feature; the pulse self-attention branch is composed of a pulse self-attention module, used to perform self-attention processing on the current input feature and output the current self-attention feature; the first fusion layer is used to fuse the current first linear feature, the current input feature, and the current self-attention feature, and output the current first fused feature; wherein, the first parallel cyclic pulse attention module inputs a structured pulse feature sequence; The first recurrent linear layer utilizes the weight-sharing mechanism of the cyclic matrix to perform a lightweight linear transformation on the current input features. Its core principle is to leverage the special structural properties of the cyclic matrix to decompose the weight matrix of a traditional fully connected layer into multiple cyclic blocks. Using a one-dimensional parameter vector as the cyclic basis, a weight matrix is ​​generated according to the index relationship of the cyclic matrix. This weight matrix is ​​then used to perform a linear mapping on the current input features, outputting the current first linear feature. Compared to traditional linear layers, the first recurrent linear layer optimizes parameter storage and computation processes through the periodicity and symmetry of the cyclic matrix, significantly reducing the number of parameters while maintaining the functionality of traditional linear layers.

[0028] During the initialization phase, the first recurrent linear layer calculates the recurrent block parameters and index matrix based on the input and output feature dimensions and the preset number of recurrent blocks, and encodes and stores the index relationship of the recurrent matrix. Simultaneously, it initializes the weight parameters and bias terms. During forward propagation, the first recurrent linear layer constructs a recurrent weight matrix from the one-dimensional parameter vector using the index matrix, and then completes the mapping from the current input feature to the current first linear feature using a standard linear transformation function.

[0029] Specifically, for the current input features The first recurrent linear layer uses a one-dimensional parameter vector. Generate the cyclic weight matrix according to the index relationship of the cyclic matrix. , The dimension is The space of real matrices, The dimension is The real vector space. The linear mapping process can be represented as: (1); in, This represents the current first linear feature. This is the bias value.

[0030] The PCSA module employs a three-branch parallel architecture, integrating three key components: the SSA (Spiking Self-Attention) module, the first recurrent linear layer, and residual connections. The SSA is responsible for modeling long-range spatiotemporal dependencies; the recurrent linear layer performs a lightweight linear transformation on the original input through a parameter-shared cyclic matrix, serving as a direct feature supplement that bypasses the attention mechanism; and the first identity mapping branch maintains the integrity of the original input information.

[0031] refer to Figure 5 As shown, the pulse self-attention module in the current parallel cyclic pulse attention module includes three mapping sub-paths, a matrix-vector inner product operation unit, a feature scaling layer, a fifth spiking neural network, a fourth linear transformation layer, a fourth batch normalization layer, and a sixth spiking neural network. The three mapping sub-paths are a query sub-path, a key pulse sub-path, and a value pulse sub-path, respectively. The query sub-path, key pulse sub-path, and value pulse sub-path input the current input feature and output the input connected to the matrix-vector inner product operation unit. The matrix-vector inner product operation unit, feature scaling layer, fifth spiking neural network, fourth linear transformation layer, fourth batch normalization layer, and sixth spiking neural network are connected sequentially, and the sixth spiking neural network outputs the current self-attention feature. The current self-attention feature output by the sixth spiking neural network in the last parallel cyclic pulse attention module is the final self-attention feature.

[0032] Spike Self-Attention (SSA) is optimized for the characteristics of SNNs. Traditional self-attention mechanisms in Transformers suffer from two key problems in SNNs: first, floating-point matrix operations and the softmax function do not conform to the computational rules of SNNs; second, the quadratic complexity is insufficient for the high-efficiency computational requirements of SNNs. SSA addresses these problems through three innovative designs: first, queries, keys, and values ​​are converted into 0 / 1 spike sequences through a spiking neuron layer; second, a self-attention map is generated using pure spike-based similarity calculations; and finally, an adjustable scaling factor is introduced to control the magnitude. Notably, SSA eliminates the need for softmax operations because the spike sequences are inherently non-negative, enabling direct feature selection and aggregation.

[0033] Continue to refer to Figure 5 The query sub-route is composed of a first linear transformation layer, a second batch normalization layer, and a second spiking neural network connected in sequence; the current input feature is linearly transformed by the first linear transformation layer and then input into the second batch normalization layer for normalization, and then mapped into a binary query pulse sequence by the second spiking neural network; The key pulse sub-routes are composed of a second linear transformation layer, a third batch normalization layer, and a third spiking neural network connected in sequence; the current input features are linearly transformed by the second linear transformation layer and then input into the third batch normalization layer for normalization, and then mapped into a binary key pulse sequence by the third spiking neural network; The value pulse sub-routes are composed of a third linear transformation layer, a fourth batch normalization layer, and a fourth spiking neural network connected in sequence; the outputs of the second, third, and fourth spiking neural networks are connected to the input of the matrix-vector inner product operation unit; the current input features are linearly transformed by the third linear transformation layer and then input to the fourth batch normalization layer for normalization, and then mapped into a binary value pulse sequence by the fourth spiking neural network.

[0034] The query pulse sequence ( ), key pulse sequence ( ) and value pulse sequence ( ) is represented as: (2); in, For the current input features, Indicates will After being mapped by the first linear transformation layer and normalized by the second batch normalization layer, the signal is input into the second spiking neural network and converted into a binary query pulse sequence. ; Indicates will After being mapped by the second linear transformation layer and normalized by the third batch normalization layer, the signal is input into the third spiking neural network and transformed into a binary key pulse sequence. ; Indicates will After being mapped by the third linear transformation layer and normalized by the fourth batch normalization layer, the signal is input into the fourth spiking neural network and transformed into a binary value pulse sequence. .

[0035] The query pulse sequence, the key pulse sequence, and the value pulse sequence are subjected to matrix multiplication by the matrix vector inner product operation unit, and then scaled by the feature scaling layer to obtain self-attention intermediate features; The first fusion layer is used to fuse the current first linear feature, the current input feature, and the current self-attention feature, and output the current first fused feature; (3); in, As the current primary fusion feature, Indicates will The current self-attention feature output after the input pulse self-attention branch. Indicates will The current first linear feature output after inputting the first cyclic linear transformation branch.

[0036] The current first fusion feature output by the first fusion layer in the last parallel cyclic pulse attention module is used as the input to the first dual-pooling residual channel attention multilayer perceptron module.

[0037] The fusion of the three branches continues the characteristics of residual connection optimization of gradient propagation and support for deep network expansion. At the same time, through multi-path collaborative modeling, the input features are strengthened and reconstructed from different perspectives, which significantly improves the model's ability to express complex data patterns and its generalization performance without significantly increasing the number of parameters.

[0038] The matrix-vector inner product operation unit calculates the pulse similarity between the query pulse sequence and the key pulse sequence, generating a pulse attention map. Specifically, the process includes: performing matrix multiplication on the query pulse sequence and the key pulse sequence to obtain a similarity matrix, which is then used as the pulse attention map. Since both the query and key pulse sequences are binary pulse sequences with non-negative values, feature similarity can be directly measured without the need for Softmax normalization. This process can be represented as: (4); in, This is a pulse attention map. This represents the conjugate transpose of a matrix.

[0039] The pulse attention map and the value pulse sequence are multiplied by a matrix multiplication operation, and then multiplied by a scaling factor to complete feature aggregation, outputting the intermediate features from the attention process. : (5); in, The impulse attention map generated by formula (4), The value pulse sequence generated by formula (2), This is the scaling factor.

[0040] The self-attention intermediate features are pulse-mapped by the fifth spiking neural network, then linearly transformed by the fourth linear transformation layer, normalized by the fourth batch normalization layer, and pulse-mapped by the sixth spiking neural network to obtain the current self-attention features. (6); In the formula, Indicates the current self-attention features, This represents the mapping function of the fifth spiking neural network. This represents the mapping function of the fourth linear transformation layer. This represents the mapping function for the fourth batch normalization layer. This represents the mapping function of the sixth spiking neural network.

[0041] The DPSER-MLP module aims to enhance the feature representation capability after PCSA processing. Compared with the original MLP module in Spikformer, traditional LIF-based MLP structures struggle to fully extract multi-level spatial features due to features such as feature information loss caused by the sparsity of pulse firing during feature transmission in LIF neurons and insufficient spatial context modeling capabilities. To address this, this application introduces a dual-pooling squeezed excitation residual network. Its advantage lies in the synergistic integration of global and local spatial features through a dual-pooling mechanism and the use of channel attention in the SE module for feature recalibration, thereby significantly enhancing the network's ability to capture spatial structural features and its representation robustness.

[0042] refer to Figure 6 As shown, the DPSER-MLP module includes three parallel second branches and a second fusion layer; the outputs of the three second branches are connected to the input of the second fusion layer. The three second branches are a second cyclic linear transformation branch, a second identity mapping branch, and a feature transformation branch, respectively. In the current dual-pooling residual channel attention multilayer perceptron module, the second cyclic linear transformation branch consists of a second cyclic linear layer, which is used to input the current first fused feature and perform a linear transformation on it to obtain the second linear feature; the second identity mapping branch directly outputs the current first fused feature. The feature transformation branch consists of two feature transformation sub-modules, which are used to perform nonlinear transformation and channel feature recalibration on the current first fused feature to obtain the current semantic feature; the second fusion layer is used to fuse the second linear feature, the current first fused feature and the current semantic feature to obtain the current high-level pulse feature; wherein, the current high-level pulse feature output by the last dual-pooling residual channel attention multilayer perceptron module is the final high-level pulse feature.

[0043] Continue to refer to Figure 6 Both feature transformation submodules are composed of a fifth linear transformation layer, a fifth batch normalization layer, a second dual-pooling residual channel attention module, and a seventh spiking neural network connected in sequence. The input of the first feature transformation submodule is the current first fused feature. The input of the second feature transformation submodule is the output of the first feature transformation submodule. The current first fused feature undergoes linear transformation by the fifth linear transformation layer, normalization by the fifth batch normalization layer, nonlinear transformation and channel feature recalibration by the second dual-pooling residual channel attention module, and pulse mapping by the seventh spiking neural network to obtain the current semantic feature. The current second fused feature output by the last dual-pooling residual channel attention multilayer perceptron module is the final high-level pulse feature.

[0044] The second recurrent linear layer performs a recurrent linear transformation on the input first fused feature (i.e., in the same way as the recurrent linear layer in the PCSA module), outputting a second linear feature. The output of the second recurrent linear layer is then added element-wise with the calibrated feature through a residual connection, and this sum is used as the output of the current feature transformation submodule. (7); in, The current first fused feature is the input to the current feature transformation submodule (for the first feature transformation submodule). This represents the current semantic feature output by the seventh spiking neural network. This is the output of the current dual-pooling residual channel attention multilayer perceptron module. Indicates to The second linear characteristic is obtained by performing a cyclic linear transformation.

[0045] refer to Figure 7The first dual-pooling residual channel attention module and the second dual-pooling residual channel attention module have the same internal structure. Each of the first dual-pooling residual channel attention module and the second dual-pooling residual channel attention module receives the normalized features output from the batch normalization layer connected to it, and performs global average pooling and global max pooling operations to obtain the first global average pooling feature and the first global max pooling feature. The first global average pooling feature and the first global max pooling feature are fused in the channel dimension, and then sequentially passed through the first fully connected layer for dimensionality reduction, ReLU function activation, the second fully connected layer for dimensionality increase, and Sigmoid activation function mapping to generate the first channel weight vector. The first channel weight vector is multiplied by each channel of the input normalized feature to recalibrate the input normalized feature. Then, the recalibrated feature is added element-wise to the input normalized feature through residual connections to output the calibrated feature.

[0046] Before describing the specific processing steps of the DPSER (Dual Pooling Residual Channel Attention) module, let's first explain the basic principles of the channel attention mechanism. Figure 7 The diagram illustrates the structure of the first and second DPSER modules. The DPSER module first performs compression operations on the input feature map using parallel branches of AP (Average Pooling) and MP (Max Pooling), capturing global statistical features and local salient features respectively. Then, it fuses these two features along the channel dimension, sequentially passing them through a first fully connected layer for dimensionality reduction, a ReLU activation function, a second fully connected layer for dimensionality increase, and a Sigmoid activation function for mapping, generating channel weight vectors. Finally, it performs feature scaling, multiplying the channel weight vectors channel-by-channel with each channel of the original input feature map to recalibrate the features. The recalibrated features are then added element-by-element to the original input feature map via residual connections before outputting the final result.

[0047] The original SE-ResNet (Squeeze-Excitation Residual Network) is a Seblock (Squeeze-and-Excitation Block) network that improves the model's ability to focus on important features by adaptively adjusting the channel weights of the feature map. Seblock consists of three main parts: First, the Squeeze operation compresses the feature map spatially using global average pooling, resulting in a 1×1 vector with the same number of channels, thus capturing global information for each channel. Next, the Excitation operation models the channel information using a bottleneck structure composed of two fully connected layers. The first fully connected layer compresses the number of channels, and the second fully connected layer restores the number of channels to their original value. This learns the dependencies between channels and outputs a weight vector representing the importance of each channel. Finally, the recalibration operation multiplies the learned weight vector with each channel of the original feature map, recalibrating the features to enhance the feature responses of important channels and suppress the features of unimportant channels. Introducing residual connections can effectively solve the gradient vanishing problem that occurs as network depth increases. It allows gradients to propagate more smoothly to earlier layers through the residual path during backpropagation, making model training easier to converge and greatly accelerating the training speed. At the same time, it allows the network to be built deeper, avoiding the performance degradation caused by increasing the number of network layers. Furthermore, residual connections help the model learn richer features, enhance the model's generalization ability, and reduce overfitting.

[0048] In the original SEblock module, traditional global average pooling only aggregates the global mean of the feature maps. While this effectively captures global statistical information, it inevitably loses salient features in local regions of the feature maps, resulting in insufficient sensitivity of the channel attention mechanism to discriminative features. To overcome this limitation, this application integrates AP and MP to replace the original average pooling layer, forming the DPSER module, the structure of which is as follows: Figure 7 As shown.

[0049] Global average pooling fully extracts global semantic information from feature maps, providing a stable statistical feature foundation for the channel attention mechanism; global max pooling precisely focuses on the local regions with the strongest responses in the feature maps, enhancing the ability to extract key features. The synergistic effect of these two pooling strategies expands the network's perception of features from a single global average response to a multi-level representation of global and local features, thereby generating more discriminative and representative channel weights in the channel attention mechanism, significantly enhancing SEblock's adaptive calibration capability for complex features. This improvement, without changing the number of parameters, not only improves the model's accuracy in capturing diverse data patterns but also enhances its robustness in different scenarios.

[0050] Global average pooling features and global max pooling features are fused along the channel dimension, and then successively passed through a first fully connected layer for dimensionality reduction, a second fully connected layer for dimensionality increase, and a sigmoid activation function for mapping to generate channel weight vectors. Let the feature map input to the DPSER module be... Global average pooling features are The global max pooling feature is After merging, the two layers pass through the first fully connected layer in sequence. Second fully connected layer and Sigmoid activation function Processing to generate channel weight vectors : (8); in, This indicates feature fusion along the channel dimension.

[0051] The channel weight vector is multiplied channel by channel of the normalized feature to recalibrate the normalized feature. Then, the recalibrated feature is added element-wise to the normalized feature via a residual join, outputting the calibrated feature. This process can be represented as: (9); in, This indicates multiplication by channel. This is the calibrated feature map.

[0052] The classification head of this application includes a linear layer. The classification head aggregates the high-level impulse features output by the DPSER-MLP module, maps them to the classification category dimension through the linear layer (e.g., two categories in the oral lesion classification task: benign and malignant), outputs the predicted probability of each category through the Softmax activation function, and takes the category with the highest probability value as the lesion category label corresponding to the oral image.

[0053] In this application, all spiking neurons in the spiking neural network are LIF spiking neurons. The working mechanism of a LIF neuron is as follows: it continuously receives input signals and accumulates membrane potential. When the membrane potential reaches a preset firing threshold, it outputs a pulse signal and resets the membrane potential. When the membrane potential does not reach the preset firing threshold, there is no pulse output. This mechanism simulates the "integration-leakage-firing" characteristics of biological neurons and is the foundation for realizing event-driven sparse computing.

[0054] In one specific embodiment of this application, the training process of the trained enhanced impulsive Transformer model is as follows: S01, Obtain the training dataset, which contains multiple original oral cavity image samples and a category label corresponding to each original oral cavity image sample. The original oral cavity image samples are images of various oral lesions collected from an oral cavity detector, and the category label is the type of oral lesion (benign, malignant). In this embodiment, the training dataset can be derived from desensitized data collected clinically in a dental hospital, or it can be a publicly available oral lesion image dataset.

[0055] S02, each original oral cavity image sample in the training dataset is input into the preset enhanced pulse Transformer model. After being processed sequentially by the dual-pooling residual channel attention input patch embedding module, the parallel cyclic pulse attention module, the dual-pooling residual channel attention multilayer perceptron module, and the classification head, the corresponding prediction classification result is output.

[0056] S03, calculate the loss value based on the predicted classification results and corresponding category labels of each original oral cavity image sample. The loss function can be the cross-entropy loss function: (10); in, For the first The true category label of the original oral cavity image sample. For the first The predicted probability of a sample of original oral cavity images.

[0057] S04. Update the network parameters of the pre-set enhanced impulsive Transformer model according to the loss value, and iterate through steps S02 to S04 until the model converges to obtain the trained enhanced impulsive Transformer model.

[0058] In this embodiment, the training process is performed on an NVIDIA GPU (NVIDIA Graphics Processing Unit), and the software environment uses the Python programming language and the deep learning frameworks PyTorch, SpikingJelly, and the PyTorch image model library Timm. Backpropagation and gradient descent optimizers are used to update network parameters during training.

[0059] Taking a time step of 4, a batch size of 128, an input image of a 3-channel oral cavity with a height and width of 32 pixels as an example, and a final classification of 2 types (benign and malignant), the dimensionality transformation of the data in the enhanced impulse Transformer network is shown in Table 1 below: Table 1 Data Dimension Transformation

[0060] This application validates its method on an oral lesion image dataset. Experimental results show that the proposed enhanced pulse Transformer model achieves superior classification accuracy compared to the existing Spikformer model in the task of classifying benign / malignant oral lesions.

[0061] It should be noted that the application scenarios of this application are not limited to oral cavity image classification, but can also be widely applied to various image classification tasks such as static image classification and neuromorphic event stream image classification. The following examples demonstrate the performance of this application on different types of image classification tasks.

[0062] Example 1: Still Image Classification Task This embodiment uses the CIFAR10 and CIFAR100 datasets for validation. Both CIFAR10 and CIFAR100 are 32×32 resolution image datasets, providing 50,000 training samples and 10,000 test samples, respectively. The former contains 10 categories, and the latter contains 100 categories. Similar to Spikformer, the batch size is set to 128, and a four-module DPSER-IPE is used (only the last two modules contain average pooling layers) to segment the input image into 64 4×4 pixel blocks.

[0063] Table 2. Performance comparison of this application (AugSpikformer) and the traditional solution (Spikformer) on the CIFAR10 / 100 dataset.

[0064] Table 2 shows the performance comparison between AugSpikformer (this application) and Spikformer (the traditional approach) on the CIFAR10 / 100 dataset. AugSpikformer-4-256 and AugSpikformer-2-384 outperformed their respective Spikformer models by 1.13% and 1.02% on the CIFAR-10 dataset, and by 1.47% and 1.33% on the CIFAR-100 dataset, respectively. At the same time step size, AugSpikformer-4-384 significantly outperforms Spikformer-4-384 on the CIFAR dataset. With step sizes of 1, 2, 4, and 6, AugSpikformer-4-384 achieves 1.22%, 1.90%, 0.74%, and 0.82% higher accuracy on CIFAR-10, respectively, and 2.11%, 2.13%, 1.55%, and 1.18% higher accuracy on CIFAR-100. Notably, with a step size of 2, AugSpikformer-4-384 surpasses Spikformer-4-384 with a step size of 6 on CIFAR-10, and also outperforms Spikformer-4-384 with a step size of 4 on CIFAR-100. Furthermore, AugSpikformer-2-384 achieved a 0.63% improvement in accuracy despite having significantly fewer parameters than Spikformer-4-384. AugSpikformer's ability to maintain excellent performance while reducing time steps and parameter size validates the effectiveness of the designed model.

[0065] At different time steps (T=1, 2, 4, 6), AugSpikformer-4-384 achieved accuracies of 94.73%, 95.49%, 95.93%, and 96.16% on CIFAR-10, respectively, and 76.47%, 78.41%, 79.41%, and 79.79% on CIFAR-100, respectively. The larger the time step, the more information the network can extract, and the higher the accuracy becomes.

[0066] Example 2: Neuromorphic event flow image classification: This embodiment uses two neuromorphic vision datasets for validation: CIFAR10-DVS (CIFAR10-Dynamic Vision Sensor dataset) and DVS128-Gesture dataset. CIFAR10-DVS is a dataset generated by converting raw CIFAR10 static images into a stream of impulse events using a dynamic vision sensor. It contains 10,000 samples, with 9,000 used as the training set and 1,000 as the test set. DVS128-Gesture is a dataset of 11 types of hand gestures performed by 29 subjects under three different lighting conditions, totaling 1,342 samples, including 940 training samples and 422 test samples.

[0067] For these two 128×128 image datasets, a four-module SPS architecture was adopted, with max pooling layers only set in the last two SPS modules to highlight salient features. The patch embedding dimension was set to 256, and the patch size was 16×16. A shallow Spikformer model (containing two Transformer encoding blocks) was used, with the number of self-attention heads configured as 8 (DVS128 Gesture) and 16 (CIFAR10-DVS), respectively. The time step of the spiking neurons was set to 10 or 16, and the training epochs were 200 (DVS128 Gesture) and 106 (CIFAR10-DVS), respectively.

[0068] Table 3 Performance comparison of this application (AugSpikformer) and the traditional solution (Spikformer) on the CIFAR10-DVS / DVS128Gseture dataset.

[0069] Table 3 shows the performance comparison between AugSpikformer and Spikformer on the CIFAR10-DVS and DVS128 Gesture datasets. At a small time step (T=10), AugSpikformer improves accuracy by 0.7% on both the CIFAR10-DVS and DVS128 Gesture datasets compared to Spikformer, indicating that AugSpikformer can effectively extract features from the input data even at shorter time steps. With a time step of 16, AugSpikformer also improves accuracy by 0.1% and 0.3% on the CIFAR10-DVS and DVS128 Gesture datasets respectively, validating its stability and superiority across different time step settings.

[0070] Example 3 Ablation Experiment: This embodiment verifies the performance contribution of each improved module through ablation experiments. Taking the CIFAR-100 dataset as an example, the accuracy improvement and parameter changes of the basic Spikformer model, the model with only max pooling in the SPS layer replaced by average pooling, the model with only recurrent linear layers, the model with only the DPSER module, and the AugSpikformer model which integrates the above three innovations are discussed respectively. See Table 4: Table 4 Performance comparison of this application (AugSpikformer) and the traditional solution (Spikformer) on different architectures.

[0071] Experimental results show that the basic Spikformer (Spikformer-4-384) achieves an accuracy of 77.86% with 9.36M parameters. Replacing max pooling with average pooling in the SPS layer keeps the number of parameters in the AugSpikformer-AP-4-384 model unchanged, while improving accuracy by 0.07%. This indicates that average pooling better preserves features useful for classification tasks and reduces feature loss or bias that max pooling might cause. The AugSpikformer-CL-4-384 model, using a recurrent linear layer structure, increases the number of parameters to 9.38M (an increase of only 0.02M) while improving accuracy by 0.63%, demonstrating that this structure effectively enhances the model's expressive power. The AugSpikformer-DPSER-4-384 model, using only the DPSER module, increases the number of parameters to 10.66M and achieves an accuracy of 78.54%, showcasing the positive impact of the DPSER module on performance improvement. After integrating the three improvement strategies mentioned above, the AugSpikformer-4-384 model increased the number of parameters to 10.68M while achieving a maximum accuracy of 79.41%, which is 1.55% higher than the basic version. This verifies the significant performance improvement effect of multi-component collaborative optimization on model performance.

[0072] Figure 8 This demonstrates a comparison of the changes in loss function and accuracy for each modification on the CIFAR100 dataset. For example... Figure 8 As shown in (a), both the Base model and the AugSpikformer model have high loss values ​​in the early stages of training, but both show a decreasing trend as the number of training epochs increases. The AugSpikformer model's loss decreases faster, approaching a lower value in fewer epochs, and its final convergence value is significantly lower than that of the Base model, demonstrating its stronger fitting ability to the training data. Figure 8As shown in Figure (b), the AugSpikformer model achieves higher accuracy thanks to its superior loss function optimization process, demonstrating the significant effect of its structural improvements in enhancing model performance and effectively balancing model fitting and generalization capabilities.

[0073] In summary, the enhanced pulse Transformer model proposed in this application can significantly improve the classification accuracy of static images and neuromorphic event stream images with a slight increase in the number of parameters; it can also achieve improved accuracy while reducing the number of parameters. Furthermore, this model maintains excellent recognition performance even with a shorter time step, providing an effective technical solution for practical applications such as low-power edge vision and oral image-assisted diagnosis.

[0074] The above embodiments are merely preferred embodiments of this application and do not limit the patent scope of this application. Any equivalent structural or procedural transformations made based on the description and drawings of this application, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application. The above content is a further detailed description of this application in conjunction with specific preferred embodiments, and should not be construed as limiting the specific implementation of this application to these descriptions. For those skilled in the art, several simple deductions or substitutions can be made without departing from the concept of this application, and all such modifications or substitutions should be considered within the protection scope of this application.

Claims

1. A method for classifying oral cavity images based on an enhanced pulse transformer, characterized in that, include: S1. Acquire raw oral cavity images from an oral cavity detector and obtain a trained enhanced pulse Transformer model; wherein, the trained enhanced pulse Transformer model includes a dual-pooling residual channel attention input patch embedding module, a pulse position embedding layer, N parallel cyclic pulse attention modules, N dual-pooling residual channel attention multilayer perceptron modules, and a classification head; wherein, the parallel cyclic pulse attention modules and the dual-pooling residual channel attention multilayer perceptron modules are alternately connected; S2, the original oral cavity image is input into the trained enhanced pulse Transformer model to extract a structured pulse feature sequence from the original oral cavity image using the dual-pooling residual channel attention input patch embedding module. Position encoding information is superimposed onto the structured pulse feature sequence using the pulse position embedding layer. Then, the final high-level pulse features are alternately extracted from the structured pulse feature sequence carrying position encoding information using the N parallel cyclic pulse attention modules and the N dual-pooling residual channel attention multilayer perceptron modules. The classification head is used to classify the final high-level pulse features to obtain the lesion category label corresponding to the original oral cavity image.

2. The oral cavity image classification method based on enhanced pulse Transformer according to claim 1, characterized in that, The dual-pooling residual channel attention input patch embedding module includes four cascaded embedding sub-modules, with the output of the previous embedding sub-module serving as the input of the next embedding sub-module; each embedding sub-module includes a two-dimensional convolutional layer, a first batch normalization layer, a first dual-pooling residual channel attention module, and a first spiking neural network, and only the third and fourth embedding sub-modules have an average pooling layer. Each of the N parallel cyclic pulse attention modules includes three parallel first branches and a first fusion layer, and the outputs of the three first branches are connected to the input of the first fusion layer; Each of the N dual-pooling residual channel attention multilayer perceptron modules includes three parallel second branches and a second fusion layer; the outputs of the three second branches are connected to the input of the second fusion layer.

3. The oral cavity image classification method based on enhanced pulse Transformer according to claim 2, characterized in that, The two-dimensional convolutional layers in the first to fourth embedded sub-modules are used to perform convolution operations on the input features to obtain convolutional features; the first batch normalization layer is used to normalize the convolutional features to obtain normalized features; the first dual-pooling residual channel attention module is used to perform channel recalibration on the normalized features to obtain calibrated features; the first spiking neural network is used to input the calibrated features into the built-in LIF leaky integral firing spiking neurons, converting them into binary pulse time-series sequences; The average pooling layers in the third and fourth embedded sub-modules are used to perform average pooling downsampling on the pulse time sequence output by the corresponding first spiking neural network in the spatial dimension, compressing the spatial size to half of the original size.

4. The oral cavity image classification method based on enhanced pulse Transformer according to claim 2, characterized in that, The three first branches set in parallel in the N parallel cyclic pulse attention modules are the first cyclic linear transformation branch, the first identity mapping branch, and the pulse self-attention branch, respectively. The first cyclic linear transformation branch in the current parallel cyclic pulse attention module is composed of a first cyclic linear layer, used to perform a linear transformation on the current input feature to obtain the current first linear feature; the first identity mapping branch is used to directly output the current input feature; the pulse self-attention branch is composed of a pulse self-attention module, used to perform self-attention processing on the current input feature and output the current self-attention feature; the first fusion layer is used to fuse the current first linear feature, the current input feature, and the current self-attention feature and output the current first fused feature; wherein, the first parallel cyclic pulse attention module inputs a structured pulse feature sequence.

5. The oral cavity image classification method based on enhanced pulse Transformer according to claim 4, characterized in that, The pulse self-attention module in the current parallel cyclic pulse attention module includes three mapping sub-paths, a matrix-vector inner product operation unit, a feature scaling layer, a fifth spiking neural network, a fourth linear transformation layer, a fourth batch normalization layer, and a sixth spiking neural network. The three mapping sub-paths are a query sub-path, a key pulse sub-path, and a value pulse sub-path, respectively. The query sub-path, key pulse sub-path, and value pulse sub-path take the current input feature as input and output the input connected to the matrix-vector inner product operation unit. The matrix-vector inner product operation unit, feature scaling layer, fifth spiking neural network, fourth linear transformation layer, fourth batch normalization layer, and sixth spiking neural network are connected sequentially, and the sixth spiking neural network outputs the current self-attention feature. The current self-attention feature output by the sixth spiking neural network in the last parallel cyclic pulse attention module is the final self-attention feature.

6. The oral cavity image classification method based on enhanced pulse Transformer according to claim 5, characterized in that, The query sub-route is composed of a first linear transformation layer, a second batch normalization layer, and a second spiking neural network connected in sequence; the current input feature is linearly transformed by the first linear transformation layer and then input into the second batch normalization layer for normalization, and then mapped into a binary query pulse sequence by the second spiking neural network; The key pulse sub-routes are composed of a second linear transformation layer, a third batch normalization layer, and a third spiking neural network connected in sequence; the current input features are linearly transformed by the second linear transformation layer and then input into the third batch normalization layer for normalization, and then mapped into a binary key pulse sequence by the third spiking neural network; The value pulse sub-routes are composed of a third linear transformation layer, a fourth batch normalization layer, and a fourth spiking neural network connected in sequence; the outputs of the second, third, and fourth spiking neural networks are connected to the input of the matrix-vector inner product operation unit; the current input features are linearly transformed by the third linear transformation layer and then input to the fourth batch normalization layer for normalization, and then mapped into a binary value pulse sequence by the fourth spiking neural network.

7. The oral cavity image classification method based on enhanced pulse Transformer according to claim 6, characterized in that, The query pulse sequence, the key pulse sequence, and the value pulse sequence are subjected to matrix multiplication by the matrix vector inner product operation unit, and then scaled by the feature scaling layer to obtain self-attention intermediate features; The self-attention intermediate features are pulse-mapped by the fifth spiking neural network, then linearly transformed by the fourth linear transformation layer, normalized by the fourth batch normalization layer, and pulse-mapped by the sixth spiking neural network to obtain the current self-attention features.

8. The oral cavity image classification method based on enhanced pulse Transformer according to claim 2, characterized in that, The three second branches set in parallel in the N dual-pooling residual channel attention multilayer perceptron modules are the second cyclic linear transformation branch, the second identity mapping branch, and the feature transformation branch, respectively. In the current dual-pooling residual channel attention multilayer perceptron module, the second cyclic linear transformation branch consists of a second cyclic linear layer, which is used to input the current first fused feature and perform a linear transformation on it to obtain the second linear feature; the second identity mapping branch directly outputs the current first fused feature. The feature transformation branch consists of two feature transformation sub-modules, which are used to perform nonlinear transformation and channel feature recalibration on the current first fused feature to obtain the current semantic feature; the second fusion layer is used to fuse the second linear feature, the current first fused feature and the current semantic feature to obtain the current high-level pulse feature; wherein, the current high-level pulse feature output by the last dual-pooling residual channel attention multilayer perceptron module is the final high-level pulse feature.

9. The oral cavity image classification method based on enhanced pulse Transformer according to claim 8, characterized in that, Both feature transformation submodules are composed of a fifth linear transformation layer, a fifth batch normalization layer, a second dual-pooling residual channel attention module, and a seventh spiking neural network connected in sequence; The input to the first feature transformation submodule is the current first fused feature; The input of the second feature transformation submodule is the output of the first feature transformation submodule. The current first fused feature is linearly transformed by the fifth linear transformation layer, normalized by the fifth batch normalization layer, nonlinearly transformed and recalibrated by the second dual-pooling residual channel attention module, and the current semantic feature is obtained by pulse mapping by the seventh spiking neural network. Among them, the current second fusion feature output by the last dual-pooling residual channel attention multilayer perceptron module is the final high-level impulse feature.

10. The oral cavity image classification method based on enhanced pulse Transformer according to claim 9, characterized in that, The first dual-pooling residual channel attention module and the second dual-pooling residual channel attention module have the same internal structure. Each of the first dual-pooling residual channel attention module and the second dual-pooling residual channel attention module receives the normalized features output from the batch normalization layer connected to it, and performs global average pooling and global max pooling operations to obtain the first global average pooling feature and the first global max pooling feature. The first global average pooling feature and the first global max pooling feature are fused in the channel dimension, and then sequentially passed through the first fully connected layer for dimensionality reduction, ReLU function activation, the second fully connected layer for dimensionality increase, and Sigmoid activation function mapping to generate the first channel weight vector. The first channel weight vector is multiplied by each channel of the input normalized feature to recalibrate the input normalized feature. Then, the recalibrated feature is added element-wise to the input normalized feature through residual connections to output the calibrated feature.

Images