A SNN multi-modal target recognition method, system, device and medium
By introducing a convolutional time alignment module and a modality adjustment factor into the SNN, the problems of modality imbalance and time scale mismatch in the multimodal model are solved, thereby improving the accuracy and efficiency of multimodal target recognition.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- ZHEJIANG UNIV
- Filing Date
- 2024-01-16
- Publication Date
- 2026-07-21
AI Technical Summary
Existing SNN multimodal algorithms suffer from modality imbalance and time scale mismatch, which limits their effectiveness in multimodal tasks.
The key features of multimodal information are extracted by a feature extractor, the time scale is adjusted by a convolutional time alignment module, and modality adjustment factor and cross-entropy loss are introduced to dynamically adjust the loss function to solve the problems of modality imbalance and time scale mismatch.
It significantly improves the overall performance of multimodal models in target recognition tasks, and realizes full utilization and accurate recognition of modal information.
Smart Images
Figure CN117892175B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of artificial intelligence, and in particular to an SNN multimodal target recognition method, system, device, and medium. Background Technology
[0002] In today's information explosion era, multimodal learning has become an important branch of artificial intelligence. Multimodal learning integrates information from different modalities to build models with better performance and stronger robustness. Multimodal models utilize multiple information sources such as text, images, and sound, and are applied in various complex tasks such as visual question answering, sentiment analysis, medical image processing, and cross-modal retrieval. With the continuous development of Spiking Neural Networks (SNN) training algorithms, the applicability of SNNs in various application scenarios has gradually increased, and their high biological interpretability has also shown certain advantages in handling multimodal problems. On the one hand, the neurons in SNNs simulate the dynamic behavior of real neurons, enabling them not only to learn the spatial features of multimodal data but also to efficiently utilize the temporal information of modalities, making them particularly suitable for processing data containing a time dimension, such as video and audio. On the other hand, the event-driven nature of SNNs allows them to remain efficient when processing sparse modal data, significantly reducing the energy consumption of multimodal processing. Researchers have already used SNNs to process event, visual, and audio data to complete tasks such as lip reading, speech recognition, and object recognition. Reference 1 utilizes information fusion from a Dynamic Vision Sensor (DVS) and a Dynamic Audio Sensor (DAS) to achieve lip-reading recognition. It filters out noisy events by calculating the cross-correlation between different modal event streams, achieving intermodal alignment and addressing the time offset problem in audiovisual event stream recording. Furthermore, Reference 2 borrows gating logic, employing excitatory and inhibitory synaptic connections to form a supermodal layer, achieving cross-modal information coupling. Another study, Reference 3, uses convolutional spiking neural networks (SNNs) and recurrent spiking neural networks (RNNs) to process DVS and DAS data respectively, and employs an attention mechanism to weight the output of the single-modal network to achieve selective fusion. Multimodal algorithms based on SNNs significantly improve the overall performance of the models across various tasks.
[0003] Compared to unimodal models, the complementary information from different modalities in multimodal models can often help optimize model performance and improve the model's robustness to noise and missing data. However, existing SNN multimodal algorithms still suffer from modality imbalance and time scale mismatch, which limit the effectiveness of SNNs in multimodal tasks.
[0004] Modality imbalance refers to a situation in multimodal joint learning scenarios where the performance of the optimal single-modal model outperforms the multimodal model, and the performance degradation of the multimodal model becomes more significant as the number of input streams increases. Even if the multimodal model's joint training effect is better than the single-modal model, its single-modal branch rarely surpasses the performance of the single-modal model trained alone. Modality imbalance makes it difficult for single-modal branches to converge completely, thus weakening the multimodal model's ability to utilize information from different modalities. From the perspective of the optimization process, multimodality imbalance is mainly caused by the difference in convergence speed among different modalities. When training multimodal models jointly, using a uniform learning rate may lead to the faster-converging modality dominating the entire optimization process, while other modalities fail to receive sufficient training. Therefore, most existing algorithms addressing the modality imbalance problem regulate the learning process by monitoring the convergence of different modalities, aiming to slow down the optimization speed of the dominant modality to alleviate the underfitting problem of other modalities, thereby improving the overall performance of the multimodal model. For example, Reference 4 first proposed an offline gradient fusion adjustment algorithm, which adjusts the loss function of each modality by calculating the ratio of overfitting to generalization ability. However, this algorithm can only achieve offline adjustment and cannot optimize in real time. Reference 5 introduces an adaptive tracking factor to adjust the learning rate of each modality in real time, so that the learning rate of the modality closer to convergence is lower. To measure the learning dependence of the multimodal model on each modality, Reference 6 proposes the concept of conditional utilization and optimizes it as conditional learning speed to achieve online adjustment. Reference 7 proposes an online gradient adjustment algorithm, which dynamically monitors the difference in contribution of different modalities to the learning objective during the training phase, and then uses this difference to adaptively adjust the gradient, and introduces Gaussian noise to achieve generalization enhancement. However, the online gradient adjustment algorithm cannot effectively adjust the gradient when the accuracy difference between modalities is large, which limits the improvement of multimodal fusion effect.
[0005] In applying Sub-Neural Networks (SNNs) to handle multimodal tasks, besides the modality imbalance problem caused by differences in convergence speed among different modalities, the timescale mismatch and varying event sparsity of different modalities in SNNs also pose greater challenges to the full utilization of multimodal information. The main reason for the timescale mismatch is that the information from different SNN modalities has different sensitivities to time. Static visual modalities themselves do not involve a time dimension, and using repeated static visual information across multiple time steps may lead to information redundancy. Dynamic visual modal data inherently possesses high temporal resolution and dynamic change characteristics, requiring a certain number of time steps to capture rapid and fine-grained changes in the event stream. Auditory modalities typically contain rich temporal dimension information, requiring more time steps to accurately capture sound changes, especially when processing complex audio scenes. However, existing multimodal SNN algorithms usually allocate the same number of time steps to all modalities without effectively addressing the timescale mismatch between modalities. This approach can lead to overexpression or loss of information from certain modalities, further increasing the difficulty of solving the modality imbalance problem.
[0006] References:
[0007] Literature 1. Li,
[0008] Document 2. Wysoski, SG, Benuskova, L. & Kasabov, N. Brain-like evolving spiking neural networks for multimodal information processing. in Brain-InspiredInformation Technology 15-27 (Springer, 2010).
[0009] Reference 3. Liu, Q., Xing, D., Feng, L., Tang, H. & Pan, G. Event-based multimodal spiking neural network with attention mechanism. in ICASSP 2022 - 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) 8922 - 8926 (IEEE, 2022).
[0010] Reference 4. Wang, W., Tran, D. & Feiszli, M. What makes training multi-modal classification networks hard? in Proceedings of the IEEE / CVF conference on computer vision and pattern recognition 12695 - 12705 (2020).
[0011] Reference 5. Sun, Y., Mai, S. & Hu, H. Learning to balance the learning rates between various modalities via adaptive tracking factor. IEEE Signal Process. Lett. 28, 1650 - 1654 (2021).
[0012] Reference 6. Wu, N., Jastrzebski, S., Cho, K. & Geras, K. J. Characterizing and overcoming the greedy nature of learning in multi-modal deep neural networks. in International Conference on Machine Learning 24043 - 24055 (PMLR, 2022).
[0013] Document 7. Peng, Summary of the Invention
[0014] The purpose of this invention is to provide an SNN multimodal target recognition method, system, device, and medium to solve the problems of multimodal convergence imbalance and time scale mismatch.
[0015] To achieve the above objectives, the present invention provides the following solution:
[0016] A multimodal target recognition method using SNN includes:
[0017] Key features are extracted from multimodal information using a feature extractor; these key features include visual and auditory information; the feature extractor includes a visual feature extractor and an auditory feature extractor.
[0018] The convolutional temporal alignment module is used to adjust the temporal scale of key features corresponding to different single-modal information to determine the temporally aligned single-modal key features; the convolutional temporal alignment module includes a visual convolutional temporal alignment module and an auditory convolutional temporal alignment module;
[0019] Each time-aligned single-modal key feature is simulated by forward propagation with the multimodal classifier parameters through matrix multiplication to obtain single-modal output and multimodal output. The contribution ratio of different single-modal outputs to the target task is evaluated based on the single-modal output and the multimodal output to determine the modality adjustment factor.
[0020] The cross-entropy loss corresponding to the single-modal output and the multimodal output is determined based on the single-modal output and the multimodal output, and the loss function is dynamically adjusted based on the modality adjustment factor and the cross-entropy loss to determine the final loss of the spiking neural network, so as to identify the multimodal target of the spiking neural network; the final loss is the optimization target of the spiking neural network.
[0021] Optionally, key features in the multimodal information are extracted using a feature extractor, which previously included:
[0022] The visual information and the auditory information are encoded respectively to generate encoded visual information and encoded auditory information;
[0023] A multimodal dataset is constructed based on the encoded visual information and the encoded auditory information; the multimodal dataset includes multimodal information.
[0024] Optionally, key features in the multimodal information can be extracted using a feature extractor, specifically including:
[0025] The visual information is processed through a visual feature extractor consisting of three pulsed fully connected layers to generate T. e Visual features at each time step;
[0026] The auditory information is processed through an auditory feature extractor composed of a three-layer recurrent pulse network to generate T. a Auditory characteristics at each time step.
[0027] Optionally, the convolutional temporal alignment module is used to adjust the temporal scale of key features corresponding to different single-modal information to determine the temporally aligned single-modal key features. Specifically, the visual convolutional temporal alignment module includes T 1×1×T modules. e The convolution kernels; the auditory convolution time alignment module includes T 1×1×T kernels. a The convolution kernel;
[0028] The visual features are processed using a visual convolutional temporal alignment module to generate visual alignment features at T time steps;
[0029] The auditory features are processed using an auditory convolution time alignment module to generate auditory alignment features at T time steps.
[0030] Optionally, each time-aligned single-modal key feature is simulated by forward propagation with the multimodal classifier parameters through matrix multiplication to obtain single-modal output and multimodal output. The contribution ratio of different single-modal outputs to the target task is evaluated based on the single-modal output and the multimodal output to determine the modality adjustment factor, specifically including:
[0031] According to the formula Determine the multimodal output; where, For multimodal output; W represents the parameters of the fully connected layer; Visual alignment features; b represents auditory alignment features; b represents the bias term.
[0032] According to the formula Determine the visual modality output; where, For visual modal output; W e For visual classifier parameters;
[0033] According to the formula Determine the auditory modal output; where, For auditory modal output; W a For auditory classifier parameters;
[0034] The visual modal output and the auditory modal output are input into the softmax function to determine the visual modal score and the auditory modal score;
[0035] The visual contribution ratio of the visual modality to the target task and the auditory contribution ratio of the auditory modality to the target task are determined based on the visual modality score and the auditory modality score.
[0036] The modality adjustment factor is determined based on the visual contribution ratio and the auditory contribution ratio.
[0037] Optionally, the cross-entropy loss corresponding to the single-modal output includes the cross-entropy loss corresponding to the visual modality and the cross-entropy loss corresponding to the auditory modality;
[0038] The cross-entropy loss L corresponding to the visual modality e for: Where C is the number of sample types, j is the sample type index, and N is the number of samples; y i This represents the true class label of the i-th sample;
[0039] The cross-entropy loss L corresponding to the auditory modality a for:
[0040] The cross-entropy loss L corresponding to the multimodal output f for:
[0041] Optionally, the final loss L is: Where β is a hyperparameter, As an auditory modulation factor, It is a visual modulation factor.
[0042] A multimodal target recognition system using an SNN (Search Engine Neural Network) includes:
[0043] The feature extraction module is used to extract key features from multimodal information using a feature extractor; the key features include visual information and auditory information; the feature extractor includes a visual feature extractor and an auditory feature extractor.
[0044] The convolutional temporal alignment module is used to adjust the temporal scale of key features corresponding to different single-modal information and determine the temporally aligned single-modal key features.
[0045] The simulated feedforward module is used to simulate forward propagation by matrix multiplication of each time-aligned single-modal key feature with the multimodal classifier parameters to obtain single-modal output and multimodal output, and to evaluate the contribution ratio of different single-modal outputs to the target task based on the single-modal output and the multimodal output to determine the modality adjustment factor.
[0046] An online loss adjustment module is used to determine the cross-entropy loss corresponding to the single-modal output and the multimodal output based on the single-modal output and the multimodal output, and to dynamically adjust the loss function based on the modality adjustment factor and the cross-entropy loss to determine the final loss of the spiking neural network; the final loss is the optimization objective of the spiking neural network.
[0047] An electronic device includes a memory and a processor, the memory storing a computer program, and the processor running the computer program to enable the electronic device to perform the above-described SNN multimodal target recognition method.
[0048] A computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described SNN multimodal target recognition method.
[0049] According to specific embodiments provided by the present invention, the present invention discloses the following technical effects:
[0050] This invention modifies the traditional multimodal model loss function by introducing an additional cross-entropy loss corresponding to the single-modal output. This cross-entropy loss serves as the supervised loss for single-modal feature extraction, ensuring that the invention not only focuses on the multimodal classification effect but also on the representation effect of the single-modal feature extractor, allowing the supervised model to fully learn information within each single modality. Furthermore, addressing the issue that some algorithms can only solve the mode imbalance problem offline, this invention borrows the method from online gradient adjustment algorithms for evaluating the contribution of single modes to the target. A mode adjustment factor is introduced before the single-modal loss to reduce the learning rate of the dominant mode that converges faster online, thereby alleviating the mode imbalance problem. The multimodal model involved in this invention is an SNN multimodal model.
[0051] This invention also proposes a convolutional time alignment module, which adjusts the time scale of key features corresponding to different single-modal information by adding a fixed number of convolutional kernels after the feature extractor of each modality, determines the time-aligned single-modal key features, and adapts to the time step differences of different modal data.
[0052] This invention employs two key technologies—online loss adjustment and convolution time alignment—to effectively solve two major problems: multimodal convergence imbalance and time scale mismatch, thereby significantly improving the overall performance of multimodal models in target recognition tasks. Attached Figure Description
[0053] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0054] Figure 1 Here is a flowchart of the SNN multimodal target recognition method provided by the present invention;
[0055] Figure 2 This is a structural diagram of the SNN multimodal target recognition system provided by the present invention. Detailed Implementation
[0056] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0057] The purpose of this invention is to provide an SNN multimodal target recognition method, system, device, and medium that can effectively solve the problems of multimodal convergence imbalance and time scale mismatch, and significantly improve the overall performance of multimodal models in target recognition tasks.
[0058] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.
[0059] Example 1
[0060] like Figure 1 As shown, this invention provides an SNN multimodal target recognition method, comprising:
[0061] Step 101: Extract key features from multimodal information using a feature extractor; the key features include visual information and auditory information; the feature extractor includes a visual feature extractor and an auditory feature extractor.
[0062] In practical applications, step 101 is preceded by encoding preprocessing of the input multimodal information to adapt to the input requirements of multiple modal information.
[0063] Visual Information Encoding: In this multimodal target recognition system, the N-MNIST dataset is selected as the input for visual information modality. This dataset is obtained by sequentially displaying original handwritten images from the MNIST dataset on a slowly moving display and recording them using an event sensor. The N-MNIST dataset includes 60,000 training samples and 10,000 test samples, each sample being 34*34 pixels in size. When encoding visual event information, the following steps are taken: the event stream is divided into T elements approximately uniformly according to the total number N. e Each segment is an integral of the event stream within it, forming a frame. This process is used to ultimately generate T. e The visual input is calculated at 3 time steps, where the input format of each frame is defined as (t, p, h, w), where t represents the current time step, p represents the event polarity, and (h, w) represents the frame size. The final visual input is 2312 × T. e In this experiment, the time step T e Set to 5.
[0064] Auditory Information Encoding: To maintain consistency across multimodal categories, the auditory modalities in this multimodal target recognition system utilize 0-9 digit audio samples from the Google Voice Command dataset. This dataset contains 38,908 samples, divided into training and test sets in a 9:1 ratio. During the preprocessing of the auditory signals, each auditory sample is first converted into a Mel spectrogram. The number of frames in the spectrogram is equal to the given auditory modality time step T. a The third-order difference is calculated based on the spectrogram, thus forming a three-channel input data, which is then normalized across time steps. The final auditory input format is 120×T. a In this experiment, the time step T a The time step T of the subsequent multimodal fusion module is also set to 20.
[0065] Multimodal dataset construction: First, randomly select the class label y of the samples from 0 to 9. i Then, random sampling is performed within the data corresponding to the category labels of the visual and auditory modalities to obtain visual data. and auditory data Through this process, a training set D = {x} is constructed. i ,y i} {i＝1,2,…,N} .in The same method was used to construct the test set. In the experimental verification of this invention, the training set was set to 10,000 samples, while the test set was set to 2,000 samples. This step aims to ensure that the system can effectively process and identify sample data from different modalities, thereby verifying the system's practicality and performance.
[0066] Feature extraction is performed on the encoded information to obtain visual features. and auditory characteristics Feature extraction involves the following steps:
[0067] In this invention, the visual feature extractor φ e It consists of three pulsed fully connected layers of 2312-800-128, and its parameters are the learnable network parameters θ of the visual feature extractor. e This indicates that visual information, after passing through a feature extractor, yields T. e Visual features at each time step The formula is shown below:
[0068]
[0069] Auditory feature extractor φ a It consists of a three-layer recurrent pulse network of 120-240-128, whose parameters are the learnable network parameters θ of the auditory feature extractor. a This indicates that auditory information, after passing through a feature extractor, yields T. a Auditory characteristics at each time step The formula is shown below:
[0070]
[0071] Step 102: Use the convolutional time alignment module to adjust the time scale of key features corresponding to different single-modal information to determine the time-aligned single-modal key features; the convolutional time alignment module includes a visual convolutional time alignment module and an auditory convolutional time alignment module.
[0072] In practical applications, the alignment of modal features with different time scales is achieved, ultimately resulting in visual alignment features. Alignment features with auditory perception Convolutional temporal alignment involves the following steps:
[0073] Temporal alignment of visual features: Visual features are processed by a visual convolutional temporal alignment module to obtain visually aligned features at T time steps. This visual convolutional temporal alignment module consists of T 1×1×T... e The convolution kernels are composed of the following formulas:
[0074]
[0075] in, These represent the model parameters of the visual convolution temporal alignment module.
[0076] Auditory feature temporal alignment: Auditory features are processed by an auditory convolutional temporal alignment module to obtain auditory aligned features at T time steps. This auditory convolutional temporal alignment module consists of T 1×1×T... a The convolution kernels are composed of the following formulas:
[0077]
[0078] in, This represents the model parameters for the auditory convolution time alignment module.
[0079] The time alignment operation in this step ensures the consistency of different modal features over time, providing support for the effective operation of the multimodal target recognition system.
[0080] Step 103: Simulate forward propagation by matrix multiplication of each time-aligned single-modal key feature with the multimodal classifier parameters to obtain single-modal output and multimodal output. Evaluate the contribution ratio of different single-modal outputs to the target task based on the single-modal output and the multimodal output, and determine the modality adjustment factor.
[0081] In practical applications, multimodal and single-modal outputs are obtained by simulating the feedforward process, and the target contribution of different modes is calculated, including the following steps:
[0082] Calculating the multimodal output: The model classifier consists of 256-10 pulsed fully connected layers. W represents the parameters of the fully connected layer, and b is the bias term. The multimodal output is first obtained through feedforward. The calculation formula is:
[0083]
[0084] Simulated unimodal feedforward: To estimate the contribution of different modalities to the target, the original classifier is divided into two parts: a 128-10 visual classifier and a 128-10 auditory classifier, W = [W e W a ]. W e W a These are the parameters for the visual and auditory classifiers, respectively. Simulating visual and auditory feedforward classification yields the visual and auditory modal outputs.
[0085]
[0086]
[0087] Multimodal contribution ratio calculation: Calculate the single-mode output The scores s for different modalities are obtained by passing each value into the softmax function. es a The calculation formulas are as follows:
[0088]
[0089]
[0090] Where C represents the number of sample categories, j is the sample type index; y i Let i be the true class label of the i-th sample. Let j be the indicator function, and when j equals the true class y of sample i. i When the score is y, its value is 1; otherwise, it is 0. It ensures that only the correct category score is counted. In the formula, j = y i hour, The j-th component is added to L. e Then, the visual and auditory proportions of the current batch of samples are calculated based on the modal scores. and B t The sample set of the current batch is represented by the following formulas:
[0091]
[0092]
[0093] Modality adjustment factor calculation: If the scaling factor of the current mode is greater than 1, it indicates that the current mode contributes significantly to the overall objective and is the dominant mode; therefore, its optimization process should be slowed down by adjusting the scaling factor. Conversely, if the scaling factor is less than 1, it indicates that the current mode is a weak mode and should not be suppressed. This step yields the modality adjustment factor using the following formula. Where u = {e, a} represents a specific mode.
[0094]
[0095] Here, α is a hyperparameter used to control the degree of regulation; the larger α is, the stronger the suppression of the dominant mode.
[0096] Step 104: Determine the cross-entropy loss corresponding to the single-modal output and the multimodal output based on the single-modal output and the multimodal output, and dynamically adjust the loss function based on the modality adjustment factor and the cross-entropy loss to determine the final loss of the spiking neural network, so as to identify the multimodal target of the spiking neural network; the final loss is the optimization target of the spiking neural network.
[0097] In practical applications, the calculation of cross-entropy loss corresponding to single-modal output and multi-modal output, and the adjustment based on the regulation factor, includes the following steps:
[0098] Loss calculation: This step is based on the visual modality output. Auditory modal output and multimodal output Calculate the corresponding cross-entropy loss L respectively e L a and L f The formulas for calculating each loss are as follows.
[0099]
[0100]
[0101]
[0102] Loss adjustment: The final loss L of the model is calculated based on the weighted average of the target contribution.
[0103]
[0104] Where β is a manually set hyperparameter, representing the proportion of single-modal supervision loss in the overall loss function. As an auditory modulation factor, , where L is the visual moderating factor. During training, the loss L is used as the overall optimization objective of the model.
[0105] The present invention has the following advantages:
[0106] (1) Innovation: This invention is the first to propose a solution to the multimodal imbalance problem in spiking neural networks, effectively overcoming the problem that multimodal joint training may lead to a decrease in the feature extraction capability of single-modal models.
[0107] (2) High efficiency: By integrating two key technologies—an online loss adjustment module and a convolutional temporal alignment module—this invention achieves comprehensive optimization of feature extractors for different modalities. In the target recognition task on the example dataset, the accuracy is improved by 2.3%, thus verifying the high efficiency of this invention in multimodal learning.
[0108] (3) Online adjustment: The online loss adjustment module in this invention relies on the single-mode feedforward process to monitor the contribution of different modes to the target, effectively weakening the dominant role of strong modes in the overall optimization process, while alleviating the suppression of other mode optimization, thus achieving the effect of real-time loss adjustment.
[0109] (4) Time synchronization: The convolution time alignment module in this invention allows the system to process multiple modalities with different time scales, realize adaptive learning of modal information at different time scales, and achieve effective alignment of modal time scales, with good scalability.
[0110] (5) Information complementarity: Experiments on the target recognition task conducted by this invention on the example dataset show that the accuracy of the auditory modality is improved by 40.65% compared with the original system, and exceeds the training accuracy of the auditory single-modality model. This result proves that the adjustment algorithm of this invention can efficiently utilize cross-modal information and achieve information complementarity.
[0111] Example 2
[0112] In order to implement the method corresponding to Embodiment 1 above and achieve the corresponding functions and technical effects, an SNN multimodal target recognition system is provided below.
[0113] like Figure 2 As shown, an SNN multimodal target recognition system includes:
[0114] The feature extraction module is used to extract key features from multimodal information through a feature extractor; the key features include visual information and auditory information; the feature extractor includes a visual feature extractor and an auditory feature extractor.
[0115] The convolutional temporal alignment module is used to adjust the temporal scale of key features corresponding to different single-modal information and determine the temporally aligned single-modal key features.
[0116] The simulated feedforward module is used to simulate forward propagation of each time-aligned single-modal key feature with the multimodal classifier parameters through matrix multiplication to obtain single-modal output and multimodal output. Based on the single-modal output and the multimodal output, the contribution ratio of different single-modal outputs to the target task is evaluated, and the modality adjustment factor is determined.
[0117] An online loss adjustment module is used to determine the cross-entropy loss corresponding to the single-modal output and the multimodal output based on the single-modal output and the multimodal output, and to dynamically adjust the loss function based on the modality adjustment factor and the cross-entropy loss to determine the final loss of the spiking neural network; the final loss is the optimization objective of the spiking neural network.
[0118] Example 3
[0119] An electronic device includes a memory and a processor, the memory storing a computer program, and the processor running the computer program to enable the electronic device to perform the SNN multimodal target recognition method described above.
[0120] A computer-readable storage medium storing a computer program that, when executed by a processor, implements the SNN multimodal target recognition method described above.
[0121] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the systems disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the descriptions are relatively simple; relevant parts can be referred to the method section.
[0122] This document uses specific examples to illustrate the principles and implementation methods of the present invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of the present invention. Furthermore, those skilled in the art will recognize that, based on the ideas of the present invention, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of the present invention.
Claims
1. A multimodal target recognition method using SNN, characterized in that, include: Key features are extracted from multimodal information using a feature extractor; these key features include visual and auditory information; the feature extractor includes a visual feature extractor and an auditory feature extractor. The convolutional temporal alignment module is used to adjust the temporal scale of key features corresponding to different single-modal information to determine the temporally aligned single-modal key features; the convolutional temporal alignment module includes a visual convolutional temporal alignment module and an auditory convolutional temporal alignment module; The visual convolutional temporal alignment module includes T 1×1× The convolution kernel; the auditory convolution time alignment module includes T 1×1× The convolution kernel; The visual features are processed using a visual convolutional temporal alignment module to generate visual alignment features at T time steps; The auditory features are processed using an auditory convolution time alignment module to generate auditory alignment features at T time steps; Each time-aligned single-modal key feature is simulated by forward propagation with the multimodal classifier parameters through matrix multiplication to obtain single-modal output and multimodal output. The contribution ratio of different single-modal outputs to the target task is evaluated based on the single-modal output and the multimodal output to determine the modality adjustment factor, specifically including: According to the formula Determine the multimodal output; where, Multimodal output; These are the parameters for the fully connected layer; Visual alignment features; b represents auditory alignment features; b represents the bias term. According to the formula Determine the visual modality output; where, Output for visual modality; For visual classifier parameters; According to the formula Determine the auditory modal output; where, For auditory modal output; For auditory classifier parameters; The visual modal output and the auditory modal output are input into the softmax function to determine the visual modal score and the auditory modal score; The visual contribution ratio of the visual modality to the target task and the auditory contribution ratio of the auditory modality to the target task are determined based on the visual modality score and the auditory modality score. The modality adjustment factor is determined based on the visual contribution ratio and the auditory contribution ratio; The cross-entropy loss corresponding to the single-modal output and the multimodal output is determined based on the single-modal output and the multimodal output, and the loss function is dynamically adjusted based on the modality adjustment factor and the cross-entropy loss to determine the final loss of the spiking neural network, so as to identify the multimodal target of the spiking neural network; the final loss is the optimization target of the spiking neural network.
2. The SNN multimodal target recognition method according to claim 1, characterized in that, Previously, key features were extracted from multimodal information using feature extractors, which also included: The visual information and the auditory information are encoded respectively to generate encoded visual information and encoded auditory information; A multimodal dataset is constructed based on the encoded visual information and the encoded auditory information; the multimodal dataset includes multimodal information.
3. The SNN multimodal target recognition method according to claim 1, characterized in that, Key features are extracted from multimodal information using a feature extractor, specifically including: The visual information is then processed by a visual feature extractor consisting of three pulsed fully connected layers to generate... Visual features at each time step; The auditory information is processed through an auditory feature extractor composed of a three-layer recurrent pulse network to generate... Auditory characteristics at each time step.
4. The SNN multimodal target recognition method according to claim 1, characterized in that, The cross-entropy loss corresponding to the single-modal output includes the cross-entropy loss corresponding to the visual modality and the cross-entropy loss corresponding to the auditory modality; The cross-entropy loss corresponding to the visual modality for: Where C is the number of sample types, j is the sample type index, and N is the number of samples; This represents the true class label of the i-th sample; Cross-entropy loss corresponding to the auditory modality for: ; The cross-entropy loss corresponding to the multimodal output for: .
5. The SNN multimodal target recognition method according to claim 4, characterized in that, The final loss for: ;in, For hyperparameters, As an auditory modulation factor, It is a visual modulation factor.
6. An SNN multimodal target recognition system, characterized in that, The method for performing the SNN multimodal target recognition method according to any one of claims 1-5 includes: The feature extraction module is used to extract key features from multimodal information using a feature extractor; the key features include visual information and auditory information; the feature extractor includes a visual feature extractor and an auditory feature extractor. The convolutional temporal alignment module is used to adjust the temporal scale of key features corresponding to different single-modal information and determine the temporally aligned single-modal key features. The simulated feedforward module is used to simulate forward propagation by matrix multiplication of each time-aligned single-modal key feature with the multimodal classifier parameters to obtain single-modal output and multimodal output, and to evaluate the contribution ratio of different single-modal outputs to the target task based on the single-modal output and the multimodal output to determine the modality adjustment factor. An online loss adjustment module is used to determine the cross-entropy loss corresponding to the single-modal output and the multimodal output based on the single-modal output and the multimodal output, and to dynamically adjust the loss function based on the modality adjustment factor and the cross-entropy loss to determine the final loss of the spiking neural network; the final loss is the optimization objective of the spiking neural network.
7. An electronic device, characterized in that, The device includes a memory and a processor, the memory being used to store a computer program, and the processor running the computer program to cause the electronic device to perform the SNN multimodal target recognition method as described in any one of claims 1-5.
8. A computer-readable storage medium, characterized in that, It stores a computer program that, when executed by a processor, implements the SNN multimodal target recognition method as described in any one of claims 1-5.