Multi-modal pulse signal identification method and device embedded into biological network element structure

Abstract

The invention provides a multi-mode pulse signal identification method and device embedded in a biological network element structure, and the method comprises the steps: obtaining a pulse sequence corresponding to a to-be-identified multi-mode signal, and enabling the pulse sequence to comprise a voice part and an image part; inputting the pulse sequence into a multi-modal recognition model to obtain a recognition result corresponding to the to-be-recognized multi-modal signal, inputting the voice part into a voice input channel of the multi-modal recognition model, and inputting the image part into an image input channel of the multi-modal recognition model; carrying out dimension unification and integration on the voice part and the image part through an integration layer of the multi-modal recognition model to obtain an integrated signal; identifying the integrated signal through a biological network element structure of a hidden layer of the multi-modal identification model to obtain an identification result corresponding to the to-be-identified multi-modal signal; outputting a recognition result through an output layer of the multi-modal recognition model; according to the invention, the accuracy and robustness of multi-modal signal identification can be improved.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a multimodal pulse signal recognition method and device embedded in a biological network metastructure. Background Art

[0002] Spiking Neural Networks (SNNs) are considered the third generation of artificial neural networks (ANNs). They have biological significance in both network architecture and learning paradigm. The neurons, synapses, networks, and learning principles in SNNs are far more complex and powerful than those used in ANNs. The timing spikes and synaptic connections in SNNs constitute the powerful spatiotemporal information processing capabilities of SNNs.

[0003] Spiking neural networks with forward connections can realize the functions of identifying and classifying input signals under the premise of low power consumption. However, due to the discrete encoding method of real-valued values ​​by spiking neural networks, the model robustness and recognition accuracy of the model using only feedforward spiking neural networks are low.

[0004] It can be seen that the multimodal recognition method in the related art has technical problems of low accuracy and poor robustness. Summary of the invention

[0005] The present invention provides a multimodal pulse signal recognition method and device embedded in a biological network element structure, which are used to solve the defects of low accuracy and poor robustness of the multimodal recognition method in the prior art, and to achieve more efficient processing of multimodal information.

[0006] The present invention provides a multimodal pulse signal recognition method embedded in a biological network metastructure, comprising the following steps. Obtain a pulse sequence corresponding to a multimodal signal to be recognized, wherein the pulse sequence includes a speech part and an image part; input the pulse sequence into a multimodal recognition model to obtain a recognition result corresponding to the multimodal signal to be recognized; wherein the inputting the pulse sequence into the multimodal recognition model to obtain a recognition result corresponding to the multimodal signal to be recognized comprises: inputting the speech part into the speech input channel of the multimodal recognition model, and inputting the image part into the image input channel of the multimodal recognition model; unifying and integrating the speech part and the image part in dimension through the integration layer of the multimodal recognition model to obtain an integrated signal; recognizing the integrated signal through the biological network metastructure of the hidden layer of the multimodal recognition model to obtain a recognition result corresponding to the multimodal signal to be recognized; and outputting the recognition result through the output layer of the multimodal recognition model.

[0007] According to a multimodal pulse signal recognition method embedded in a biological network metastructure provided by the present invention, the multimodal recognition model is constructed based on a forward pulse neural network, the biological network metastructure includes a Motif unit, and the method also includes: feed-forward connecting the same layer of neurons in the multimodal recognition model with the previous layer of neurons to obtain a forward channel, wherein the forward channel is used to connect any neuron in the same layer of neurons with the neurons in the previous hidden layer; cyclically connecting the same layer of neurons in the multimodal recognition model through the Motif unit to obtain cyclically connected neurons, wherein the cyclically connected neurons are a micro-loop structure, the micro-loop structure is used to connect the same layer of neurons, and the micro-loop structure is connected to the neurons of the output layer of the multimodal recognition model.

[0008] According to a multimodal pulse signal recognition method embedded in a biological network meta-structure provided by the present invention, after the neurons in the same layer in the multimodal recognition model are cyclically connected through the Motif unit to obtain the cyclically connected neurons, the method also includes: determining the input pulse signal of the current neuron based on the sum of the forward channel of the input of the current neuron and the neuron pulse mark of the micro-loop structure; determining the membrane potential value of the current neuron based on the input pulse signal of the current neuron; and comparing the membrane potential value with a preset membrane potential threshold to determine whether the current neuron is releasing potential.

[0009] According to a multimodal pulse signal recognition method embedded in a biological network meta-structure provided by the present invention, after determining the input pulse signal of the current neuron based on the sum of the forward channel of the input of the current neuron and the neuron pulse mark of the micro-loop structure, the method also includes: performing image data set training based on the input pulse signal to obtain an image Motif mask; performing speech data set training based on the input pulse signal to obtain a speech Motif mask; and determining the average value of the image Motif mask and the speech Motif mask as the multimodal Motif mask corresponding to the input pulse signal.

[0010] According to a multimodal pulse signal recognition method embedded with a biological network meta-structure provided by the present invention, the integrated signal is recognized through the biological network meta-structure of the hidden layer of the multimodal recognition model to obtain a recognition result corresponding to the multimodal signal to be recognized, including: based on the integrated signal, determining a speech Motif mask corresponding to the speech part and an image Motif mask corresponding to the image part; taking the average value of the speech Motif mask and the image Motif mask as the multimodal Motif mask of the multimodal signal to be recognized; multiplying the multimodal Motif mask with the output of the recurrently connected neurons to obtain the output signal of the current hidden layer processed by the Motif unit as the recognition result.

[0011] According to a multimodal pulse signal recognition method embedded in a biological network meta-structure provided by the present invention, before the pulse sequence is input into the multimodal recognition model to obtain the recognition result corresponding to the multimodal signal to be recognized, the method also includes: resetting the multimodal recognition model to obtain an initialized multimodal recognition model; obtaining image information and voice information in the multimodal input sample to be recognized; performing layer-by-layer back propagation learning from pulses to membrane potentials and then to synaptic connection weights based on local gradients according to the image information and the voice information, respectively, to obtain an image Motif mask and a voice Motif mask; summing and averaging the image Motif mask and the voice Motif mask to obtain a multimodal Motif mask; obtaining a label of the multimodal input sample to be recognized; based on the multimodal Motif mask, using a random matrix to map the label in parallel to each hidden layer in the initialized multimodal recognition model, and determining the local gradient from the neuron membrane potential to the synaptic connection weight in each hidden layer; and training the initialized multimodal recognition model based on the local gradient from the neuron membrane potential to the synaptic connection weight.

[0012] The present invention also provides a multimodal pulse signal recognition device embedded in a biological network meta-structure, comprising the following modules: an acquisition module, used to acquire a pulse sequence corresponding to a multimodal signal to be recognized, wherein the pulse sequence includes a speech part and an image part; a recognition module, used to input the pulse sequence into a multimodal recognition model to obtain a recognition result corresponding to the multimodal signal to be recognized; wherein the inputting of the pulse sequence into the multimodal recognition model to obtain a recognition result corresponding to the multimodal signal to be recognized comprises: inputting the speech part into the speech input channel of the multimodal recognition model, and inputting the image part into the image input channel of the multimodal recognition model; dimensional unification and integration of the speech part and the image part are performed through the integration layer of the multimodal recognition model to obtain an integrated signal; the integrated signal is recognized through the biological network meta-structure of the hidden layer of the multimodal recognition model to obtain a recognition result corresponding to the multimodal signal to be recognized; and the recognition result is output through the output layer of the multimodal recognition model.

[0013] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, it implements a multimodal pulse signal recognition method embedded in a biological network metastructure as described in any one of the above.

[0014] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements a multimodal pulse signal recognition method embedded in a biological network metastructure as described in any one of the above.

[0015] The present invention also provides a computer program product, comprising a computer program, which, when executed by a processor, implements any of the multimodal pulse signal recognition methods embedded in a biological network metastructure as described above.

[0016] The multimodal pulse signal recognition method and device embedded with a biological network meta-structure provided by the present invention obtain the speech part and the image part of the pulse sequence corresponding to the multimodal signal to be recognized, and input the speech part and the image part into the multimodal recognition model through the speech input channel and the image input channel respectively; unify and integrate the speech part and the image part in dimension through the integration layer of the multimodal recognition model to obtain integrated information; recognize the integrated information through the biological network meta-structure of the hidden layer of the multimodal recognition model to obtain the recognition result corresponding to the multimodal signal to be recognized, and output the recognition result through the output layer of the multimodal recognition model, thereby, the biological network meta-structure is fused and embedded into the neural network structure, which can help the network to be more efficient in processing multimodal information, and can have higher recognition accuracy and robust performance; thereby solving the technical problems of low accuracy and poor robustness of the multimodal recognition method in the related art. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced one by one below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0018] Figure 1 It is a flow chart of the multimodal pulse signal recognition method embedded in the biological network meta-structure provided by the present invention.

[0019] Figure 2 It is a structural schematic diagram of the multimodal recognition model provided by the present invention.

[0020] Figure 3 This is a schematic diagram of generating a Motif unit provided by the present invention.

[0021] Figure 4 The invention provides a multi-modal pulse signal recognition device embedded with a biological network element structure.

[0022] Figure 5 It is a schematic diagram of the physical structure of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0023] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0024] Spiking neural networks with forward connections can realize the functions of identifying and classifying input signals under the premise of low power consumption. However, due to the discrete encoding method of real-valued values ​​by spiking neural networks, the model robustness and recognition accuracy of the model using only feedforward spiking neural networks are low.

[0025] The method and device provided in the embodiment of the present invention are applicable to processing multimodal signals, and are also applicable to processing audio data, image data, etc., and are described below using a multimodal pulse signal as an example.

[0026] refer to Figure 1 , Figure 1 : is a flow chart of a multimodal pulse signal recognition method embedded in a biological network metastructure provided by the present invention, such as Figure 1 As shown, the method includes the following: Step 101: Acquire a pulse sequence corresponding to a multimodal signal to be identified, wherein the pulse sequence includes a speech part and an image part.

[0027] In an embodiment of the present invention, the multimodal signal to be recognized includes voice data and image data, and the multimodal signal is separated into voice data and image data.

[0028] The separated multimodal signals to be identified are input from the speech channel and the image channel respectively, and then the input signals are converted into pulse sequences using pulse coding methods, which can be Poisson coding, frequency coding, radial basis function coding, etc. Taking Poisson coding as an example, the input real number input value will perform a Bernoulli experiment of the sequence length (compared with a random number) to generate a pulse sequence that conforms to the Poisson distribution.

[0029] Step 102: Input the pulse sequence into the multimodal recognition model to obtain a recognition result corresponding to the multimodal signal to be recognized.

[0030] refer to Figure 2 , Figure 2 It is a structural diagram of the multimodal recognition model provided by the present invention, which includes a speech input channel, an image input channel, an input layer, an integration layer, a Motif layer (hidden layer) and an output layer.

[0031] The pulse sequence is input into the multimodal recognition model to obtain the recognition result corresponding to the multimodal signal to be recognized, including: Step 1021, input the speech part into the speech input channel of the multimodal recognition model, and input the image part into the image input channel of the multimodal recognition model.

[0032] Step 1022, the speech part and the image part are dimensionally unified and integrated through the integration layer of the multimodal recognition model to obtain an integrated signal.

[0033] In the embodiment of the present invention, the information integration process uses a direct splicing method, that is, different input signals are spliced ​​after their dimensions are unified, and the integrated signal is used as the processing in the subsequent steps.

[0034] Step 1023, identifying the integrated signal through the biological network element structure of the hidden layer of the multimodal recognition model to obtain a recognition result corresponding to the multimodal signal to be identified.

[0035] In the embodiment of the present invention, the multimodal recognition model is constructed based on a forward pulse neural network, and a biological network metastructure (Motif unit) is added to enhance the learning ability and performance of the network; the Motif unit added to the network acts on the recurrent connection in the form of a Motif mask, and the neurons in the same layer are connected to each other in addition to the feedforward connection with the neurons in the previous or next layer, and these connections contain 13 special Motif structures with different functions. These connections will form a micro-loop structure and connect to the neurons in the output layer.

[0036] The forward channel is used to connect any neuron with the neurons in the previous hidden layer; the microloop structure is used to connect neurons in the same layer.

[0037] Among them, Motif refers to a subgraph with a specific structure that appears in the network. These subgraphs contain specific information and help to deeply understand the graph structure.

[0038] Step 1024, output the recognition result through the output layer of the multimodal recognition model.

[0039] Through the above steps of the embodiment of the present invention, a multimodal information recognition model is constructed based on a forward pulse neural network. The input of any layer of neurons in the multimodal recognition model is the weighted sum of the pulse signals of the neurons in the previous layer or the input port. The use of Motif units to connect neurons in the same layer helps to improve the recognition accuracy and robustness of the network. According to the theory of neural network pruning, a fully connected network is not the optimal network structure. Pruning a fully connected network to make it sparse helps to find the best network structure topology, which can achieve a higher recognition accuracy and has the advantages of noise resistance. The Motif unit structure proposed in the present invention is a sparse processing of the network, and different Motif structural units correspond to different functions. The embodiment of the present invention extracts an effective Motif topological structure from an existing data set, and embeds an effective Motif unit combination into the network.

[0040] According to a multimodal pulse signal recognition method embedded in a biological network metastructure provided by the present invention, the multimodal recognition model is constructed based on a forward pulse neural network, the biological network metastructure includes a Motif unit, and the method further includes: The neurons in the same layer of the multimodal recognition model are connected to the neurons in the previous layer by feedforward connection to obtain a forward channel, wherein the forward channel is used to connect any neuron in the same layer of neurons with the neurons in the previous hidden layer; The neurons in the same layer of the multimodal recognition model are cyclically connected through the Motif unit to obtain cyclically connected neurons, wherein the cyclically connected neurons are a micro-loop structure, and the micro-loop structure is used to connect the neurons in the same layer, and the micro-loop structure is connected to the neurons in the output layer of the multimodal recognition model.

[0041] Among them, the multimodal recognition model is constructed based on a forward pulse neural network, and a Motif unit is added to enhance the learning ability and performance of the network; the Motif unit added to the network acts on the recurrent connection in the form of a Motif mask. In addition to the feedforward connection with the neurons in the previous or next layer, the neurons in the same layer are also connected to each other. These connections contain 13 special Motif structures with different functions. These connections will form a micro-loop structure and connect to the neurons in the output layer; the forward channel is used to connect any of the neurons with the neurons in the previous hidden layer; the micro-loop structure is used to connect neurons in the same layer.

[0042] Specifically, the recognition result of the multimodal input signal is the image and voice information corresponding to the multimodal. For example, if the multimodal signal to be recognized is images 1 to 9, corresponding to voices 1 to 9, then the recognition result is the corresponding number.

[0043] Among them, the Motif mask can provide multiple advantages such as noise resistance, robustness, and high recognition accuracy. The way it is added is: a recurrent connection is added on the basis of the forward pulse neural network connection, and the Motif mask scales the output of the recurrent connection. According to the different scaling results, the Motif mask is divided into two types: soft mask and hard mask. Among them, the soft mask is pre-trained based on the image and speech data sets, so it can reflect the prior knowledge of multimodal input signals, thereby speeding up the training speed; the hard mask adds a threshold judgment on the basis of the soft mask, so that it only contains two elements of 0 or 1. The part of the hard mask that is 0 is equivalent to completely disconnecting the recurrent connection, so it can simplify the network structure and reduce network power consumption.

[0044] In an embodiment of the present invention, a multimodal information recognition model is constructed based on a forward pulse neural network. The input of any layer of neurons in the multimodal recognition model is the weighted sum of the pulse signals of the neurons in the previous layer or the input port. The neurons in the same layer are connected using Motif units. Since different structures of Motif units correspond to different functions, the interpretability of the model is enhanced by using Motif units pre-trained with image data sets and speech data sets. Motif units with different topological structures can help the network to process multimodal information more efficiently, and can have higher recognition accuracy and robust performance. After being placed on the chip, the low energy consumption advantage of SNN can be used to achieve efficient multimodal information recognition and improve the robust performance of the network.

[0045] Based on the above embodiment, the neuron membrane potential is updated based on the following formula: Spike i(t) =Spike i f (t)+Spike i r (t)

[0046] in, For the number The neurons in The membrane potential at the moment, For the number The neurons at the moment The forward membrane potential, For the number The neurons in The cyclic membrane potential at time For the number The membrane capacitance of the neuron, For the number The synaptic conductivity of the neuron, For the number The resting membrane potential of a neuron, is the number in the previous hidden layer The number of neurons connected to a neuron, is the number in the previous hidden layer The neurons are numbered The synaptic connection weights between neurons, The number in the same layer is The neurons are numbered The synaptic connection weights between neurons, for Moment Motif mask of layer neurons, For the number The neuron receives the signal from The input of the neuron, It is a neuron pulse marker. For the number The neuron pulse mark of the neuron in the forward channel, For the number The neuron pulse logo of the neuron in the loop channel.

[0047] According to a multimodal pulse signal recognition method embedded in a biological network metastructure provided by the present invention, after cyclically connecting neurons in the same layer of a multimodal recognition model through Motif units to obtain cyclically connected neurons, the method further includes: Determine the input pulse signal of the current neuron according to the sum of the input forward channel of the current neuron and the neuron pulse mark of the microloop structure; According to the input pulse signal of the current neuron, the membrane potential value of the current neuron is determined; The membrane potential value is compared with the preset membrane potential threshold to determine whether the current neuron is firing.

[0048] In an embodiment of the present invention, the membrane potential of any of the above neurons is updated based on the following steps: the input pulse signal of the neuron is determined based on the sum of the neuron pulse flags of the forward connection and the recurrent connection of the current neuron input. The membrane potential value of the neuron is determined based on the pulse emission threshold of the current neuron and the input value of the current neuron, and whether the neuron emits potential is determined based on the relative size of the membrane potential value and the threshold.

[0049] According to a multimodal pulse signal recognition method embedded in a biological network metastructure provided by the present invention, after determining the input pulse signal of the current neuron according to the sum of the input forward channel of the current neuron and the neuron pulse mark of the microloop structure, the method further includes: The image data set is trained based on the input pulse signal to obtain the image Motif mask; Based on the input pulse signal, the speech data set is trained to obtain the speech motif mask; The average value of the image motif mask and the speech motif mask is determined as the multimodal motif mask corresponding to the input pulse signal.

[0050] In an embodiment of the present invention, the recurrently connected pulse output value of the neuron is scaled according to the Motif mask shown in the following formula:

[0051] in, represents the Motif mask learned from the image dataset, Represents the Motif mask learned from the speech dataset.

[0052] refer to Figure 3 , Figure 3 This is a schematic diagram of generating a Motif unit provided by the present invention.

[0053] like Figure 3As shown, the original 4-node graph 310 can generate three 3-node Motif units 320. According to the above definition, the non-connected three-node subgraph connected by the dashed arrow on the far right is not a Motif unit.

[0054] In an embodiment of the present invention, the Motif mask of the neuron of the multimodal recognition model can be decomposed into Motif units according to the following formula:

[0055] in, Be original - a network of nodes, It is a picture The subgraph sequence of is a non-connected three-node subgraph, where It is called a sub-image unit, or Motif unit.

[0056] In addition, in the spiking neural network, the main form of information transmission is pulses, and in the multimodal recognition model of the present invention, the pulse output and input of any neuron have two types: forward pulses and cyclic pulses using Motif masks. The generation methods of these two pulses are defined according to the following formula:

[0057] in, is the neuron firing threshold, each neuron has the same firing threshold, is the pulse mark of the forward channel, is the pulse mark of the cyclically connected channel with Motif mask, is the moment when the forward pulse flag is 1, is the moment when the cycle pulse flag is 1, is the time window parameter of the forward channel, Temporal window parameters for recurrently connected channels with Motif masks.

[0058] in, is the neuron firing threshold, each neuron has the same firing threshold, is the pulse mark of the forward channel, is the pulse mark of the cyclically connected channel with Motif mask, is the moment when the forward pulse flag is 1, is the moment when the cycle pulse flag is 1, is the time window parameter of the forward channel, Temporal window parameters for recurrently connected channels with Motif masks.

[0059] Through the embodiments of the present invention, Motif masks are used and various different types of Motif structures are integrated, so that the network can operate with higher accuracy and higher robustness and have stronger biological interpretability.

[0060] According to a multimodal pulse signal recognition method embedded in a biological network metastructure provided by the present invention, the integrated signal is recognized through the biological network metastructure of the hidden layer of the multimodal recognition model to obtain a recognition result corresponding to the multimodal signal to be recognized, including: Based on the integrated signal, determining a speech motif mask corresponding to the speech part and an image motif mask corresponding to the image part; The average value of the speech motif mask and the image motif mask is used as the multimodal motif mask of the multimodal signal to be identified; The multimodal Motif mask is multiplied by the output of the recurrently connected neurons to obtain the output signal of the current hidden layer processed by the Motif unit as the recognition result.

[0061] In an embodiment of the present invention, the Motif masks of the image data set and the speech data set are determined, and the average value is calculated to obtain the Motif mask corresponding to the multimodal input. The Motif mask corresponding to the multimodal signal is multiplied by the output of the neuron of the recurrent connection of the corresponding layer to obtain the output signal of the current hidden layer processed by the Motif unit.

[0062] According to a multimodal pulse signal recognition method embedded in a biological network metastructure provided by the present invention, before inputting the pulse sequence into the multimodal recognition model to obtain the recognition result corresponding to the multimodal signal to be recognized, the method further includes: Resetting the multimodal recognition model to obtain an initialized multimodal recognition model; Obtain image information and voice information from a multimodal input sample to be recognized; According to the image information and speech information, back propagation learning is performed layer by layer from pulses to membrane potentials and then to synaptic connection weights based on local gradients to obtain image Motif masks and speech Motif masks; Based on the sum and average of the image motif mask and the speech motif mask, a multimodal motif mask is obtained; Obtain labels for multimodal input samples to be identified; Based on the multimodal Motif mask, a random matrix is ​​used to map the labels in parallel to each hidden layer in the initialized multimodal recognition model, and the local gradient from the neuron membrane potential to the synaptic connection weight in each hidden layer is determined; The initialized multimodal recognition model is trained based on the local gradient from neuronal membrane potential to synaptic connection weights.

[0063] Based on the above embodiments, the multimodal recognition model is trained according to the following steps: Initialize the network by resetting weights and all related parameters; The image information in the multimodal input is extracted separately, and based on the local gradient, it is back-propagated layer by layer from the pulse to the membrane potential and then to the synaptic connection weight to learn the image Motif mask; The speech information in the multimodal input is extracted separately, and based on the local gradient, it is back-propagated layer by layer from the pulse to the membrane potential and then to the synaptic connection weight to learn the speech motif mask; The image Motif mask and the speech Motif mask are summed and averaged to obtain a multimodal Motif mask; Obtain the label of the multimodal input to be identified to calculate the global gradient; The Motif mask is fixed, and a random matrix is ​​used to map the input labels in parallel to each hidden layer in the multimodal recognition model, and determine the local gradient from the neuron membrane potential to the synaptic connection weight in each hidden layer; The multimodal recognition model is trained using the local gradient from neuron membrane potential to synaptic connection weights.

[0064] Specifically, in the process of calculating local gradients, the model uses the layer-by-layer backpropagation algorithm for gradient calculation to perform training. In the process of calculating global gradients, the model uses the label remote projection method to map the label to the hidden layer using a random matrix. The calculation formula is as follows:

[0065] in, As the The global gradient of the neurons in the layer, from the label After The random matrix corresponding to the layer neural dimension After the dimension transformation, The feedforward output gradient of the layer neurons guides the modification of the hidden layer synaptic connection weights; is the weight change of the synaptic connection in the forward channel; is the change in the synaptic connection weight of the recurrent channel with the Motif mask; is the learning rate of the forward channel; is the learning rate of the cyclic connection channel; For the Motif mask of layer neurons.

[0066] For the problem of discontinuity of the gradient from pulse to membrane potential of the spiking neural network, when calculating the pulse of each layer Yes, membrane potential The derivative of is calculated according to the following formula:

[0067] in for Neuronal impulses at a time arrive Membrane potential at time The gradient of is the neuron threshold, It is the preset gradient judgment range.

[0068] Based on the above embodiments, the present invention provides a multimodal pulse signal recognition method embedded in a biological network metastructure, which specifically includes the following steps: Step 1: Extract the image information from the multimodal input separately and perform pulse coding.

[0069] Step 2: Extract the speech information from the multimodal input separately and perform pulse coding.

[0070] Step 3: Integrate the pulses containing speech information and the pulses containing image information in steps 1 and 2 and use them as input into the network containing the Motif unit.

[0071] refer to Figure 2 , Figure 2 Schematic diagram of the multi-modal pulse neural network recognition structure with Motif mask provided by the present invention. Figure 2 As shown, the pulses containing speech information and the pulses containing image information in step 1 and step 2 are integrated and enter the network containing Motif units as input; the network includes a forward connection layer and a recurrent connection layer, and the dotted lines in the figure represent recurrent connections.

[0072] Step 4: Based on the local gradient, back propagate layer by layer from the pulse to the membrane potential and then to the synaptic connection weight to learn the image Motif mask.

[0073] Step 5: Based on the local gradient, back propagate layer by layer from the pulse to the membrane potential and then to the synaptic connection weight to learn the speech motif mask.

[0074] Step 6: Perform sum and average calculation on the image motif mask and the speech motif mask to obtain a multimodal motif mask.

[0075] Step 7, fix the Motif mask, use a random matrix, map the input label as a reward signal in parallel to each hidden layer in the multimodal recognition model, determine the local gradient from the neuron membrane potential to the synaptic connection weight in each hidden layer, and use the local gradient from the neuron membrane potential to the synaptic connection weight to train the multimodal recognition model.

[0076] Step 8: Use the improved spiking neural network based on Motif structure and reward learning to realize the recognition of multimodal signals. The output layer uses the maximum firing rate determination method to select the category corresponding to the neuron with the highest pulse firing rate as the category of model classification.

[0077] The multimodal signal recognition method provided by the embodiment of the present invention combines the advantages of the Motif unit and the spiking neural network and has the following advantages: (1) Robust computing: The Motif structure with specific functions provides a micro-loop structure with noise-resistant function, which can improve the robustness of single-modal and multi-modal classification tasks.

[0078] (2) Low computational cost: Compared with other spiking neural networks without motifs, motif units that are learned specifically according to tasks can provide prior knowledge in multimodal recognition, thereby accelerating convergence speed and recognition accuracy, thus achieving relatively low computational costs.

[0079] (3) Interpretability: The analysis of Motif units increases the interpretability of the network and provides a new design idea for specific functions (such as robustness and low computational cost), which will help further research on spiking neural networks.

[0080] Through the multimodal pulse signal recognition method embedded in the biological network meta-structure provided by the present invention, a multimodal information recognition model is constructed based on a forward pulse neural network. The input of any layer of neurons in the multimodal recognition model is the weighted sum of the pulse signals of the neurons in the previous layer or the input port. The use of Motif units to connect neurons in the same layer helps to improve the recognition accuracy and robustness of the network. According to the theory of neural network pruning, a fully connected network is not the optimal network structure. Pruning makes the fully connected network sparse, which helps to find the best network structure topology. This network topology can achieve a high recognition accuracy and has the advantages of noise resistance.

[0081] The Motif unit structure proposed in the present invention is a sparse processing of the network, and different Motif structural units correspond to different functions. An effective Motif topological structure is extracted from an existing data set, and an effective Motif unit combination is embedded in the network. In the present invention, the existing small-scale visual and auditory data sets are first used to train the neural network to obtain a Motif structure with auditory and visual features. Then the Motif is embedded in the target network. Compared with the fully connected network, the method of Motif fusion embedding neural network structure is more energy-efficient and has a higher recognition accuracy. Compared with random sparsification, the Motif structure has a stronger expression ability for input information. Using Motif units pre-trained with image data sets and speech data sets will help process image and speech data respectively, enhance model capabilities, and increase interpretability. According to the above theory, using the Motif fusion method, the fused Motif can be applied in the network to have a better recognition effect on a richer data set. Motif units with different topological structures can help the network to be more efficient in processing multimodal information, and can have higher recognition accuracy and robust performance.

[0082] The multimodal pulse signal recognition device embedded in a biological network metastructure provided by the present invention is described below. The multimodal pulse signal recognition device embedded in a biological network metastructure described below and the multimodal pulse signal recognition method embedded in a biological network metastructure described above can be referenced to each other.

[0083] refer to Figure 4 , Figure 4 The invention provides a multimodal pulse signal recognition device embedded with a biological network element structure, which includes an acquisition module 401 and a recognition module 402.

[0084] An acquisition module 401 is used to acquire a pulse sequence corresponding to a multimodal signal to be identified, wherein the pulse sequence includes a speech part and an image part; The recognition module 402 is used to input the pulse sequence into the multimodal recognition model to obtain the recognition result corresponding to the multimodal signal to be recognized; wherein, the pulse sequence is input into the multimodal recognition model to obtain the recognition result corresponding to the multimodal signal to be recognized, including: inputting the speech part into the speech input channel of the multimodal recognition model, and inputting the image part into the image input channel of the multimodal recognition model; unifying and integrating the speech part and the image part in dimension through the integration layer of the multimodal recognition model to obtain an integrated signal; recognizing the integrated signal through the biological network metastructure of the hidden layer of the multimodal recognition model to obtain the recognition result corresponding to the multimodal signal to be recognized; and outputting the recognition result through the output layer of the multimodal recognition model.

[0085] Specifically, the multimodal pulse signal recognition device embedded in the biological network element structure provided by the present invention can implement all the method steps implemented in the multimodal pulse signal recognition method embodiment embedded in the biological network element structure, and can achieve the same technical effect. The parts and beneficial effects of this embodiment that are the same as those of the method embodiment will not be described in detail here.

[0086] Figure 5 is a schematic diagram of the physical structure of the electronic device provided by the present invention, such as Figure 5 As shown, the electronic device may include: a processor (processor) 510, a communication interface (Communications Interface) 520, a memory (memory) 530 and a communication bus 540, wherein the processor 510, the communication interface 520, and the memory 530 communicate with each other through the communication bus 540. The processor 510 can call the logic instructions in the memory 530 to execute a multimodal pulse signal recognition method embedded in a biological network meta-structure, the method comprising: obtaining a pulse sequence corresponding to the multimodal signal to be recognized, wherein the pulse sequence includes a speech part and an image part; inputting the pulse sequence into a multimodal recognition model to obtain a recognition result corresponding to the multimodal signal to be recognized; wherein, inputting the pulse sequence into the multimodal recognition model to obtain a recognition result corresponding to the multimodal signal to be recognized, comprising: inputting the speech part into the speech input channel of the multimodal recognition model, and inputting the image part into the image input channel of the multimodal recognition model; dimensionalizing and integrating the speech part and the image part through the integration layer of the multimodal recognition model to obtain an integrated signal; recognizing the integrated signal through the biological network meta-structure of the hidden layer of the multimodal recognition model to obtain a recognition result corresponding to the multimodal signal to be recognized; and outputting the recognition result through the output layer of the multimodal recognition model.

[0087] In addition, the logic instructions in the above-mentioned memory 530 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0088] On the other hand, the present invention also provides a computer program product, which includes a computer program, which can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the multimodal pulse signal recognition method embedded with a biological network meta-structure provided by the above methods, the method including: obtaining a pulse sequence corresponding to the multimodal signal to be recognized, wherein the pulse sequence includes a speech part and an image part; inputting the pulse sequence into a multimodal recognition model to obtain a recognition result corresponding to the multimodal signal to be recognized; wherein, inputting the pulse sequence into the multimodal recognition model to obtain a recognition result corresponding to the multimodal signal to be recognized, including: inputting the speech part into the speech input channel of the multimodal recognition model, and inputting the image part into the image input channel of the multimodal recognition model; dimensionalizing and integrating the speech part and the image part through the integration layer of the multimodal recognition model to obtain an integrated signal; recognizing the integrated signal through the biological network meta-structure of the hidden layer of the multimodal recognition model to obtain a recognition result corresponding to the multimodal signal to be recognized; and outputting the recognition result through the output layer of the multimodal recognition model.

[0089] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the multimodal pulse signal recognition method embedded with a biological network meta-structure provided by the above-mentioned methods, the method comprising: obtaining a pulse sequence corresponding to the multimodal signal to be recognized, wherein the pulse sequence comprises a speech part and an image part; inputting the pulse sequence into a multimodal recognition model to obtain a recognition result corresponding to the multimodal signal to be recognized; wherein, inputting the pulse sequence into the multimodal recognition model to obtain a recognition result corresponding to the multimodal signal to be recognized, comprises: inputting the speech part into the speech input channel of the multimodal recognition model, and inputting the image part into the image input channel of the multimodal recognition model; dimensionalizing and integrating the speech part and the image part through the integration layer of the multimodal recognition model to obtain an integrated signal; recognizing the integrated signal through the biological network meta-structure of the hidden layer of the multimodal recognition model to obtain a recognition result corresponding to the multimodal signal to be recognized; and outputting the recognition result through the output layer of the multimodal recognition model.

[0090] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0091] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0092] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A multimodal pulse signal recognition method embedded in a biological network metastructure, characterized in that: include: Acquire a pulse sequence corresponding to the multimodal signal to be identified, wherein the pulse sequence includes a speech part and an image part; Inputting the pulse sequence into a multimodal recognition model to obtain a recognition result corresponding to the multimodal signal to be recognized; The step of inputting the pulse sequence into a multimodal recognition model to obtain a recognition result corresponding to the multimodal signal to be recognized includes: Inputting the speech part into a speech input channel of a multimodal recognition model, and inputting the image part into an image input channel of the multimodal recognition model; Unifying and integrating the dimensions of the speech part and the image part through the integration layer of the multimodal recognition model to obtain an integrated signal; Identifying the integrated signal through the biological network element structure of the hidden layer of the multimodal recognition model to obtain a recognition result corresponding to the multimodal signal to be identified; The recognition result is outputted through the output layer of the multimodal recognition model.

2. The multimodal pulse signal recognition method embedded in the biological network metastructure according to claim 1 is characterized in that: The multimodal recognition model is constructed based on a forward pulse neural network, the biological network metastructure includes a Motif unit, and the method further includes: Performing feed-forward connection between neurons in the same layer and neurons in the previous layer in the multimodal recognition model to obtain a forward channel, wherein the forward channel is used to connect any neuron in the same layer of neurons with neurons in the previous hidden layer; The neurons in the same layer of the multimodal recognition model are cyclically connected through the Motif unit to obtain cyclically connected neurons, wherein the cyclically connected neurons are micro-loop structures, and the micro-loop structures are used to connect the neurons in the same layer, and the micro-loop structures are connected to the neurons in the output layer of the multimodal recognition model.

3. The multimodal pulse signal recognition method embedded in the biological network metastructure according to claim 2 is characterized in that: After the Motif unit is used to cyclically connect neurons in the same layer of the multimodal recognition model to obtain cyclically connected neurons, the method further includes: Determine the input pulse signal of the current neuron according to the sum of the forward channel of the input of the current neuron and the neuron pulse mark of the micro-loop structure; Determining a membrane potential value of the current neuron according to an input pulse signal of the current neuron; The membrane potential value is compared with a preset membrane potential threshold to determine whether the current neuron is performing potential discharge.

4. The multimodal pulse signal recognition method embedded in the biological network metastructure according to claim 3 is characterized in that: After determining the input pulse signal of the current neuron based on the sum of the forward channel of the input of the current neuron and the neuron pulse mark of the micro-loop structure, the method further includes: Perform image data set training based on the input pulse signal to obtain an image Motif mask; Perform speech data set training based on the input pulse signal to obtain a speech Motif mask; An average value of the image Motif mask and the speech Motif mask is determined as a multimodal Motif mask corresponding to the input pulse signal.

5. The multimodal pulse signal recognition method embedded in the biological network metastructure according to claim 1 is characterized in that: The step of identifying the integrated signal through the biological network element structure of the hidden layer of the multimodal recognition model to obtain a recognition result corresponding to the multimodal signal to be identified includes: Based on the integrated signal, determining a speech motif mask corresponding to the speech part and an image motif mask corresponding to the image part; Taking the average value of the speech Motif mask and the image Motif mask as the multimodal Motif mask of the multimodal signal to be identified; The multimodal Motif mask is multiplied by the output of the recurrently connected neuron to obtain an output signal of the current hidden layer processed by the Motif unit as a recognition result.

6. The multimodal pulse signal recognition method embedded in a biological network metastructure according to claim 1, characterized in that: Before inputting the pulse sequence into the multimodal recognition model to obtain the recognition result corresponding to the multimodal signal to be recognized, the method further includes: Resetting the multimodal recognition model to obtain an initialized multimodal recognition model; Obtain image information and voice information from a multimodal input sample to be recognized; According to the image information and the speech information, back propagation learning is performed layer by layer from pulses to membrane potentials and then to synaptic connection weights based on local gradients to obtain an image Motif mask and a speech Motif mask; Based on the image motif mask and the speech motif mask, summing and averaging, a multimodal motif mask is obtained; Obtaining a label of the multimodal input sample to be identified; Based on the multimodal Motif mask, using a random matrix to map the labels in parallel to each hidden layer in the initialized multimodal recognition model, and determining the local gradient from the neuron membrane potential to the synaptic connection weight in each hidden layer; The initialized multimodal recognition model is trained based on the local gradient from the neuron membrane potential to the synaptic connection weight.

7. A multimodal pulse signal recognition device embedded in a biological network metastructure, characterized in that: include: An acquisition module, used for acquiring a pulse sequence corresponding to the multimodal signal to be identified, wherein the pulse sequence includes a speech part and an image part; A recognition module, used for inputting the pulse sequence into a multimodal recognition model to obtain a recognition result corresponding to the multimodal signal to be recognized; wherein, the step of inputting the pulse sequence into the multimodal recognition model to obtain a recognition result corresponding to the multimodal signal to be recognized comprises: inputting the speech part into a speech input channel of the multimodal recognition model, and inputting the image part into an image input channel of the multimodal recognition model; dimensionalizing and integrating the speech part and the image part through an integration layer of the multimodal recognition model to obtain an integrated signal; recognizing the integrated signal through a biological network metastructure of a hidden layer of the multimodal recognition model to obtain a recognition result corresponding to the multimodal signal to be recognized; and outputting the recognition result through an output layer of the multimodal recognition model.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the multimodal pulse signal recognition method embedded in the biological network metastructure as described in any one of claims 1 to 6 is implemented.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the multimodal pulse signal recognition method embedded in a biological network metastructure as described in any one of claims 1 to 6 is implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the multimodal pulse signal recognition method embedded in a biological network metastructure as described in any one of claims 1 to 6 is implemented.

Images