Multi-task image classification method and device, equipment and storage medium

Abstract

The embodiment of the invention provides a multi-task image classification method and device, equipment and a storage medium, and relates to the technical field of image processing. The method comprises the following steps: performing pulse feature extraction on a sample image sequence by using a task encoder to obtain a task encoding vector; and obtaining model adjustment parameters based on the task coding vector by using a parameter generator, performing parameter adjustment on a basic pulse neural module in the basic network to obtain a modulation basic network, performing feature mapping on the sample image sequence by using the modulation basic network to obtain a sample embedding vector, and further obtaining a prototype-like vector. And determining a classification result of the to-be-processed image sequence according to the vector distance between the to-be-processed image sequence and each class prototype vector. Only one integral image classification model is needed to realize the classification process of image data of different tasks and different modalities. Because the time step length of each sample image sequence is basically consistent, the training time consumption is small, the training overhead is reduced, and the overall classification performance and efficiency are improved.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a multi-task image classification method, apparatus, device and storage medium. Background Art

[0002] Spiking Neural Networks (SNNs) are a type of bionic neural network model that mimics the way biological neurons transmit information. In spiking neural networks (SNNs), information is transmitted through pulses, and pulses are only emitted when the membrane potential of the neuron exceeds a certain threshold. Different information is represented by the time interval or frequency of the pulses.

[0003] In related technologies, when spiking neural networks (SNNs) are applied to multi-modal and multiple meta-learning task scenarios for image classification, a corresponding classification network is usually trained for each modality’s meta-learning task. However, the problem with this approach is that, on the one hand, more models are required to handle different classification tasks, and on the other hand, since the time steps of image data of different modalities under different tasks are not uniform, it leads to a high learning and training cost, resulting in low classification efficiency. Summary of the invention

[0004] The main purpose of the embodiments of the present application is to propose a multi-task image classification method, apparatus, device and storage medium to reduce the training cost of pulse neural networks in multi-modal and multi-task image classification scenarios and improve the overall classification efficiency.

[0005] To achieve the above objectives, a first aspect of an embodiment of the present application proposes a multi-task image classification method, comprising:

[0006] Using at least one pulse neural module cascaded in the task encoder to extract pulse features from the sample image sequence in the support data set to obtain a task encoding vector;

[0007] Using a parameter generator to generate parameters based on the task encoding vector to obtain model adjustment parameters, and using the model adjustment parameters to adjust parameters of a basic pulse neural module in a basic network to obtain a modulated basic network, wherein the structure of the basic network is consistent with that of the task encoder;

[0008] Performing feature mapping on the sample image sequence using the modulation base network to obtain a sample embedding vector, and averaging the sample embedding vectors belonging to the same class in the support data set to obtain a class prototype vector;

[0009] A sequence of images to be processed in the query data set is obtained, and a classification result of the sequence of images to be processed is determined according to a vector distance between the sequence of images to be processed and each of the class prototype vectors.

[0010] In some embodiments, the at least one cascaded pulse neural module includes a first pulse convolution module, a second pulse convolution module, a third pulse convolution module, and a fourth pulse convolution module; the at least one cascaded pulse neural module in the task encoder is used to extract pulse features from a sample image sequence in a support data set to obtain a task encoding vector, including:

[0011] Using the first pulse convolution module to extract pulse features from the sample image sequence to obtain first image features;

[0012] Using the second pulse convolution module to perform pulse feature extraction on the first image feature to obtain a second image feature;

[0013] Using the third pulse convolution module to perform pulse feature extraction on the second image feature to obtain a third image feature;

[0014] The fourth pulse convolution module is used to extract pulse features from the third image features to obtain the task coding vector.

[0015] In some embodiments, the first pulse convolution module, the second pulse convolution module, the third pulse convolution module and the fourth pulse convolution module each include cascaded convolution layers, a first batch of normalization layers and a first pulse neuron, and the output channels of the convolution layers in the first pulse convolution module, the second pulse convolution module, the third pulse convolution module and the fourth pulse convolution module gradually increase.

[0016] In some embodiments, the parameter generator includes a first linear module and a second linear module, and the step of using the parameter generator to generate parameters based on the task encoding vector to obtain the model adjustment parameters includes:

[0017] Using the first linear module to perform vector dimension reduction on the task encoding vector to obtain a task embedding vector;

[0018] The second linear module is used to perform parameter prediction on the task embedding vector to obtain the model adjustment parameters.

[0019] In some embodiments, the second linear module includes: a first fully connected layer, a second fully connected layer, and a third fully connected layer, the model adjustment parameters include a synaptic weight modulation matrix and a synaptic bias modulation matrix, and the step of performing parameter prediction on the task embedding vector using the second linear module to obtain the model adjustment parameters includes:

[0020] Using the first fully connected layer to perform feature extraction on the task embedding vector to obtain a first parameter vector, and using the second fully connected layer to perform feature extraction on the task embedding vector to obtain a second parameter vector, and performing outer product calculation on the first parameter vector and the second parameter vector to obtain the synaptic weight modulation matrix;

[0021] The third fully connected layer is used to perform feature extraction on the task embedding vector to obtain the synaptic bias modulation matrix.

[0022] In some embodiments, the basic network includes cascaded basic pulse neural modules, each of which includes a cascaded basic convolutional layer, a basic normalization layer, and a basic pulse neuron. The basic pulse neural module in the basic network is adjusted using the model adjustment parameters to obtain a modulated basic network, including:

[0023] Using the synaptic weight modulation matrix and the synaptic bias modulation matrix, parameters of the basic convolution layer are adjusted to obtain an adjusted convolution layer;

[0024] Using the synaptic weight modulation matrix to adjust the parameters of the basic spiking neuron to obtain an adjusted spiking neuron;

[0025] The basic convolutional layer is replaced by the adjusted convolutional layer, and / or the basic spiking neuron is replaced by the adjusted spiking neuron to obtain the modulated basic network.

[0026] In some embodiments, the step of adjusting parameters of the basic convolution layer using the synaptic weight modulation matrix and the synaptic bias modulation matrix to obtain an adjusted convolution layer includes:

[0027] Adding the identity matrix to the synaptic weight modulation matrix to obtain a synaptic unit weight matrix;

[0028] Performing element-by-element multiplication of the synaptic unit weight matrix and the initial synaptic weight matrix of the base convolutional layer to obtain an adjusted synaptic weight matrix;

[0029] Adding the synaptic bias modulation matrix and the initial synaptic bias matrix of the base convolutional layer to obtain an adjusted synaptic bias matrix;

[0030] The parameters of the basic convolution layer are adjusted based on the adjusted synaptic weight matrix and the adjusted synaptic bias matrix to obtain the adjusted convolution layer.

[0031] In some embodiments, the step of adjusting the parameters of the basic spiking neuron using the synaptic weight modulation matrix to obtain the adjusted spiking neuron includes:

[0032] Obtaining the initial membrane potential voltage matrix of the basic pulse neuron;

[0033] Performing element-by-element multiplication of the synaptic weight modulation matrix and the initial membrane potential voltage matrix to obtain an adjusted membrane potential voltage matrix;

[0034] The basic pulse neuron is parameter-adjusted according to the adjusted membrane potential voltage matrix to obtain the adjusted pulse neuron.

[0035] In some embodiments, the first linear module includes a linear layer, a second batch of normalization layers, and a second spiking neuron; and inputting the task encoding vector into the first linear module for vector dimension reduction to obtain the task embedding vector includes:

[0036] Using the linear layer to perform linear feature extraction on the task encoding vector to obtain a first embedding vector;

[0037] Using the second batch normalization layer to batch normalize the first embedding vector input to obtain a second embedding vector;

[0038] The second pulse neuron is used to extract pulse features from the second embedding vector to obtain the task embedding vector.

[0039] To achieve the above-mentioned purpose, a second aspect of an embodiment of the present application proposes a multi-task image classification device, comprising:

[0040] Pulse feature extraction module: used to extract pulse features from the sample image sequence in the support data set using at least one pulse neural module cascaded in the task encoder to obtain a task encoding vector;

[0041] A basic parameter adjustment module: used to use a parameter generator to generate parameters based on the task encoding vector to obtain model adjustment parameters, and use the model adjustment parameters to adjust the parameters of the basic pulse neural module in the basic network to obtain a modulated basic network, the structure of which is consistent with the task encoder;

[0042] A class prototype vector calculation module is used to perform feature mapping on the sample image sequence using the modulation basic network to obtain a sample embedding vector, and calculate and average the sample embedding vectors belonging to the same class in the support data set to obtain a class prototype vector;

[0043] Classification module: used to obtain the image sequence to be processed in the query data set, and determine the classification result of the image sequence to be processed according to the vector distance between the image sequence to be processed and each of the class prototype vectors.

[0044] To achieve the above-mentioned purpose, the third aspect of an embodiment of the present application proposes an electronic device, which includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the method described in the first aspect is implemented.

[0045] To achieve the above-mentioned purpose, the fourth aspect of an embodiment of the present application proposes a storage medium, which is a storage medium. The storage medium stores a computer program, and when the computer program is executed by a processor, the method described in the first aspect is implemented.

[0046] The multi-task image classification method, device, equipment and storage medium proposed in the embodiment of the present application use at least one pulse neural module cascaded in the task encoder to extract pulse features from the sample image sequence in the support data set to obtain a task encoding vector. Then, a parameter generator is used to generate parameters based on the task encoding vector to obtain model adjustment parameters, and the model adjustment parameters are used to adjust the parameters of the basic pulse neural module in the basic network to obtain a modulated basic network, wherein the structure of the basic network is consistent with the task encoder. Then, the modulated basic network is used to perform feature mapping on the sample image sequence to obtain a sample embedding vector, and the sample embedding vectors belonging to the same class in the support data set are averaged to obtain a class prototype vector. Finally, the image sequence to be processed in the query data set is obtained, and the classification result of the image sequence to be processed is determined according to the vector distance between the image sequence to be processed and each class prototype vector. In the embodiment of the present application, for the sample image sequence in each meta-learning task in the support data set of a small number of samples, different model adjustment parameters are generated for each sample image sequence using the task encoder and the parameter generator. Then, the parameters of the basic network are adjusted based on these model adjustment parameters. Then, the basic network after adjusting the parameters is used to extract the feature vector of the sample image sequence, and the task prototype of meta-learning is obtained accordingly. Finally, the image sequences to be processed in the query dataset are classified based on these task prototypes. This method only requires an overall image classification model to realize the classification process of image data of different tasks and modalities. And because the time step of each sample image sequence is basically the same, the training time is short, the training overhead is reduced, and the overall classification performance and efficiency are improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 A schematic diagram showing the differences between the multi-task image classification model of the embodiment of the present application and the related models in the related art.

[0048] Figure 2 It is a flowchart of the multi-task image classification method provided in an embodiment of the present application.

[0049] Figure 3 It is a flowchart of the multi-task image classification method provided in an embodiment of the present application.

[0050] Figure 4 It is a schematic diagram of the structure of the task encoder provided in an embodiment of the present application.

[0051] Figure 5 This is a flowchart provided by an embodiment of the present application for inputting a task encoding vector into a first linear module for vector dimensionality reduction to obtain a task embedding vector.

[0052] Figure 6 This is a flowchart of using the second linear module to predict the parameters of the task embedding vector and obtain the model adjustment parameters provided in an embodiment of the present application.

[0053] Figure 7 The embodiment of the present application provides a flow chart for adjusting the parameters of a basic pulse neural module in a basic network by using a model adjustment parameter to obtain a modulated basic network.

[0054] Figure 8 The embodiment of the present application provides a flowchart of adjusting the parameters of the basic convolution layer by using the synaptic weight modulation matrix and the synaptic bias modulation matrix to obtain the adjusted convolution layer.

[0055] Fig. 9 It is a schematic diagram of adjusting the parameters of the basic convolutional layer based on adjusting the synaptic weight matrix and adjusting the synaptic bias matrix provided in an embodiment of the present application.

[0056] Fig.10 The present invention provides a flowchart for adjusting the parameters of a basic pulse neuron by using a synaptic weight modulation matrix to obtain an adjusted pulse neuron.

[0057] Fig.11 It is a schematic diagram of adjusting parameters of basic pulse neurons according to adjusting the membrane potential voltage matrix provided in an embodiment of the present application.

[0058] Fig.12 It is a schematic diagram of the firing rates of different multi-task image classification models provided in the embodiments of the present application in 5 static image data sets.

[0059] Fig.13 This is a schematic diagram of the firing rates of different multi-task image classification models provided in the embodiments of the present application in 5 static image data sets and 2 dynamic neural data sets.

[0060] Fig.14 It is a performance diagram of the basic pulse neuron in each basic pulse neural module in the basic network provided in an embodiment of the present application.

[0061] Fig.15 It is a schematic diagram of the task migration performance of the multi-task image classification model provided in an embodiment of the present application.

[0062] Fig.16 It is a schematic diagram comparing the classification accuracy of the multi-task image classification model provided in the embodiment of the present application and the model in the related technology.

[0063] Fig.17 This is a structural block diagram of a multi-task image classification device provided in yet another embodiment of the present application.

[0064] Fig.18 It is a schematic diagram of the hardware structure of the electronic device provided in the embodiment of the present application. DETAILED DESCRIPTION

[0065] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0066] It should be noted that although the functional modules are divided in the device schematic and the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart.

[0067] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0068] First, some nouns involved in this application are analyzed:

[0069] Artificial intelligence (AI) is a new technical science that studies and develops theories, methods, technologies and application systems for simulating, extending and expanding human intelligence. AI is a branch of computer science that attempts to understand the essence of intelligence and produce a new type of intelligent machine that can respond in a similar way to human intelligence. Research in this field includes robotics, speech recognition, image recognition, natural language processing and expert systems. AI can simulate the information process of human consciousness and thinking. AI is also a theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0070] Spiking Neural Networks (SNNs) are a type of bionic neural network model that mimics the way biological neurons transmit information. In spiking neural networks (SNNs), information is transmitted through pulses, and pulses are only emitted when the membrane potential of the neuron exceeds a certain threshold. Different information is represented by the time interval or frequency of the pulses.

[0071] In the related art, when pulse neural networks (SNNs) are applied to multi-modal and multiple meta-learning task scenarios of image classification, a corresponding classification network is usually trained for the meta-learning task of each modality. However, the problem with this approach is that, on the one hand, more models are required to handle different classification tasks respectively, and different tasks need to be trained separately, resulting in a large training overhead, which limits the practical application of pulse neural networks (SNNs) in the field of meta-learning. On the other hand, since the time steps of image data of different modalities under different tasks are not uniform, for example, the time step for static image data may be 12, but the time step for dynamic image data may be 100, resulting in a high learning and training cost, making the overall classification efficiency low.

[0072] Based on this, the embodiments of the present application provide a multi-task image classification method, apparatus, device and storage medium. For the sample image sequence in each meta-learning task in the supporting data set of a small number of samples, different model adjustment parameters are generated for each sample image sequence using a task encoder and a parameter generator. Then, the parameters of the basic network are adjusted based on these model adjustment parameters. Then, the feature vector of the sample image sequence is extracted using the basic network after adjusting the parameters, and the task prototype of meta-learning is obtained based on this. Finally, the image sequence to be processed in the query data set is classified based on these task prototypes. In this way, only an overall image classification model is required to realize the classification process of image data of different tasks and different modalities. And because the time step of each sample image sequence is basically the same, the training time is short, the training overhead is reduced, and the overall classification performance and efficiency are improved.

[0073] The embodiments of the present application provide a multi-task image classification method, apparatus, device and storage medium, which are specifically described through the following embodiments. First, the multi-task image classification method in the embodiments of the present application is described.

[0074] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Among them, artificial intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines so that the machines have the functions of perception, reasoning and decision-making.

[0075] Artificial intelligence technology is a comprehensive discipline that covers a wide range of fields, including both hardware-level and software-level technologies. The basic technologies of artificial intelligence generally include sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics and other technologies. Artificial intelligence software technology mainly includes computer vision technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0076] The multi-task image classification method provided in the embodiment of the present application relates to the field of image processing technology. The multi-task image classification method provided in the embodiment of the present application can be applied to a terminal, can also be applied to a server side, and can also be a computer program running in a terminal or a server side. For example, a computer program can be a native program or software module in an operating system; it can be a local (Native) application (APP, Application), that is, a program that needs to be installed in the operating system to run, such as a client that supports multi-task image classification, or it can be a small program, that is, a program that can be run only by downloading it to a browser environment; it can also be a small program that can be embedded in any APP. In short, the above-mentioned computer program can be an application, module or plug-in in any form. Among them, the terminal communicates with the server via a network. The multi-task image classification method can be executed by a terminal or a server, or by a terminal and a server in collaboration.

[0077] In some embodiments, the terminal may be a smart phone, a tablet computer, a laptop computer, a desktop computer or a smart watch. The server may be an independent server, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms; or a service node in a blockchain system, wherein each service node in the blockchain system forms a peer-to-peer (P2P, PeerTo Peer, P2P) network, and the P2P protocol is an application layer protocol running on the Transmission Control Protocol (TCP) protocol. The terminal and the server may be connected via Bluetooth, Universal Serial Bus (USB) or a network or other communication connection methods, which are not limited in this embodiment.

[0078] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0079] It should be noted that in each specific implementation of the present application, when it comes to the need to perform relevant processing based on data related to user identity or characteristics such as user information, user behavior data, user historical data, and user location information, the user's permission or consent will be obtained first, and the collection, use, and processing of these data will comply with the relevant laws, regulations, and standards of the relevant countries and regions. In addition, when the embodiment of the present application needs to obtain the user's sensitive personal information, the user's separate permission or consent will be obtained through a pop-up window or by jumping to a confirmation page. After clearly obtaining the user's separate permission or consent, the necessary user-related data for the normal operation of the embodiment of the present application will be obtained.

[0080] The following describes a multi-task image classification model provided in an embodiment of the present application, which is used to perform a multi-task image classification method.

[0081] Reference Figure 1 , Figure 1 A schematic diagram showing the differences between the multi-task image classification model of the embodiment of the present application and the related models in the related art. Figure 1 It can be seen that in the implementation scheme of the related technology, a corresponding classification network is trained for each modality of the meta-learning task. Specifically, different spiking neural networks are set for NS different meta-learning tasks, and then each spiking neural network extracts features according to the image sequence of the corresponding task to obtain corresponding features. The technical solution of the embodiment of the present application is in the related technology. Similarly, for NS different meta-learning tasks, only a multi-task image classification model based on spiking neural networks SNNs is required.

[0082] It is understandable that the meta-learning tasks here can be different image classification and recognition tasks, such as recognizing text in an image, recognizing people in an image, recognizing specific animals in an image, etc., which can be set according to actual needs.

[0083] Next, the multi-task image classification method in an embodiment of the present application is described.

[0084] Figure 2 is an optional flowchart of the multi-task image classification method provided in an embodiment of the present application. Figure 2 The method may include but is not limited to steps 110 to 140. It can also be understood that this embodiment Figure 2 The order of step 110 to step 140 is not specifically limited, and the order of steps can be adjusted or some steps can be reduced or added according to actual needs.

[0085] Step 110: Use at least one pulse neural module cascaded in the task encoder to extract pulse features from the sample image sequence in the support data set to obtain a task encoding vector.

[0086] In one embodiment, in a meta-learning task, the overall data set is divided into a support data set and a query data set. The support data set contains a sample image sequence, while the query data set contains an image sequence to be processed. The number of image sequences in the support data set is much less than the number of image sequences in the query data set. At the same time, the sample image sequence in the support data set has more than one modality, where the modality refers to static images or dynamic images. According to the specific classification requirements of the task, each type includes at least one sample image sequence. The goal of the embodiment of the present application is to use a support data set of a small number of samples to perform class prototype learning, so as to obtain class prototypes corresponding to each classification under different tasks. Then, we use these class prototypes to perform a specific classification process on a large number of samples in the query data set.

[0087] Reference Figure 3 , Figure 3 It is a flowchart of the multi-task image classification method provided in an embodiment of the present application. Figure 3 In the paper, the multi-task image classification model includes a parameter modulation network and a base network. The parameter modulation network includes a task encoder and a parameter generator. The task encoder extracts features from the sample image sequence to obtain a task encoding vector. Next, the parameter generator calculates the model adjustment parameters based on the task encoding vector, and then performs synaptic weight modulation or voltage modulation on the base network based on the model adjustment parameters to obtain a modulated base network. The modulated base network is used to process the sample image sequence, and finally a class prototype vector is obtained. Finally, for the image sequence to be processed in the query data set, the class prototype vector is used to obtain the corresponding classification result.

[0088] In one embodiment, referring to Figure 4 , Figure 4 It is a schematic diagram of the structure of the task encoder provided in an embodiment of the present application. Figure 4 The task encoder includes multiple cascaded pulse neural modules, and 4 are used as examples in the figure, such as the cascaded first pulse convolution module, the second pulse convolution module, the third pulse convolution module and the fourth pulse convolution module, but it does not mean that the number is limited. Each pulse neural module includes a cascaded convolution layer, a first batch of normalization layers and a first pulse neuron, wherein the output channels of the convolution layers in the first pulse convolution module, the second pulse convolution module, the third pulse convolution module and the fourth pulse convolution module gradually increase. Assuming that the output dimension of the convolution layer in the fourth pulse convolution module is D, the output channel of the convolution layer in the first pulse convolution module is D / 8, the output channel of the convolution layer in the second pulse convolution module is D4, and the output channel of the convolution layer in the third pulse convolution module is D / 2.

[0089] In one embodiment, in combination Figure 4In the embodiment of the present application, the process of using the convolution module in the task encoder to extract features from the sample image sequence in the support data set to obtain the task coding vector includes: using the first pulse convolution module to extract pulse features from the sample image sequence to obtain the first image feature; using the second pulse convolution module to extract pulse features from the first image feature to obtain the second image feature; using the third pulse convolution module to extract pulse features from the second image feature to obtain the third image feature; using the fourth pulse convolution module to extract pulse features from the third image feature to obtain the task coding vector.

[0090] It can be seen that the embodiment of the present application utilizes four cascaded pulse convolution modules to extract features in sequence. As the number of channels increases, each convolution kernel can capture more local features and learn richer feature representations, thereby better retaining the spatial information of the sample image sequence.

[0091] In one embodiment, each pulse convolution module uses a convolution layer to obtain preliminary image features based on input data and then performs pulse feature extraction. Figure 4 , the process of using the first pulse neuron to extract pulse features from the preliminary image features to obtain the task coding vector specifically includes: using the first batch of normalization layers to batch normalize the preliminary image features to obtain normalized image features, and then using the first pulse neuron to extract pulse features from the normalized image features to obtain an output vector. The input of the first pulse convolution module is the sample image sequence, and the output of the last pulse convolution module is the task coding vector.

[0092] Among them, the number of channels of the first batch of normalized layers is twice the dimension of the received input data, and the preliminary image features output by the convolutional layer are converted into two-dimensional normalized image features, with the purpose of having the same scale and distribution, so as to improve the stability of the training process. The first normalized image feature is represented as a two-dimensional matrix, and each element represents the eigenvalue of the corresponding position. Next, the normalized image features are processed and encoded using the first pulse neuron, for example, pulse encoding is performed according to preset rules such as thresholds and emission frequencies, thereby obtaining a discrete pulse sequence. Through the pulse encoding process of the first pulse neuron, the two-dimensional normalized image features can be converted into a D-dimensional task coding vector. In the embodiment of the present application, the use of pulse neurons enables the multi-task image classification model to better simulate the working mode of the biological brain, improve the efficiency and accuracy of feature extraction, and thus improve the overall classification accuracy.

[0093] Step 120: Utilize a parameter generator to generate parameters based on the task coding vector to obtain model adjustment parameters, and utilize the model adjustment parameters to adjust parameters of a basic pulse neural module in the basic network to obtain a modulated basic network.

[0094] In one embodiment, in combination Figure 3 After obtaining the task encoding vector, a parameter generator is used to generate parameters based on the task encoding vector to obtain model adjustment parameters. The parameter generator includes: a first linear module and a second linear module. Therefore, the process of using the parameter generator to generate parameters based on the task encoding vector to obtain model adjustment parameters specifically includes: using the first linear module to perform vector dimension reduction on the task encoding vector to obtain a task embedding vector, and using the second linear module to perform parameter prediction on the task embedding vector to obtain model adjustment parameters.

[0095] In one embodiment, the first linear module includes: a linear layer, a second batch of normalization layers, and a second pulse neuron. Figure 5 , Figure 5 The flowchart of inputting the task encoding vector into the first linear module to perform vector dimension reduction to obtain the task embedding vector provided by the embodiment of the present application specifically includes the following steps:

[0096] Step 510: Use a linear layer to perform linear feature extraction on the task encoding vector to obtain a first embedding vector.

[0097] Step 520: Use a second batch normalization layer to batch normalize the first embedding vector input to obtain a second embedding vector.

[0098] Step 530: Use the second spike neuron to extract spike features from the second embedding vector to obtain a task embedding vector.

[0099] In one embodiment, the first linear module uses a linear layer to perform a linear transformation on the input task encoding vector, extracts linear features, expresses different components of the task encoding vector in the form of a linear combination, and obtains a first embedding vector. This process reduces the dimension of the task encoding vector and maps it to a lower dimensional space, which helps to reduce the computational complexity. Next, based on the first embedding vector, the second batch normalization layer performs batch normalization processing on it to standardize the features so that they have the same distribution and scale, thereby avoiding the problem of gradient disappearance or explosion during the training process and accelerating the training process. Finally, the second embedding vector processed by the second batch normalization layer is sent to the second pulse neuron for further feature extraction. Similar to the first pulse neuron, the second pulse neuron outputs a discrete pulse sequence as the task embedding vector.

[0100] In one embodiment, the second linear module then performs a parameter prediction process of the basic network according to the task embedding vector. The second linear module includes: a first fully connected layer, a second fully connected layer and a third fully connected layer, and the model adjustment parameters include: a synaptic weight modulation matrix and a synaptic bias modulation matrix. Figure 6 , Figure 6This is a flowchart of using the second linear module to predict the parameters of the task embedding vector to obtain the model adjustment parameters provided by the embodiment of the present application, which specifically includes the following steps:

[0101] Step 610: Use the first fully connected layer to extract features from the task embedding vector to obtain a first parameter vector, and use the second fully connected layer to extract features from the task embedding vector to obtain a second parameter vector, and perform outer product calculation on the first parameter vector and the second parameter vector to obtain a synaptic weight modulation matrix.

[0102] In one embodiment, the second linear module uses the first fully connected layer and the second fully connected layer to extract features from the task embedding vector, respectively obtaining a first parameter vector and a second parameter vector, where the first parameter vector and the second parameter vector can be regarded as representations of different aspects of the task embedding vector learned through the network. Next, the outer product of the first parameter vector and the second parameter vector is calculated to obtain a synaptic weight modulation matrix. The synaptic weight modulation matrix obtained in this way can be regarded as a weight matrix obtained by dynamically adjusting the input task embedding vector, which can adjust the strength of the connection between neurons under different inputs, thereby achieving adaptation to sample image sequences corresponding to different tasks.

[0103] In one embodiment, the first parameter vector is expressed as: The second parameter vector is expressed as: Among them, v T represents the task embedding vector, φ1 represents the parameters of the first fully connected layer, φ2 represents the parameters of the second fully connected layer, and l represents the number of parameters that need to be adjusted in the basic network set according to actual needs.

[0104] Therefore, the synaptic weight modulation matrix is ​​expressed as:

[0105]

[0106] Among them, M l (v T ,φ) represents the synaptic weight modulation matrix, represents an outer product operation, wherein each element in the synaptic weight modulation matrix obtained by the outer product calculation is the product of the corresponding elements in the first parameter vector and the second parameter vector.

[0107] Step 620: Use the third fully connected layer to perform feature extraction on the task embedding vector to obtain a synaptic bias modulation matrix.

[0108] In one embodiment, the second linear module uses the third fully connected layer to perform feature extraction on the task embedding vector to obtain a synaptic bias modulation matrix. The synaptic bias modulation matrix is ​​used to modulate the threshold of the activation function corresponding to the pulse neuron in the basic network to adjust the firing rate of the neuron. Through this adjustment process, the basic network can dynamically adjust the activation mode of the neuron according to the sample image sequence corresponding to different input tasks to adapt to different task requirements.

[0109] In one embodiment, the synaptic bias modulation matrix is ​​represented as:

[0110]

[0111] Among them, φ3 represents the parameters of the third fully connected layer.

[0112] The synaptic weight modulation matrix and synaptic bias modulation matrix finally generated in the above process are consistent with the weights and biases of the relevant convolutional layers in the technical pulse neural module in the basic network. Since the embodiment of the present application hopes to modulate all relevant parameters in the basic network, a simple multiple fully connected layers are used for parameter generation. One of the benefits of this design is that the network is simple, and another benefit is that the number of parameters can be easily controlled. It is understandable that the above fully connected layer can also be replaced by a multilayer perceptron (MLP).

[0113] Next, we describe the process of adjusting the parameters of the basic pulse neural module in the basic network using the above two models to obtain the modulated basic network. Through this adjustment process, the basic network can achieve adaptive learning capabilities for different tasks, so that it can better handle changing task requirements.

[0114] In one embodiment, the structure of the basic network is consistent with that of the task encoder, and the basic network includes cascaded basic pulse neural modules, wherein the basic pulse neural modules all include cascaded basic convolutional layers, basic normalization layers, and basic pulse neurons, which are the same as the corresponding network structures in the task encoder. The process of adjusting the parameters of the basic network is specifically divided into synaptic weight modulation (SML) and voltage modulation (VML) according to different adjustment targets.

[0115] The two adjustment processes are described below.

[0116] Reference Figure 7 , Figure 7 The present invention provides a flowchart of using a model adjustment parameter to adjust the parameters of a basic pulse neural module in a basic network to obtain a modulated basic network, which specifically includes the following steps:

[0117] Step 710: Use the synaptic weight modulation matrix and the synaptic bias modulation matrix to adjust the parameters of the basic convolution layer to obtain an adjusted convolution layer.

[0118] In one embodiment, the above steps refer to synaptic weight modulation, the purpose of which is to use the synaptic weight modulation matrix and the synaptic bias modulation matrix to adjust the parameters of the basic convolution layer to obtain an adjusted convolution layer after parameter adaptation. Figure 8 , Figure 8 The present invention provides a flowchart of adjusting the parameters of a basic convolutional layer by using a synaptic weight modulation matrix and a synaptic bias modulation matrix to obtain an adjusted convolutional layer, which specifically includes the following steps:

[0119] Step 810: Add the identity matrix to the synaptic weight modulation matrix to obtain the synaptic unit weight matrix.

[0120] Step 820: perform element-by-element multiplication of the synaptic unit weight matrix and the initial synaptic weight matrix of the base convolutional layer to obtain an adjusted synaptic weight matrix.

[0121] In one embodiment, the synaptic unit weight matrix is ​​expressed as:

[0122] J+M l (v T ,φ)

[0123] In one embodiment, adjusting the synaptic weight matrix is ​​expressed as:

[0124]

[0125] in, Represents the adjustment of the synaptic weight matrix, W l represents the initial synaptic weight matrix, and ⊙ represents element-by-element multiplication, which is used to reduce the number of parameters.

[0126] Step 830: Add the synaptic bias modulation matrix and the initial synaptic bias matrix of the base convolutional layer to obtain an adjusted synaptic bias matrix.

[0127] In one embodiment, adjusting the synaptic bias matrix is ​​expressed as:

[0128]

[0129] in, represents the adjustment of the synaptic bias matrix, b l represents the initial synaptic bias matrix.

[0130] Step 840: Adjust parameters of the basic convolutional layer based on the adjusted synaptic weight matrix and the adjusted synaptic bias matrix to obtain an adjusted convolutional layer.

[0131] In one embodiment, taking one of the basic pulse neural modules in the basic network as an example, referring to Fig. 9 , Fig. 9 It is a schematic diagram of adjusting the parameters of the basic convolutional layer based on adjusting the synaptic weight matrix and adjusting the synaptic bias matrix provided in an embodiment of the present application. Fig. 9 In , the parameters of the basic convolution layer of the basic pulse neural module are adjusted by adjusting the synaptic weight matrix and the synaptic bias matrix to obtain the adjusted convolution layer. Next, the input data is input into the cascaded adjusted convolution layer, the basic batch normalization layer and the basic pulse neural module to obtain the output data.

[0132] Step 720: Use the synaptic weight modulation matrix to adjust the parameters of the basic spiking neuron to obtain an adjusted spiking neuron.

[0133] In one embodiment, referring to Fig.10 , Fig.10 The present invention provides a flowchart of adjusting the parameters of a basic pulse neuron by using a synaptic weight modulation matrix to obtain an adjusted pulse neuron, which specifically includes the following steps:

[0134] Step 1010: Obtain the initial membrane potential voltage matrix of the basic spiking neuron.

[0135] Step 1020: perform element-by-element multiplication of the synaptic weight modulation matrix and the initial membrane potential voltage matrix to obtain an adjusted membrane potential voltage matrix.

[0136] In one embodiment, when using the synaptic weight modulation process to train the model, there is a certain pulse attenuation phenomenon, that is, the basic pulse neuron layer of the basic pulse neural module in the basic network gradually decreases with the deepening of the number of layers of the basic pulse neural module. Its firing rate is about 0.02 in the deepest layer. The extremely low firing rate causes the learned pulse sequence features to be extremely sparse, thereby affecting its meta-learning performance in multi-modal multi-task. Therefore, the embodiment of the present application proposes a voltage modulation process to directly modulate the membrane potential of the basic pulse neurons in the basic network, thereby increasing its firing rate, obtaining richer pulse sequence features, and ultimately improving its multi-modal multi-task meta-learning performance.

[0137] Among them, in voltage modulation, the modulated parameters are regarded as neurotransmitters. Neurotransmitters are released from the presynaptic end of one neuron and bind to the receptors of the postsynaptic membrane of another neuron. Depending on the type of receptors they bind to, neurotransmitters can have an excitatory effect (such as glutamate) or an inhibitory effect (such as glycine) on the postsynaptic neuron. Therefore, the membrane potential voltage matrix is ​​adjusted to adjust the receptor type, thereby changing the relevant membrane potential of the basic pulse neuron.

[0138] In one embodiment, the membrane potential voltage matrix is ​​expressed as:

[0139]

[0140] in, Represents the voltage matrix for adjusting the membrane potential, V l represents the initial membrane potential voltage matrix.

[0141] It is understandable that the voltage modulation process can directly use the synaptic weight modulation matrix generated by the synaptic weight modulation process, or set another two fully connected layers to regenerate a matrix similar to the synaptic weight modulation matrix to participate in the calculation, which is not limited in this embodiment.

[0142] In one embodiment, after using voltage modulation, the firing rate of the corresponding basic pulse neurons in the basic network is improved, and the firing rate of the deepest layer is increased from about 0.02 to about 0.08. At the same time, the accuracy of multimodal multi-task meta-learning is also improved.

[0143] Step 1030: Adjust the parameters of the basic pulse neuron according to the adjusted membrane potential voltage matrix to obtain an adjusted pulse neuron.

[0144] In one embodiment, taking one of the basic pulse neural modules in the basic network as an example, referring to Fig.11 , Fig.11 It is a schematic diagram of adjusting parameters of basic pulse neurons according to adjusting the membrane potential voltage matrix provided in an embodiment of the present application. Fig.11 In the example, the membrane potential of the basic spiking neuron is adjusted by adjusting the synaptic weight matrix to obtain the adjusted spiking neuron. Next, the input data is input into the cascaded basic convolutional layer, basic batch normalization layer and modulated spiking neuron to obtain the output data.

[0145] Step 730: Replace the basic convolutional layer with the adjusted convolutional layer, and / or replace the basic spiking neuron with the adjusted spiking neuron to obtain a modulated basic network.

[0146] In one embodiment, an adjustment convolution layer is obtained by synaptic weight modulation, and an adjustment pulse neuron is obtained by voltage modulation. These two adjustment processes can be selected either or both. Therefore, for each basic pulse neural module, the basic convolution layer can be replaced by the adjustment convolution layer, and / or the basic pulse neuron can be replaced by the adjustment pulse neuron to obtain a modulated pulse neural module, thereby obtaining a modulated basic network corresponding to the sample image sequence.

[0147] Step 130: Use the modulation basis network to perform feature mapping on the sample image sequence to obtain a sample embedding vector, and average the sample embedding vectors belonging to the same class in the support data set to obtain a class prototype vector.

[0148] In one embodiment, the network parameters of the modulation base network are expressed as The modulation base network can be expressed as an embedding function Therefore, the modulation base network is used to perform feature mapping on the sample image sequence, and the sample embedding vector is expressed as:

[0149]

[0150] in, represents a sample image sequence, Represents the sample embedding vector.

[0151] In one embodiment, the sample embedding vector is solved for each sample image sequence in the support data set according to the above process to obtain the sample embedding vector corresponding to each sample image sequence. Then, according to different classes corresponding to different pre-set tasks, the sample embedding vectors belonging to the same class in the support data set are used as the vector set corresponding to the class, that is, the vector set corresponding to each class is obtained.

[0152] Next, the average vector of all sample embedding vectors in each vector set is calculated as the class prototype vector corresponding to the class, expressed as:

[0153]

[0154] Where N is the total number of classes, c n Represents the class prototype vector corresponding to class n, represents the set of sample image sequences corresponding to class n, Represents the number of sample image sequences in the vector set.

[0155] Step 140: Obtain a sequence of images to be processed in the query data set, and determine a classification result of the sequence of images to be processed according to a vector distance between the sequence of images to be processed and each class prototype vector.

[0156] In one embodiment, after obtaining the class prototype vectors of each class in different tasks based on the sample image sequence in the supporting data set of a small number of samples, for the image sequence to be processed in the query data set, it is only necessary to calculate the vector distance between each image sequence to be processed and each class prototype vector, and select the class corresponding to the minimum value of the vector distance as the classification result of the image sequence to be processed.

[0157] In one embodiment, during the training process, the multi-task image classification model uses vector distance to obtain query samples. The predicted category distribution is then compared with the category label to calculate the loss value, where the loss value is calculated by the cross entropy loss function. This loss value is then optimized by stochastic gradient descent, which involves adjusting all parameters in the base network, task encoder, and parameter generator.

[0158] During the above training process, the category distribution It is expressed as:

[0159]

[0160] d(x,y)=(xy) 2

[0161] in, Represents a query sample The vector distance between the prototype vector of class n and the class n, the denominator is the distance between all possible classes n ′ The exponential function values ​​are normalized to ensure that the sum of all probabilities is 1.

[0162] It can be seen that the embodiment of the present application utilizes the timing processing characteristics of pulse neurons, combined with the multi-tasking mechanism of brain science, to synchronously optimize the basic convolutional layer and basic pulse neurons of the basic network during the calculation process. For static image data and dynamic neural data, unified training is performed, and the time step is unified to 8, so as to better adapt to the multi-modal multi-task scenario. In addition, the problem of low multi-task performance when data is scarce is solved through the meta-learning mechanism, and the accuracy of multi-modal multi-task image classification is improved.

[0163] The following describes the verification process of the multi-task image classification method of the embodiment of the present application.

[0164] In one embodiment, the verification process of five static image datasets (Omniglot, mini-ImageNet, FC100, CUB, Aircraft) and two dynamic neural datasets (Double NMNIST and Double ASL-DVS) is taken as an example. At the same time, in order to verify the effect of synaptic weight modulation and voltage modulation on performance improvement, the multi-task image classification model without synaptic weight modulation and voltage modulation is recorded as SMILE, the multi-task image classification model with only synaptic weight modulation is recorded as SMILE+S, the multi-task image classification model with only voltage modulation is recorded as SMILE+V, and the multi-task image classification model with both synaptic weight modulation and voltage modulation is recorded as SMILE+S+V. Four pulse neural networks in related technologies are introduced for performance comparison, namely, prototype-based meta-learning method (ProtoNet), model-agnostic meta-learning method (MAML), prototype-based multi-task meta-learning method (MProtoNet), and model-agnostic multi-task meta-learning method (MMAML).

[0165] Reference Fig.12 , Fig.12 It is a schematic diagram of the firing rates of different multi-task image classification models provided in the embodiments of the present application in 5 static image data sets. Fig.12 It can be seen that the firing rate of the multi-task image classification model SMILE+S+V, which performs both synaptic weight modulation and voltage modulation, is the highest relative trend.

[0166] Reference Fig.13 , Fig.13 Schematic diagram of the firing rate of different multi-task image classification models provided in the embodiments of the present application in 5 static image datasets and 2 dynamic neural datasets. The depth of the horizontal axis refers to the order of the basic spiking neural modules in the basic network. Fig.13 It can be seen that the firing rates of the multi-task image classification model SMILE+V that only performs voltage modulation and the multi-task image classification model SMILE+S+V that performs both synaptic weight modulation and voltage modulation are the highest relative trends.

[0167] Reference Fig.14 , Fig.14It is a performance diagram of the basic spiking neuron in each basic spiking neural module in the basic network provided by the embodiment of the present application. Among them, for each basic spiking neuron, the corresponding firing rate is verified under 5 different experimental settings. The experimental settings are: 2mode, 2mode+, 3mode, 5mode, and 7mode, wherein 2mode represents the selection of the two data sets of Omniglot and mini-ImageNet, 2mode+ represents the selection of the two data sets of mini-ImageNet and FC100, 3mode represents the selection of the three data sets of Omniglot, mini-ImageNet, and FC100, 5mode represents the selection of the five data sets of Omniglot, mini-ImageNet, FC100, CUB, and Aircraft, and 7mode represents the selection of the seven data sets of Omniglot, mini-ImageNet, FC100, CUB, Aircraft, DoubleNMNIST, and DoubleASL-DVS. It can be seen that no matter which basic pulse neuron, the firing rates of the multi-task image classification model SMILE+V that only performs voltage modulation and the multi-task image classification model SMILE+S+V that performs both synaptic weight modulation and voltage modulation are the highest relative trends.

[0168] Reference Fig.15 , Fig.15 It is a schematic diagram of the task migration performance of the multi-task image classification model provided in an embodiment of the present application. Fig.15 The figure shows the accuracy of the multi-task image classification model after pre-training on two small datasets (mini-ImageNet and FC100) and fine-tuning on five static image datasets (Omniglot, mini-ImageNet, FC100, CUB, and Aircraft) and two dynamic neural datasets (Double NMNIST and Double ASL-DVS). It can be seen that the overall accuracy is on an upward trend.

[0169] Reference Fig.16 , Fig.16 It is a schematic diagram comparing the classification accuracy of the multi-task image classification model provided in the embodiment of the present application and the model in the related technology. Fig.16The figure shows the accuracy of the multi-task image classification model and the four models in the related art after pre-training on two small datasets (mini-ImageNet and FC100) and fine-tuning on five static image datasets (Omniglot, mini-ImageNet, FC100, CUB and Aircraft) and two dynamic neural datasets (DoubleNMNIST and Double ASL-DVS). It can be seen that the overall accuracy of the multi-task image classification model of the embodiment of the present application is on the rise and is higher than the models in the related art.

[0170] Refer to Table 1 below, which is a comparison of the classification accuracy of the multi-task image classification model provided in the embodiment of the present application and the model in the related technology.

[0171]

[0172]

[0173] Table 1 Comparison of classification accuracy

[0174] It can be seen that the overall accuracy of the multi-task image classification model in the embodiment of the present application is higher than that of the model in the related art.

[0175] Refer to Table 2 below, which is a comparison of the accuracy of single-task training of the multi-task image classification model provided in an embodiment of the present application on two dynamic neural data sets.

[0176]

[0177] Table 2 Comparison of single-task training accuracy on two dynamic neural datasets

[0178] It can be seen that the overall accuracy of the multi-task image classification model in the embodiment of the present application that has undergone at least one of the modulation processes of synaptic weight modulation and voltage modulation is higher than that of the model without modulation.

[0179] Refer to Table 3 below, which shows the ablation experiment results of the multi-task image classification model provided in the embodiment of the present application in terms of time step.

[0180]

[0181]

[0182] Table 3 Ablation experiment results in terms of time step

[0183] It can be seen that as the time step increases, the accuracy also improves.

[0184] Refer to Table 4, which is a comparison of the accuracy of single-task training of the multi-task image classification model provided in an embodiment of the present application on five static image data sets.

[0185]

[0186] Table 4 Comparison of single-task training accuracy on five static image datasets

[0187] It can be seen that the multi-task image classification model SMILE+V that only performs voltage modulation and the multi-task image classification model SMILE+S+V that performs both synaptic weight modulation and voltage modulation have the highest accuracy.

[0188] Refer to Table 5, which is a comparison of the accuracy of single-task training of the multi-task image classification model provided in an embodiment of the present application on seven data sets.

[0189]

[0190]

[0191] Table 5 Comparison of single-task training accuracy on seven datasets

[0192] It can be seen that the multi-task image classification model SMILE+V that only performs voltage modulation and the multi-task image classification model SMILE+S+V that performs both synaptic weight modulation and voltage modulation have the highest accuracy.

[0193] From the above figures and tables, it can be seen that the multi-task image classification model of the embodiments of the present application can achieve a high accuracy in the meta-learning scenario.

[0194] The technical solution provided by the embodiment of the present application uses at least one pulse neural module cascaded in the task encoder to extract pulse features from the sample image sequence in the support data set to obtain a task encoding vector. Then, a parameter generator is used to generate parameters based on the task encoding vector to obtain model adjustment parameters, and the model adjustment parameters are used to adjust the parameters of the basic pulse neural module in the basic network to obtain a modulated basic network, wherein the structure of the basic network is consistent with the task encoder. Then, the modulated basic network is used to perform feature mapping on the sample image sequence to obtain a sample embedding vector, and the sample embedding vectors belonging to the same class in the support data set are averaged to obtain a class prototype vector. Finally, the image sequence to be processed in the query data set is obtained, and the classification result of the image sequence to be processed is determined according to the vector distance between the image sequence to be processed and each class prototype vector. In the embodiment of the present application, for the sample image sequence in each meta-learning task in the support data set of a small number of samples, different model adjustment parameters are generated for each sample image sequence using the task encoder and the parameter generator. Then, the parameters of the basic network are adjusted based on these model adjustment parameters. Then, the basic network after adjusting the parameters is used to extract the feature vector of the sample image sequence, and the meta-learning task prototype is obtained accordingly. Finally, the image sequence to be processed in the query data set is classified based on these task prototypes. This method only requires an overall image classification model to realize the classification process of image data of different tasks and modalities. And because the time step of each sample image sequence is basically the same, the training time is short, the training cost is reduced, and the overall classification performance and efficiency are improved.

[0195] The present application also provides a multi-task image classification device, which can implement the multi-task image classification method. Fig.17 , the device comprises:

[0196] Pulse feature extraction module 1710: used to extract pulse features from a sample image sequence in a support data set using at least one pulse neural module cascaded in a task encoder to obtain a task encoding vector.

[0197] Basic parameter adjustment module 1720: used to use the parameter generator to generate parameters based on the task encoding vector to obtain model adjustment parameters, and use the model adjustment parameters to adjust the parameters of the basic pulse neural module in the basic network to obtain a modulated basic network. The structure of the basic network is consistent with the task encoder.

[0198] Class prototype vector calculation module 1730: used to use the modulation basis network to perform feature mapping on the sample image sequence to obtain the sample embedding vector, and calculate the average of the sample embedding vectors belonging to the same class in the support data set to obtain the class prototype vector.

[0199] Classification module 1740: used to obtain the image sequence to be processed in the query data set, and determine the classification result of the image sequence to be processed according to the vector distance between the image sequence to be processed and each class prototype vector.

[0200] The specific implementation of the multi-task image classification device of this embodiment is basically the same as the specific implementation of the multi-task image classification method described above, and will not be repeated here.

[0201] The present application also provides an electronic device, including:

[0202] at least one memory;

[0203] at least one processor;

[0204] at least one program;

[0205] The program is stored in the memory, and the processor executes the at least one program to implement the multi-task image classification method implemented in the present application. The electronic device can be any intelligent terminal including a mobile phone, a tablet computer, a personal digital assistant (PDA), a car computer, etc.

[0206] See also Fig.18 , Fig.18 The hardware structure of an electronic device of another embodiment is illustrated, and the electronic device includes:

[0207] The processor 1801 may be implemented by a general-purpose central processing unit (CPU), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application;

[0208] The memory 1802 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 1802 can store an operating system and other application programs. When the technical solution provided in the embodiment of this specification is implemented by software or firmware, the relevant program code is stored in the memory 1802, and the processor 1801 calls and executes the multi-task image classification method of the embodiment of this application;

[0209] Input / output interface 1803, used to implement information input and output;

[0210] Communication interface 1804, used to realize communication interaction between the device and other devices, which can be realized by wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.); and

[0211] A bus 1805 that transmits information between the various components of the device (e.g., the processor 1801, the memory 1802, the input / output interface 1803, and the communication interface 1804);

[0212] The processor 1801 , the memory 1802 , the input / output interface 1803 and the communication interface 1804 are connected to each other in communication within the device via the bus 1805 .

[0213] An embodiment of the present application further provides a storage medium, which is a storage medium storing a computer program, and when the computer program is executed by a processor, the multi-task image classification method is implemented.

[0214] As a non-transitory storage medium, the memory can be used to store non-transitory software programs and non-transitory computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transitory memory, such as at least one disk storage device, a flash memory device, or other non-transitory solid-state storage device. In some embodiments, the memory may optionally include a memory remotely arranged relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0215] The multi-task image classification method, device, equipment and storage medium proposed in the embodiment of the present application, by using at least one pulse neural module cascaded in the task encoder to extract pulse features from the sample image sequence in the support data set, to obtain a task encoding vector. Then, a parameter generator is used to generate parameters based on the task encoding vector to obtain model adjustment parameters, and the model adjustment parameters are used to adjust the parameters of the basic pulse neural module in the basic network to obtain a modulated basic network, wherein the structure of the basic network is consistent with the task encoder. Then, the modulated basic network is used to perform feature mapping on the sample image sequence to obtain a sample embedding vector, and the sample embedding vectors belonging to the same class in the support data set are averaged to obtain a class prototype vector. Finally, the image sequence to be processed in the query data set is obtained, and the classification result of the image sequence to be processed is determined according to the vector distance between the image sequence to be processed and each class prototype vector. In the embodiment of the present application, for the sample image sequence in each meta-learning task in the support data set of a small number of samples, different model adjustment parameters are generated for each sample image sequence using the task encoder and the parameter generator. Then, the parameters of the basic network are adjusted based on these model adjustment parameters. Then, the basic network after adjusting the parameters is used to extract the feature vector of the sample image sequence, and the task prototype of meta-learning is obtained accordingly. Finally, the image sequences to be processed in the query dataset are classified based on these task prototypes. This method only requires an overall image classification model to realize the classification process of image data of different tasks and modalities. And because the time step of each sample image sequence is basically the same, the training time is short, the training overhead is reduced, and the overall classification performance and efficiency are improved.

[0216] The embodiments described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0217] Those skilled in the art will appreciate that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0218] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0219] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0220] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0221] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0222] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the above units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0223] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0224] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0225] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including multiple instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, ROM), random access memory (Random Access Memory, RAM), disk or optical disk and other media that can store programs.

[0226] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but the scope of the rights of the present invention is not limited thereto. Any modification, equivalent substitution and improvement made by a person skilled in the art without departing from the scope and essence of the present invention should be within the scope of the rights of the present invention.

Claims

1. A multi-task image classification method, characterized in that: include: Using at least one pulse neural module cascaded in the task encoder to extract pulse features from the sample image sequence in the support data set to obtain a task encoding vector; Using a parameter generator to generate parameters based on the task encoding vector to obtain model adjustment parameters, and using the model adjustment parameters to adjust parameters of a basic pulse neural module in a basic network to obtain a modulated basic network, wherein the structure of the basic network is consistent with that of the task encoder; Performing feature mapping on the sample image sequence using the modulation base network to obtain a sample embedding vector, and averaging the sample embedding vectors belonging to the same class in the support data set to obtain a class prototype vector; A sequence of images to be processed in the query data set is obtained, and a classification result of the sequence of images to be processed is determined according to a vector distance between the sequence of images to be processed and each of the class prototype vectors.

2. The multi-task image classification method according to claim 1, characterized in that: The at least one cascaded pulse neural module includes a first pulse convolution module, a second pulse convolution module, a third pulse convolution module and a fourth pulse convolution module; the at least one cascaded pulse neural module in the task encoder is used to extract pulse features from the sample image sequence in the support data set to obtain a task encoding vector, including: Using the first pulse convolution module to extract pulse features from the sample image sequence to obtain first image features; Using the second pulse convolution module to perform pulse feature extraction on the first image feature to obtain a second image feature; Using the third pulse convolution module to perform pulse feature extraction on the second image feature to obtain a third image feature; The fourth pulse convolution module is used to extract pulse features from the third image features to obtain the task coding vector.

3. The multi-task image classification method according to claim 2, characterized in that: The first pulse convolution module, the second pulse convolution module, the third pulse convolution module and the fourth pulse convolution module all include cascaded convolution layers, a first batch of normalization layers and a first pulse neuron, and the output channels of the convolution layers in the first pulse convolution module, the second pulse convolution module, the third pulse convolution module and the fourth pulse convolution module gradually increase.

4. The multi-task image classification method according to claim 1, characterized in that: The parameter generator includes a first linear module and a second linear module, and the step of using the parameter generator to generate parameters based on the task encoding vector to obtain model adjustment parameters includes: Using the first linear module to perform vector dimension reduction on the task encoding vector to obtain a task embedding vector; The second linear module is used to perform parameter prediction on the task embedding vector to obtain the model adjustment parameters.

5. The multi-task image classification method according to claim 4, characterized in that: The second linear module includes: a first fully connected layer, a second fully connected layer and a third fully connected layer, the model adjustment parameters include a synaptic weight modulation matrix and a synaptic bias modulation matrix, and the second linear module is used to perform parameter prediction on the task embedding vector to obtain the model adjustment parameters, including: Using the first fully connected layer to perform feature extraction on the task embedding vector to obtain a first parameter vector, and using the second fully connected layer to perform feature extraction on the task embedding vector to obtain a second parameter vector, and performing outer product calculation on the first parameter vector and the second parameter vector to obtain the synaptic weight modulation matrix; The third fully connected layer is used to perform feature extraction on the task embedding vector to obtain the synaptic bias modulation matrix.

6. The multi-task image classification method according to claim 5, characterized in that: The basic network includes cascaded basic pulse neural modules, each of which includes a cascaded basic convolutional layer, a basic normalization layer and a basic pulse neuron. The basic pulse neural module in the basic network is adjusted by using the model adjustment parameters to obtain a modulated basic network, including: Using the synaptic weight modulation matrix and the synaptic bias modulation matrix, parameters of the basic convolution layer are adjusted to obtain an adjusted convolution layer; Using the synaptic weight modulation matrix to adjust the parameters of the basic spiking neuron to obtain an adjusted spiking neuron; The basic convolutional layer is replaced by the adjusted convolutional layer, and / or the basic spiking neuron is replaced by the adjusted spiking neuron to obtain the modulated basic network.

7. The multi-task image classification method according to claim 6, characterized in that: The step of adjusting parameters of the basic convolution layer by using the synaptic weight modulation matrix and the synaptic bias modulation matrix to obtain an adjusted convolution layer includes: Adding the identity matrix to the synaptic weight modulation matrix to obtain a synaptic unit weight matrix; Performing element-by-element multiplication of the synaptic unit weight matrix and the initial synaptic weight matrix of the base convolutional layer to obtain an adjusted synaptic weight matrix; Adding the synaptic bias modulation matrix and the initial synaptic bias matrix of the base convolutional layer to obtain an adjusted synaptic bias matrix; The parameters of the basic convolution layer are adjusted based on the adjusted synaptic weight matrix and the adjusted synaptic bias matrix to obtain the adjusted convolution layer.

8. The multi-task image classification method according to claim 6, characterized in that: The step of adjusting the parameters of the basic spiking neuron by using the synaptic weight modulation matrix to obtain an adjusted spiking neuron includes: Obtaining the initial membrane potential voltage matrix of the basic pulse neuron; Performing element-by-element multiplication of the synaptic weight modulation matrix and the initial membrane potential voltage matrix to obtain an adjusted membrane potential voltage matrix; The basic pulse neuron is parameter-adjusted according to the adjusted membrane potential voltage matrix to obtain the adjusted pulse neuron.

9. The multi-task image classification method according to claim 2, characterized in that: The first linear module includes a linear layer, a second batch of normalized layers, and a second spiking neuron; the step of inputting the task encoding vector into the first linear module for vector dimension reduction to obtain a task embedding vector includes: Using the linear layer to perform linear feature extraction on the task encoding vector to obtain a first embedding vector; Using the second batch normalization layer to batch normalize the first embedding vector input to obtain a second embedding vector; The second pulse neuron is used to extract pulse features from the second embedding vector to obtain the task embedding vector.

10. A multi-task image classification device, characterized in that: include: Pulse feature extraction module: used to extract pulse features from the sample image sequence in the support data set using at least one pulse neural module cascaded in the task encoder to obtain a task encoding vector; A basic parameter adjustment module: used to use a parameter generator to generate parameters based on the task encoding vector to obtain model adjustment parameters, and use the model adjustment parameters to adjust the parameters of the basic pulse neural module in the basic network to obtain a modulated basic network, the structure of which is consistent with the task encoder; A class prototype vector calculation module is used to perform feature mapping on the sample image sequence using the modulation basic network to obtain a sample embedding vector, and calculate and average the sample embedding vectors belonging to the same class in the support data set to obtain a class prototype vector; Classification module: used to obtain the image sequence to be processed in the query data set, and determine the classification result of the image sequence to be processed according to the vector distance between the image sequence to be processed and each of the class prototype vectors.

11. An electronic device, characterized in that: The electronic device comprises a memory and a processor, the memory stores a computer program, and the processor implements the multi-task image classification method according to any one of claims 1 to 9 when executing the computer program.

12. A storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the multi-task image classification method according to any one of claims 1 to 9 is implemented.

Images