Multi-task image classification method, device, equipment and storage medium

Abstract

The embodiments of the present application provide a multi-task image classification method, apparatus, device and storage medium, which relate to the field of image processing technology. The method comprises: using a task encoder to extract pulse features from a sample image sequence to obtain a task encoding vector. Using a parameter generator to obtain a model adjustment parameter based on the task encoding vector, adjusting the parameters of the basic pulse neural module in the basic network to obtain a modulation basic network, using the modulation basic network to perform feature mapping on the sample image sequence to obtain a sample embedding vector, and then obtaining a class prototype vector. The classification result of the image sequence to be processed is determined based on the vector distance between the image sequence to be processed and each class prototype vector. Only one overall image classification model is required to realize the classification process of image data of different tasks and different modalities. Since the time step of each sample image sequence is basically the same, the training time is short, the training overhead is reduced, and the overall classification performance and efficiency are improved.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a multi-task image classification method, apparatus, device and storage medium. Background Art

[0002] Spiking Neural Networks (SNNs) are a type of biomimetic neural network model that mimics the information transmission method of biological neurons. In SNNs, information is transmitted through pulses, and pulses are only fired when the neuron's membrane potential exceeds a certain threshold. Different information is represented by the time interval or frequency of the pulses.

[0003] In related technologies, when spiking neural networks (SNNs) are applied to multimodal, multiple meta-learning tasks for image classification, a corresponding classification network is usually trained for each meta-learning task. However, this approach has the following problems: on the one hand, more models are required to handle different classification tasks; on the other hand, because the time steps of image data of different modalities are not uniform across different tasks, this leads to high learning and training costs and low classification efficiency. Summary of the Invention

[0004] The main purpose of the embodiments of the present application is to propose a multi-task image classification method, apparatus, device and storage medium to reduce the training cost of pulse neural networks in multimodal and multi-task image classification scenarios and improve the overall classification efficiency.

[0005] To achieve the above objectives, a first aspect of an embodiment of the present application proposes a multi-task image classification method, comprising:

[0006] Using at least one spiking neural module cascaded in the task encoder to extract spiking features from the sample image sequence in the support dataset to obtain a task encoding vector;

[0007] Using a parameter generator to generate parameters based on the task encoding vector to obtain model adjustment parameters, and using the model adjustment parameters to adjust parameters of a basic spiking neural module in a basic network to obtain a modulated basic network, wherein the structure of the basic network is consistent with that of the task encoder;

[0008] Performing feature mapping on the sample image sequence using the modulation base network to obtain a sample embedding vector, and averaging the sample embedding vectors belonging to the same class in the supporting data set to obtain a class prototype vector;

[0009] An image sequence to be processed in a query data set is obtained, and a classification result of the image sequence to be processed is determined according to a vector distance between the image sequence to be processed and each of the class prototype vectors.

[0010] In some embodiments, the at least one cascaded spiking neural module includes a first spiking convolution module, a second spiking convolution module, a third spiking convolution module, and a fourth spiking convolution module; and the step of extracting spiking features from a sample image sequence in a support dataset using the at least one cascaded spiking neural module in the task encoder to obtain a task encoding vector includes:

[0011] Using the first pulse convolution module to perform pulse feature extraction on the sample image sequence to obtain a first image feature;

[0012] performing pulse feature extraction on the first image feature using the second pulse convolution module to obtain a second image feature;

[0013] performing pulse feature extraction on the second image feature using the third pulse convolution module to obtain a third image feature;

[0014] The fourth pulse convolution module is used to perform pulse feature extraction on the third image feature to obtain the task coding vector.

[0015] In some embodiments, the first pulse convolution module, the second pulse convolution module, the third pulse convolution module and the fourth pulse convolution module each include cascaded convolution layers, a first batch of normalization layers and a first pulse neuron, and the output channels of the convolution layers in the first pulse convolution module, the second pulse convolution module, the third pulse convolution module and the fourth pulse convolution module gradually increase.

[0016] In some embodiments, the parameter generator includes a first linear module and a second linear module, and using the parameter generator to generate parameters based on the task encoding vector to obtain model adjustment parameters includes:

[0017] Using the first linear module to perform vector dimensionality reduction on the task encoding vector to obtain a task embedding vector;

[0018] The second linear module is used to perform parameter prediction on the task embedding vector to obtain the model adjustment parameters.

[0019] In some embodiments, the second linear module includes: a first fully connected layer, a second fully connected layer, and a third fully connected layer; the model adjustment parameters include a synaptic weight modulation matrix and a synaptic bias modulation matrix; and using the second linear module to perform parameter prediction on the task embedding vector to obtain the model adjustment parameters includes:

[0020] Performing feature extraction on the task embedding vector using the first fully connected layer to obtain a first parameter vector, performing feature extraction on the task embedding vector using the second fully connected layer to obtain a second parameter vector, and performing outer product calculation on the first parameter vector and the second parameter vector to obtain the synaptic weight modulation matrix;

[0021] The third fully connected layer is used to perform feature extraction on the task embedding vector to obtain the synaptic bias modulation matrix.

[0022] In some embodiments, the basic network includes cascaded basic pulse neural modules, each of which includes a cascaded basic convolutional layer, a basic normalization layer, and a basic pulse neuron. The basic pulse neural module in the basic network is adjusted using the model adjustment parameters to obtain a modulated basic network, including:

[0023] Performing parameter adjustment on the basic convolutional layer using the synaptic weight modulation matrix and the synaptic bias modulation matrix to obtain an adjusted convolutional layer;

[0024] Using the synaptic weight modulation matrix to adjust the parameters of the basic spiking neuron to obtain an adjusted spiking neuron;

[0025] The basic convolutional layer is replaced by the adjusted convolutional layer, and / or the basic spiking neuron is replaced by the adjusted spiking neuron to obtain the modulated basic network.

[0026] In some embodiments, the step of adjusting parameters of the base convolution layer using the synaptic weight modulation matrix and the synaptic bias modulation matrix to obtain an adjusted convolution layer includes:

[0027] Adding the identity matrix to the synaptic weight modulation matrix to obtain a synaptic unit weight matrix;

[0028] Performing element-by-element multiplication on the synaptic unit weight matrix and the initial synaptic weight matrix of the base convolutional layer to obtain an adjusted synaptic weight matrix;

[0029] Adding the synaptic bias modulation matrix and the initial synaptic bias matrix of the base convolutional layer to obtain an adjusted synaptic bias matrix;

[0030] Parameters of the basic convolutional layer are adjusted based on the adjusted synaptic weight matrix and the adjusted synaptic bias matrix to obtain the adjusted convolutional layer.

[0031] In some embodiments, the step of adjusting parameters of the basic spiking neuron using the synaptic weight modulation matrix to obtain an adjusted spiking neuron includes:

[0032] Obtaining the initial membrane potential voltage matrix of the basic spiking neuron;

[0033] Performing element-by-element multiplication on the synaptic weight modulation matrix and the initial membrane potential voltage matrix to obtain an adjusted membrane potential voltage matrix;

[0034] The parameters of the basic pulse neuron are adjusted according to the adjusted membrane potential voltage matrix to obtain the adjusted pulse neuron.

[0035] In some embodiments, the first linear module includes a linear layer, a second batch normalization layer, and a second spiking neuron; and inputting the task encoding vector into the first linear module for vector dimensionality reduction to obtain the task embedding vector includes:

[0036] Performing linear feature extraction on the task encoding vector using the linear layer to obtain a first embedding vector;

[0037] performing batch normalization on the first embedding vector input using the second batch normalization layer to obtain a second embedding vector;

[0038] The second spiking neuron is used to perform spiking feature extraction on the second embedding vector to obtain the task embedding vector.

[0039] To achieve the above objectives, a second aspect of an embodiment of the present application provides a multi-task image classification device, comprising:

[0040] Pulse feature extraction module: used to extract pulse features from the sample image sequence in the support dataset using at least one pulse neural module cascaded in the task encoder to obtain a task encoding vector;

[0041] A basic parameter adjustment module is configured to generate parameters based on the task encoding vector using a parameter generator to obtain model adjustment parameters, and to adjust parameters of a basic pulse neural module in a basic network using the model adjustment parameters to obtain a modulated basic network, wherein the structure of the basic network is consistent with that of the task encoder;

[0042] A class prototype vector calculation module is configured to perform feature mapping on the sample image sequence using the modulation base network to obtain a sample embedding vector, and average the sample embedding vectors belonging to the same class in the support data set to obtain a class prototype vector;

[0043] Classification module: used to obtain the image sequence to be processed in the query data set, and determine the classification result of the image sequence to be processed according to the vector distance between the image sequence to be processed and each class prototype vector.

[0044] To achieve the above-mentioned purpose, the third aspect of an embodiment of the present application proposes an electronic device, which includes a memory and a processor, wherein the memory stores a computer program, and the processor implements the method described in the first aspect when executing the computer program.

[0045] To achieve the above-mentioned purpose, the fourth aspect of the embodiment of the present application proposes a storage medium, which is a storage medium. The storage medium stores a computer program, and when the computer program is executed by a processor, the method described in the first aspect is implemented.

[0046] The multi-task image classification method, apparatus, device, and storage medium proposed in the embodiments of the present application utilize at least one cascaded spiking neural module in a task encoder to extract spiking features from a sample image sequence in a support dataset, obtaining a task encoding vector. A parameter generator then generates parameters based on the task encoding vector to obtain model adjustment parameters, and uses the model adjustment parameters to adjust the parameters of the basic spiking neural modules in a base network, obtaining a modulated base network, wherein the base network has the same structure as the task encoder. The modulated base network then performs feature mapping on the sample image sequence to obtain a sample embedding vector. The embedding vectors of samples belonging to the same class in the support dataset are averaged to obtain a class prototype vector. Finally, a to-be-processed image sequence in a query dataset is obtained, and a classification result for the to-be-processed image sequence is determined based on the vector distance between the to-be-processed image sequence and each class prototype vector. In the embodiments of the present application, for each sample image sequence in each meta-learning task in a small number of sample support datasets, a task encoder and parameter generator are used to generate different model adjustment parameters for each sample image sequence. The base network is then parameterized based on these model adjustment parameters. The adjusted base network is then used to extract feature vectors of the sample image sequence, thereby obtaining a meta-learning task prototype. Finally, based on these task prototypes, the image sequences in the query dataset are classified. This approach requires only a single, integrated image classification model to classify image data from different tasks and modalities. Furthermore, because the time steps of each sample image sequence are essentially the same, training time and overhead are reduced, improving overall classification performance and efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 Schematic diagram showing the differences between the multi-task image classification model of the embodiment of the present application and the related models in the related art.

[0048] Figure 2 This is a flowchart of the multi-task image classification method provided in an embodiment of the present application.

[0049] Figure 3 This is a flowchart of the multi-task image classification method provided in an embodiment of the present application.

[0050] Figure 4 Schematic diagram of the structure of the task encoder provided in the embodiment of the present application.

[0051] Figure 5 This is a flowchart provided by an embodiment of the present application for inputting a task encoding vector into a first linear module for vector dimensionality reduction to obtain a task embedding vector.

[0052] Figure 6 This is a flowchart of an embodiment of the present application for using a second linear module to predict parameters of a task embedding vector to obtain model adjustment parameters.

[0053] Figure 7 This is a flowchart of an embodiment of the present application for adjusting the parameters of a basic pulse neural module in a basic network using model adjustment parameters to obtain a modulated basic network.

[0054] Figure 8 This is a flowchart of an embodiment of the present application for adjusting parameters of a basic convolutional layer using a synaptic weight modulation matrix and a synaptic bias modulation matrix to obtain an adjusted convolutional layer.

[0055] Figure 9 This is a schematic diagram of adjusting the parameters of the basic convolutional layer based on adjusting the synaptic weight matrix and adjusting the synaptic bias matrix provided in an embodiment of the present application.

[0056] Figure 10 This is a flowchart of an embodiment of the present application for adjusting the parameters of a basic spiking neuron using a synaptic weight modulation matrix to obtain an adjusted spiking neuron.

[0057] Figure 11 This is a schematic diagram of adjusting the parameters of a basic pulse neuron according to an adjustment of a membrane potential voltage matrix provided in an embodiment of the present application.

[0058] Figure 12 This is a schematic diagram of the firing rates of different multi-task image classification models provided in the embodiments of the present application in five static image datasets.

[0059] Figure 13 This is a schematic diagram of the firing rates of different multi-task image classification models provided in the embodiments of the present application in 5 static image datasets and 2 dynamic neural datasets.

[0060] Figure 14 It is a performance diagram of the basic pulse neuron in each basic pulse neural module in the basic network provided in the embodiment of the present application.

[0061] Figure 15 This is a diagram of the task migration performance of the multi-task image classification model provided in an embodiment of the present application.

[0062] Figure 16 This is a schematic diagram comparing the classification accuracy of the multi-task image classification model provided in the embodiment of the present application and the models in the related art.

[0063] Figure 17 This is a structural block diagram of a multi-task image classification device provided by another embodiment of the present application.

[0064] Figure 18 This is a schematic diagram of the hardware structure of the electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0065] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0066] It should be noted that although the functional modules are divided in the device schematic and the logical order is shown in the flowchart, in some cases, the steps shown or described can be performed in a different order than the module division in the device or the order in the flowchart.

[0067] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which this application pertains. The terms used herein are for the purpose of describing the embodiments of this application only and are not intended to limit this application.

[0068] First, let’s analyze some of the terms used in this application:

[0069] Artificial intelligence (AI) is a new technical discipline that studies and develops theories, methods, technologies, and application systems for simulating, extending, and expanding human intelligence. A branch of computer science, AI seeks to understand the essence of intelligence and produce new intelligent machines that can respond in a manner similar to human intelligence. Research in this field includes robotics, speech recognition, image recognition, natural language processing, and expert systems. AI can simulate the information processes of human consciousness and thinking. It also encompasses the theories, methods, technologies, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, to perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results.

[0070] Spiking Neural Networks (SNNs) are a type of biomimetic neural network model that mimics the information transmission method of biological neurons. In SNNs, information is transmitted through pulses, and pulses are only fired when the neuron's membrane potential exceeds a certain threshold. Different information is represented by the time interval or frequency of the pulses.

[0071] In related technologies, when spiking neural networks (SNNs) are applied to multimodal and multiple meta-learning task scenarios for image classification, a corresponding classification network is usually trained for the meta-learning task of each modality. However, the problem with this approach is that, on the one hand, more models are required to handle different classification tasks separately, and different tasks need to be trained separately, resulting in a large training overhead, which limits the practical application of spiking neural networks (SNNs) in the field of meta-learning. On the other hand, since the time steps of image data of different modalities are not uniform under different tasks, for example, the time step for static image data may be 12, but the time step for dynamic image data may be 100, resulting in a high learning and training cost, making the overall classification efficiency low.

[0072] Based on this, the embodiments of the present application provide a multi-task image classification method, apparatus, device and storage medium, which uses a task encoder and a parameter generator to generate different model adjustment parameters for each sample image sequence in each meta-learning task in a supporting data set of a small number of samples. The parameters of the basic network are then adjusted based on these model adjustment parameters. The basic network after adjusting the parameters is then used to extract the feature vector of the sample image sequence, and the meta-learning task prototype is obtained based on this. Finally, the image sequence to be processed in the query data set is classified based on these task prototypes. This method only requires an overall image classification model to realize the classification process of different tasks and different modal image data. And because the time step of each sample image sequence is basically the same, the training time is short, the training overhead is reduced, and the overall classification performance and efficiency are improved.

[0073] The embodiments of the present application provide a multi-task image classification method, apparatus, device, and storage medium, which are specifically illustrated by the following embodiments. First, the multi-task image classification method in the embodiments of the present application is described.

[0074] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Among them, artificial intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines so that machines have the functions of perception, reasoning and decision-making.

[0075] Artificial intelligence (AI) technology is a comprehensive discipline encompassing a wide range of fields, encompassing both hardware and software technologies. Foundational AI technologies generally include sensors, specialized AI chips, cloud computing, distributed storage, big data processing, operating / interaction systems, and mechatronics. AI software technologies primarily encompass computer vision, speech processing, natural language processing, and machine learning / deep learning.

[0076] The multi-task image classification method provided in the embodiment of the present application relates to the field of image processing technology. The multi-task image classification method provided in the embodiment of the present application can be applied to a terminal, can be applied to a server side, or can be a computer program running in a terminal or a server side. For example, a computer program can be a native program or software module in an operating system; it can be a local (Native) application (APP, Application), that is, a program that needs to be installed in the operating system to run, such as a client that supports multi-task image classification, or it can be a small program, that is, a program that can be run only by downloading it to a browser environment; it can also be a small program that can be embedded in any APP. In short, the above-mentioned computer program can be an application, module or plug-in in any form. Among them, the terminal communicates with the server through a network. The multi-task image classification method can be executed by a terminal or a server, or by a terminal and a server in collaboration.

[0077] In some embodiments, the terminal can be a smartphone, tablet computer, laptop computer, desktop computer, or smartwatch. The server can be a standalone server or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms. It can also be a service node in a blockchain system, where each service node in the blockchain system forms a peer-to-peer (P2P) network. The P2P protocol is an application layer protocol that runs on top of the Transmission Control Protocol (TCP). The terminal and server can be connected via Bluetooth, Universal Serial Bus (USB), or a network, etc., and this embodiment does not impose any restrictions on this.

[0078] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, and the like. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, and the like that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments in which tasks are performed by remote processing devices connected via a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media, including storage devices.

[0079] It should be noted that in each specific embodiment of the present application, when it comes to the need to perform relevant processing based on data related to the user's identity or characteristics, such as user information, user behavior data, user historical data, and user location information, the user's permission or consent will be obtained first, and the collection, use, and processing of such data will comply with the relevant laws, regulations, and standards of the relevant countries and regions. In addition, when the embodiment of the present application needs to obtain the user's sensitive personal information, the user's separate permission or consent will be obtained through a pop-up window or by jumping to a confirmation page. After clearly obtaining the user's separate permission or consent, the necessary user-related data for the normal operation of the embodiment of the present application will be obtained.

[0080] The following describes a multi-task image classification model provided in an embodiment of the present application, which is used to perform a multi-task image classification method.

[0081] Reference Figure 1 , Figure 1 Schematic diagram showing the differences between the multi-task image classification model of the embodiment of the present application and the related models in the related art. Figure 1 As can be seen in the above, in the implementation scheme of the related art, a corresponding classification network is trained for each modality meta-learning task. Specifically, different spiking neural networks are set for NS different meta-learning tasks, and then each spiking neural network extracts features based on the image sequence of the corresponding task to obtain corresponding features. However, in the technical solution of the embodiment of the present application, in the related art, for NS different meta-learning tasks, only a multi-task image classification model built based on spiking neural networks (SNNs) is required.

[0082] It is understandable that the meta-learning tasks here can be different image classification and recognition tasks, such as recognizing text in images, recognizing people in images, recognizing specific animals in images, etc., which can be set according to actual needs.

[0083] Next, the multi-task image classification method in an embodiment of the present application is described.

[0084] Figure 2 This is an optional flowchart of the multi-task image classification method provided in the embodiment of the present application. Figure 2 The method may include but is not limited to steps 110 to 140. It is also understood that this embodiment is Figure 2 The order of step 110 to step 140 is not specifically limited, and the order of steps can be adjusted or some steps can be reduced or added according to actual needs.

[0085] Step 110: Utilize at least one pulse neural module cascaded in the task encoder to perform pulse feature extraction on the sample image sequence in the support data set to obtain a task encoding vector.

[0086] In one embodiment, in the meta-learning task, the overall data set is divided into a support data set and a query data set. The support data set contains a sample image sequence, while the query data set contains an image sequence to be processed. The number of image sequences in the support data set is much smaller than the number of image sequences in the query data set. At the same time, the sample image sequences in the support data set have more than one modality, where the modality refers to static images or dynamic images. According to the specific classification requirements of the task, each type includes at least one sample image sequence. The goal of the embodiment of the present application is to use a support data set of a small number of samples to perform class prototype learning, so as to obtain class prototypes that represent the corresponding classifications under different tasks. Then, we use these class prototypes to perform a specific classification process on a large number of samples in the query data set.

[0087] Reference Figure 3 , Figure 3 This is a flowchart of the multi-task image classification method provided in an embodiment of the present application. Figure 3 In this paper, the multi-task image classification model consists of a parameter modulation network and a base network. The parameter modulation network includes a task encoder and a parameter generator. The task encoder extracts features from the sample image sequence to obtain a task encoding vector. Next, the parameter generator calculates model adjustment parameters based on the task encoding vector. Based on the model adjustment parameters, the base network is modulated by synaptic weight or voltage modulation to obtain a modulated base network. The modulated base network is then used to process the sample image sequence, ultimately obtaining a class prototype vector. Finally, the class prototype vector is used to obtain the corresponding classification result for the image sequence to be processed in the query dataset.

[0088] In one embodiment, referring to Figure 4 , Figure 4 Schematic diagram of the structure of the task encoder provided in the embodiment of the present application. Figure 4 The task encoder includes multiple cascaded pulse neural modules, of which 4 are used as an example in the figure, such as the cascaded first pulse convolution module, the second pulse convolution module, the third pulse convolution module and the fourth pulse convolution module, but this does not limit their number. Each pulse neural module includes a cascaded convolution layer, a first batch of normalization layers and a first pulse neuron, wherein the output channels of the convolution layers in the first pulse convolution module, the second pulse convolution module, the third pulse convolution module and the fourth pulse convolution module gradually increase. Assuming that the output dimension of the convolution layer in the fourth pulse convolution module is D, the output channel of the convolution layer in the first pulse convolution module is D / 8, the output channel of the convolution layer in the second pulse convolution module is D4, and the output channel of the convolution layer in the third pulse convolution module is D / 2.

[0089] In one embodiment, combining Figure 4In the embodiment of the present application, the process of using the convolution module in the task encoder to extract features of the sample image sequence in the support data set to obtain the task coding vector includes: using the first pulse convolution module to extract pulse features of the sample image sequence to obtain the first image feature; using the second pulse convolution module to extract pulse features of the first image feature to obtain the second image feature; using the third pulse convolution module to extract pulse features of the second image feature to obtain the third image feature; using the fourth pulse convolution module to extract pulse features of the third image feature to obtain the task coding vector.

[0090] It can be seen that the embodiment of the present application uses four cascaded pulse convolution modules to extract features in sequence. As the number of channels increases, each convolution kernel can capture more local features and learn richer feature representations, thereby better retaining the spatial information of the sample image sequence.

[0091] In one embodiment, each pulse convolution module uses a convolution layer to obtain preliminary image features based on input data and then performs pulse feature extraction. Figure 4 The process of extracting spike features from the preliminary image features using the first spike neuron to obtain the task encoding vector specifically includes: using the first normalization layer to batch normalize the preliminary image features to obtain normalized image features, and then using the first spike neuron to extract spike features from the normalized image features to obtain the output vector. The input of the first spike convolution module is the sample image sequence, and the output of the last spike convolution module is the task encoding vector.

[0092] Among them, the number of channels of the first batch of normalization layers is twice the dimension of the received input data, and the preliminary image features output by the convolutional layer are converted into two-dimensional normalized image features, with the aim of having the same scale and distribution, thereby improving the stability of the training process. The first normalized image features are represented as a two-dimensional matrix, and each element represents the eigenvalue of the corresponding position. Next, the normalized image features are processed and encoded using the first pulse neuron, for example, pulse encoding is performed according to preset rules such as thresholds and emission frequencies, thereby obtaining a discrete pulse sequence. Through the pulse encoding process of the first pulse neuron, the two-dimensional normalized image features can be converted into a D-dimensional task coding vector. In the embodiment of the present application, the use of pulse neurons enables the multi-task image classification model to better simulate the working mode of the biological brain, improve the efficiency and accuracy of feature extraction, and thus improve the overall classification accuracy.

[0093] Step 120: Utilize a parameter generator to generate parameters based on the task coding vector to obtain model adjustment parameters, and utilize the model adjustment parameters to adjust parameters of the basic pulse neural module in the basic network to obtain a modulated basic network.

[0094] In one embodiment, combining Figure 3 After obtaining the task encoding vector, a parameter generator is used to generate parameters based on the task encoding vector to obtain model adjustment parameters. The parameter generator includes a first linear module and a second linear module. Therefore, the process of using the parameter generator to generate parameters based on the task encoding vector to obtain model adjustment parameters specifically includes: using the first linear module to perform vector dimensionality reduction on the task encoding vector to obtain a task embedding vector, and using the second linear module to perform parameter prediction on the task embedding vector to obtain model adjustment parameters.

[0095] In one embodiment, the first linear module includes: a linear layer, a second batch of normalization layers, and a second pulse neuron. Figure 5 , Figure 5 This is a flowchart of inputting a task encoding vector into a first linear module for vector dimensionality reduction to obtain a task embedding vector, provided by an embodiment of the present application, specifically including the following steps:

[0096] Step 510: Use a linear layer to perform linear feature extraction on the task encoding vector to obtain a first embedding vector.

[0097] Step 520: Use a second batch normalization layer to batch normalize the first embedding vector input to obtain a second embedding vector.

[0098] Step 530: Use the second spiking neuron to extract the spiking features of the second embedding vector to obtain the task embedding vector.

[0099] In one embodiment, the first linear module uses a linear layer to perform a linear transformation on the input task encoding vector, extracts linear features, expresses the different components of the task encoding vector in the form of a linear combination, and obtains a first embedding vector. This process reduces the dimension of the task encoding vector and maps it to a lower dimensional space, which helps to reduce the computational complexity. Next, based on the first embedding vector, the second batch normalization layer performs batch normalization processing on it to standardize the features so that they have the same distribution and scale, thereby avoiding the problem of gradient disappearance or explosion during training and accelerating the training process. Finally, the second embedding vector processed by the second batch normalization layer is sent to the second pulse neuron for further feature extraction. Similar to the first pulse neuron, the second pulse neuron outputs a discrete pulse sequence as the task embedding vector.

[0100] In one embodiment, the second linear module then performs a parameter prediction process of the basic network based on the task embedding vector. The second linear module includes: a first fully connected layer, a second fully connected layer, and a third fully connected layer, and the model adjustment parameters include: a synaptic weight modulation matrix and a synaptic bias modulation matrix. Figure 6 , Figure 6This is a flowchart of using the second linear module to predict the parameters of the task embedding vector and obtain the model adjustment parameters provided by the embodiment of the present application, which specifically includes the following steps:

[0101] Step 610: Use the first fully connected layer to extract features from the task embedding vector to obtain a first parameter vector, and use the second fully connected layer to extract features from the task embedding vector to obtain a second parameter vector, and perform an outer product calculation on the first parameter vector and the second parameter vector to obtain a synaptic weight modulation matrix.

[0102] In one embodiment, the second linear module uses the first fully connected layer and the second fully connected layer to extract features from the task embedding vector, respectively obtaining a first parameter vector and a second parameter vector, where the first parameter vector and the second parameter vector can be regarded as representations of different aspects of the task embedding vector learned through the network. Next, the outer product of the first parameter vector and the second parameter vector is calculated to obtain a synaptic weight modulation matrix. The synaptic weight modulation matrix obtained in this way can be regarded as a weight matrix obtained by dynamically adjusting the input task embedding vector, which can adjust the strength of the connection between neurons under different inputs, thereby achieving adaptation to sample image sequences corresponding to different tasks.

[0103] In one embodiment, the first parameter vector is expressed as: The second parameter vector is expressed as: Among them, v T represents the task embedding vector, φ1 represents the parameters of the first fully connected layer, φ2 represents the parameters of the second fully connected layer, and l represents the number of parameters that need to be adjusted in the basic network set according to actual needs.

[0104] Therefore, the synaptic weight modulation matrix is ​​expressed as:

[0105]

[0106] Among them, M l (v T ,φ) represents the synaptic weight modulation matrix, represents an outer product operation, where each element in the synaptic weight modulation matrix obtained by the outer product calculation is the product of the corresponding elements in the first parameter vector and the second parameter vector.

[0107] Step 620: Use the third fully connected layer to perform feature extraction on the task embedding vector to obtain a synaptic bias modulation matrix.

[0108] In one embodiment, the second linear module uses the third fully connected layer to perform feature extraction on the task embedding vector to generate a synaptic bias modulation matrix. This synaptic bias modulation matrix is ​​used to modulate the threshold of the activation function corresponding to spiking neurons in the base network, thereby adjusting the neuron's firing rate. Through this adjustment process, the base network can dynamically adjust the neuron's activation pattern based on the sample image sequences corresponding to different input tasks to adapt to different task requirements.

[0109] In one embodiment, the synaptic bias modulation matrix is ​​represented as:

[0110]

[0111] Among them, φ3 represents the parameters of the third fully connected layer.

[0112] The synaptic weight modulation matrix and synaptic bias modulation matrix ultimately generated in the above process are consistent with the weights and bias shapes of the relevant convolutional layers in the technical pulse neural module in the basic network. Since the embodiment of the present application hopes to modulate all relevant parameters in the basic network, a simple multiple fully connected layers are used for parameter generation. One advantage of this design is that the network is simple, and another advantage is that the number of parameters can be easily controlled. It is understandable that a multilayer perceptron (MLP) can also be used to replace the above fully connected layer.

[0113] Next, we describe the process of adjusting the parameters of the basic spiking neural module in the basic network using the two models above to obtain the modulated basic network. This adjustment process enables the basic network to adaptively learn different tasks, enabling it to better handle changing task requirements.

[0114] In one embodiment, the structure of the base network is consistent with that of the task encoder, and the base network includes cascaded basic spiking neural modules, wherein each basic spiking neural module includes a cascaded basic convolutional layer, a basic normalization layer, and a basic spiking neuron, which are identical to the corresponding network structure in the task encoder. The process of adjusting the parameters of the base network is specifically divided into synaptic weight modulation (SML) and voltage modulation (VML) according to different adjustment targets.

[0115] The two adjustment processes are described below.

[0116] Reference Figure 7 , Figure 7 The present invention provides a flowchart of using the model adjustment parameters to adjust the parameters of the basic pulse neural module in the basic network to obtain a modulated basic network, which specifically includes the following steps:

[0117] Step 710: Parameter adjustment is performed on the base convolution layer using the synaptic weight modulation matrix and the synaptic bias modulation matrix to obtain an adjusted convolution layer.

[0118] In one embodiment, the above steps refer to synaptic weight modulation, the purpose of which is to use the synaptic weight modulation matrix and the synaptic bias modulation matrix to adjust the parameters of the basic convolution layer to obtain an adjusted convolution layer after parameter adaptation. Figure 8 , Figure 8 The present invention provides a flowchart for adjusting the parameters of a base convolution layer using a synaptic weight modulation matrix and a synaptic bias modulation matrix to obtain an adjusted convolution layer, which specifically includes the following steps:

[0119] Step 810: Add the identity matrix to the synaptic weight modulation matrix to obtain a synaptic unit weight matrix.

[0120] Step 820: Perform element-by-element multiplication on the synaptic unit weight matrix and the initial synaptic weight matrix of the base convolutional layer to obtain an adjusted synaptic weight matrix.

[0121] In one embodiment, the synaptic unit weight matrix is ​​expressed as:

[0122] J+M l (v T ,φ)

[0123] In one embodiment, adjusting the synaptic weight matrix is ​​expressed as:

[0124]

[0125] in, Represents the adjustment of the synaptic weight matrix, W l represents the initial synaptic weight matrix, and ⊙ represents element-by-element multiplication, which is used to reduce the number of parameters.

[0126] Step 830: Add the synaptic bias modulation matrix and the initial synaptic bias matrix of the base convolutional layer to obtain an adjusted synaptic bias matrix.

[0127] In one embodiment, adjusting the synaptic bias matrix is ​​expressed as:

[0128]

[0129] in, represents the adjustment of the synaptic bias matrix, b l represents the initial synaptic bias matrix.

[0130] Step 840: Adjust parameters of the basic convolutional layer based on the adjusted synaptic weight matrix and the adjusted synaptic bias matrix to obtain an adjusted convolutional layer.

[0131] In one embodiment, taking one of the basic pulse neural modules in the basic network as an example, referring to Figure 9 , Figure 9 This is a schematic diagram of adjusting the parameters of the basic convolutional layer based on adjusting the synaptic weight matrix and adjusting the synaptic bias matrix provided in an embodiment of the present application. Figure 9 In

[15] , the parameters of the basic convolutional layer of the basic spiking neuron module are adjusted by adjusting the synaptic weight matrix and the synaptic bias matrix to obtain the adjusted convolutional layer. Next, the input data is input into the cascade of the adjusted convolutional layer, the basic batch normalization layer, and the basic spiking neuron to obtain the output data.

[0132] Step 720: Use the synaptic weight modulation matrix to adjust the parameters of the basic spiking neuron to obtain an adjusted spiking neuron.

[0133] In one embodiment, referring to Figure 10 , Figure 10 The flowchart of adjusting the parameters of the basic spiking neuron using the synaptic weight modulation matrix to obtain the adjusted spiking neuron provided by the embodiment of the present application specifically includes the following steps:

[0134] Step 1010: Obtain the initial membrane potential voltage matrix of the basic spiking neuron.

[0135] Step 1020: Perform element-by-element multiplication on the synaptic weight modulation matrix and the initial membrane potential voltage matrix to obtain an adjusted membrane potential voltage matrix.

[0136] In one embodiment, when using the synaptic weight modulation process to train the model, there is a certain pulse attenuation phenomenon, that is, the basic pulse neuron layer of the basic pulse neural module in the basic network gradually decreases as the number of layers of the basic pulse neural module deepens. Its firing rate is about 0.02 at the deepest layer. The extremely low firing rate causes the learned pulse sequence features to be extremely sparse, thereby affecting its meta-learning performance in multimodal multi-task. Therefore, the embodiment of the present application proposes a voltage modulation process to directly modulate the membrane potential of the basic pulse neurons in the basic network, thereby increasing its firing rate, obtaining richer pulse sequence features, and ultimately improving its multimodal multi-task meta-learning performance.

[0137] In voltage modulation, the modulated parameter is considered a neurotransmitter. Neurotransmitters are released from the presynaptic junction of one neuron and bind to receptors on the postsynaptic membrane of another neuron. Depending on the type of receptor they bind to, neurotransmitters can have either an excitatory effect (e.g., glutamate) or an inhibitory effect (e.g., glycine) on the postsynaptic neuron. Therefore, adjusting the membrane potential voltage matrix to regulate receptor type can alter the relative membrane potential of the underlying spiking neuron.

[0138] In one embodiment, the membrane potential voltage matrix is ​​expressed as:

[0139]

[0140] in, Represents the voltage matrix for adjusting the membrane potential, V l represents the initial membrane potential voltage matrix.

[0141] It is understandable that the voltage modulation process can directly use the synaptic weight modulation matrix generated by the synaptic weight modulation process, or set two other fully connected layers to regenerate a matrix similar to the synaptic weight modulation matrix to participate in the calculation. This embodiment does not limit this.

[0142] In one embodiment, after using voltage modulation, the firing rate of the corresponding basic pulse neurons in the basic network is improved, and the firing rate of the deepest layer is increased from about 0.02 to about 0.08. At the same time, the accuracy of multimodal multi-task meta-learning is also improved.

[0143] Step 1030: Adjust the parameters of the basic spiking neuron according to the adjusted membrane potential voltage matrix to obtain an adjusted spiking neuron.

[0144] In one embodiment, taking one of the basic pulse neural modules in the basic network as an example, referring to Figure 11 , Figure 11 This is a schematic diagram of adjusting the parameters of a basic pulse neuron according to an adjustment of a membrane potential voltage matrix provided in an embodiment of the present application. Figure 11 In

[15] , the membrane potential of the basic spiking neuron is adjusted by adjusting the synaptic weight matrix to obtain the adjusted spiking neuron. Next, the input data is input into the cascade of the basic convolutional layer, the basic batch normalization layer, and the modulated spiking neuron to obtain the output data.

[0145] Step 730: Replace the basic convolutional layer with the adjusted convolutional layer, and / or replace the basic spiking neuron with the adjusted spiking neuron to obtain a modulated basic network.

[0146] In one embodiment, an adjustment convolution layer is obtained by synaptic weight modulation, and an adjustment pulse neuron is obtained by voltage modulation. These two adjustment processes can be selected either one or both. Therefore, for each basic pulse neural module, the basic convolution layer can be replaced by the adjustment convolution layer, and / or the basic pulse neuron can be replaced by the adjustment pulse neuron to obtain a modulated pulse neural module, thereby obtaining a modulated basic network corresponding to the sample image sequence.

[0147] Step 130: Use the modulation basis network to perform feature mapping on the sample image sequence to obtain a sample embedding vector, and calculate the average of the sample embedding vectors belonging to the same class in the support data set to obtain a class prototype vector.

[0148] In one embodiment, the network parameters of the modulation base network are expressed as The modulation base network can be expressed as an embedding function Therefore, the modulation base network is used to perform feature mapping on the sample image sequence, and the sample embedding vector is expressed as:

[0149]

[0150] in, represents a sample image sequence, Represents the sample embedding vector.

[0151] In one embodiment, the sample embedding vector is calculated for each sample image sequence in the supporting dataset according to the above process, obtaining the sample embedding vector corresponding to each sample image sequence. Then, according to the pre-defined classes corresponding to different tasks, the sample embedding vectors belonging to the same class in the supporting dataset are used as the vector set corresponding to that class, that is, the vector set corresponding to each class is obtained.

[0152] Next, the average vector of all sample embedding vectors in each vector set is calculated as the class prototype vector corresponding to the class, which is expressed as:

[0153]

[0154] Where N is the total number of classes, c n Represents the class prototype vector corresponding to class n, represents the set of sample image sequences corresponding to class n, Represents the number of sample image sequences in the vector set.

[0155] Step 140: Obtain a sequence of images to be processed in the query data set, and determine a classification result of the sequence of images to be processed based on the vector distance between the sequence of images to be processed and each class prototype vector.

[0156] In one embodiment, after obtaining the class prototype vectors of each class in different tasks based on the sample image sequence in the supporting data set of a small number of samples, for the image sequence to be processed in the query data set, it is only necessary to calculate the vector distance between each image sequence to be processed and each class prototype vector, and select the class corresponding to the minimum value of the vector distance as the classification result of the image sequence to be processed.

[0157] In one embodiment, during the training process, the multi-task image classification model uses vector distance to obtain query samples. The predicted category distribution is then compared with the category label to calculate the loss value, where the loss value is calculated using the cross-entropy loss function. This loss value is then optimized using stochastic gradient descent, which involves adjusting all parameters in the base network, task encoder, and parameter generator.

[0158] During the above training process, the category distribution Expressed as:

[0159]

[0160] d(x,y)=(xy) 2

[0161] in, Represents a query sample The vector distance between the prototype vector of class n and the class n, the denominator is the distance between all possible classes n ′ The exponential function values ​​are normalized to ensure that the sum of all probabilities is 1.

[0162] It can be seen that the embodiment of the present application utilizes the temporal processing characteristics of spiking neurons, combined with the multi-tasking mechanism of brain science, to synchronously optimize the basic convolutional layer and basic spiking neurons of the basic network during the calculation process. Unified training is performed for static image data and dynamic neural data, and the time step is unified to 8, so as to better adapt to multimodal multi-tasking scenarios. In addition, the meta-learning mechanism solves the problem of low multi-task performance when data is scarce, and improves the accuracy of multimodal multi-task image classification.

[0163] The following describes the verification process of the multi-task image classification method according to an embodiment of the present application.

[0164] In one embodiment, the verification process of five static image datasets (Omniglot, mini-ImageNet, FC100, CUB, and Aircraft) and two dynamic neural datasets (Double NMNIST and Double ASL-DVS) is taken as an example. At the same time, in order to verify the effect of synaptic weight modulation and voltage modulation on performance improvement, the multi-task image classification model without synaptic weight modulation and voltage modulation is denoted as SMILE, the multi-task image classification model with only synaptic weight modulation is denoted as SMILE+S, the multi-task image classification model with only voltage modulation is denoted as SMILE+V, and the multi-task image classification model with both synaptic weight modulation and voltage modulation is denoted as SMILE+S+V. Four pulse neural networks in related technologies are introduced for performance comparison, namely, prototype-based meta-learning method (ProtoNet), model-agnostic meta-learning method (MAML), prototype-based multi-task meta-learning method (MProtoNet), and model-agnostic multi-task meta-learning method (MMAML).

[0165] Reference Figure 12 , Figure 12 This is a schematic diagram of the firing rates of different multi-task image classification models provided in the embodiments of the present application in five static image datasets. Figure 12 As can be seen from the figure, the firing rate of the multi-task image classification model SMILE+S+V, which performs both synaptic weight modulation and voltage modulation, has the highest relative trend.

[0166] Reference Figure 13 , Figure 13 This is a schematic diagram of the firing rate of different multi-task image classification models provided by the embodiment of this application in 5 static image datasets and 2 dynamic neural datasets. The depth of the horizontal axis refers to the order of the basic pulse neural module in the basic network. Figure 13 As can be seen, the firing rates of the multi-task image classification model SMILE+V that performs only voltage modulation and the multi-task image classification model SMILE+S+V that performs both synaptic weight modulation and voltage modulation are relatively trending the highest.

[0167] Reference Figure 14 , Figure 14Schematic diagram of the performance of the basic spiking neurons in each basic spiking neural module in the basic network provided by the embodiment of the present application. For each basic spiking neuron, the corresponding firing rate is verified under five different experimental settings. The experimental settings are: 2mode, 2mode+, 3mode, 5mode, and 7mode, where 2mode represents the selection of two datasets, Omniglot and mini-ImageNet; 2mode+ represents the selection of two datasets, mini-ImageNet and FC100; 3mode represents the selection of three datasets, Omniglot, mini-ImageNet, and FC100; 5mode represents the selection of five datasets, Omniglot, mini-ImageNet, FC100, CUB, and Aircraft; and 7mode represents the selection of seven datasets, Omniglot, mini-ImageNet, FC100, CUB, Aircraft, DoubleNMNIST, and DoubleASL-DVS. It can be seen that regardless of which basic pulse neuron, the firing rates of the multi-task image classification model SMILE+V that performs only voltage modulation and the multi-task image classification model SMILE+S+V that performs both synaptic weight modulation and voltage modulation are relatively trending the highest.

[0168] Reference Figure 15 , Figure 15 This is a diagram of the task migration performance of the multi-task image classification model provided in an embodiment of the present application. Figure 15 Figure 2 shows the accuracy of a multi-task image classification model after pre-training on two small datasets (mini-ImageNet and FC100) and fine-tuning on five static image datasets (Omniglot, mini-ImageNet, FC100, CUB, and Aircraft) and two dynamic neural network datasets (Double NMNIST and Double ASL-DVS). The overall accuracy shows an upward trend.

[0169] Reference Figure 16 , Figure 16 This is a schematic diagram comparing the classification accuracy of the multi-task image classification model provided in the embodiment of the present application and the models in the related art. Figure 16Figure 2 shows the accuracy of the multi-task image classification model and four models in the related art after pre-training on two small datasets (mini-ImageNet and FC100) and fine-tuning on five static image datasets (Omniglot, mini-ImageNet, FC100, CUB, and Aircraft) and two dynamic neural datasets (DoubleNMNIST and Double ASL-DVS). It can be seen that the overall accuracy of the multi-task image classification model of the embodiment of the present application is on an upward trend and is higher than that of the models in the related art.

[0170] Refer to Table 1 below, which compares the classification accuracy of the multi-task image classification model provided in the embodiment of the present application and the model in the related art.

[0171]

[0172]

[0173] Table 1 Comparison of classification accuracy

[0174] It can be seen that the overall accuracy of the multi-task image classification model in the embodiment of the present application is higher than that of the model in the related art.

[0175] Refer to Table 2 below, which is a comparison of the accuracy of single-task training of the multi-task image classification model provided in an embodiment of the present application on two dynamic neural datasets.

[0176]

[0177] Table 2 Comparison of single-task training accuracy on two dynamic neural datasets

[0178] It can be seen that the overall accuracy of the multi-task image classification model that has undergone at least one of synaptic weight modulation and voltage modulation in the embodiment of the present application is higher than that of the model without modulation.

[0179] Refer to Table 3 below, which shows the ablation experiment results of the multi-task image classification model provided in the embodiment of the present application in terms of time step.

[0180]

[0181]

[0182] Table 3 Ablation experiment results in terms of time step

[0183] It can be seen that as the time step increases, the accuracy also improves.

[0184] Refer to Table 4, which is a comparison of the accuracy of single-task training of the multi-task image classification model provided in an embodiment of the present application on five static image datasets.

[0185]

[0186] Table 4 Comparison of single-task training accuracy on five static image datasets

[0187] It can be seen that the multi-task image classification model SMILE+V that only performs voltage modulation and the multi-task image classification model SMILE+S+V that performs both synaptic weight modulation and voltage modulation have the highest accuracy.

[0188] Refer to Table 5, which is a comparison of the accuracy of single-task training of the multi-task image classification model provided in an embodiment of the present application on seven data sets.

[0189]

[0190]

[0191] Table 5 Comparison of single-task training accuracy on seven datasets

[0192] It can be seen that the multi-task image classification model SMILE+V that only performs voltage modulation and the multi-task image classification model SMILE+S+V that performs both synaptic weight modulation and voltage modulation have the highest accuracy.

[0193] From the above figures and tables, it can be seen that the multi-task image classification model of the embodiments of the present application can achieve a high accuracy in the meta-learning scenario.

[0194] The technical solution provided by the embodiments of the present application utilizes at least one cascaded spiking neural module in a task encoder to extract spike features from a sample image sequence in a support dataset, generating a task encoding vector. A parameter generator then generates parameters based on the task encoding vector to obtain model adjustment parameters, which are then used to adjust the parameters of the basic spiking neural modules in a base network, resulting in a modulated base network. The base network structure is consistent with that of the task encoder. The modulated base network then performs feature mapping on the sample image sequence to obtain a sample embedding vector. The embedding vectors of samples belonging to the same class in the support dataset are averaged to obtain a class prototype vector. Finally, a to-be-processed image sequence in a query dataset is obtained, and a classification result for the to-be-processed image sequence is determined based on the vector distance between the to-be-processed image sequence and each class prototype vector. In the embodiments of the present application, a task encoder and parameter generator are used to generate different model adjustment parameters for each sample image sequence in each meta-learning task in a small number of sample support datasets. The base network is then parameterized based on these model adjustment parameters. The adjusted base network is then used to extract feature vectors of the sample image sequence, thereby obtaining meta-learning task prototypes. Finally, the to-be-processed image sequence in the query dataset is classified based on these task prototypes. This approach requires only a single, integrated image classification model to classify image data from different tasks and modalities. Furthermore, because the time steps of each sample image sequence are essentially the same, training time and overhead are reduced, improving overall classification performance and efficiency.

[0195] The present application also provides a multi-task image classification device, which can implement the multi-task image classification method. Figure 17 , the device comprises:

[0196] Pulse feature extraction module 1710: used to extract pulse features from the sample image sequence in the support data set using at least one pulse neural module cascaded in the task encoder to obtain a task coding vector.

[0197] Basic parameter adjustment module 1720: used to use the parameter generator to generate parameters based on the task encoding vector to obtain model adjustment parameters, and use the model adjustment parameters to adjust the parameters of the basic pulse neural module in the basic network to obtain a modulated basic network. The structure of the basic network is consistent with the task encoder.

[0198] Class prototype vector calculation module 1730: used to use the modulation basis network to perform feature mapping on the sample image sequence to obtain the sample embedding vector, and calculate the average of the sample embedding vectors belonging to the same class in the support data set to obtain the class prototype vector.

[0199] Classification module 1740: used to obtain a sequence of images to be processed in the query data set, and determine a classification result of the sequence of images to be processed according to the vector distance between the sequence of images to be processed and each class prototype vector.

[0200] The specific implementation of the multi-task image classification device of this embodiment is basically the same as the specific implementation of the multi-task image classification method described above, and will not be repeated here.

[0201] An embodiment of the present application further provides an electronic device, including:

[0202] at least one memory;

[0203] at least one processor;

[0204] at least one program;

[0205] The program is stored in the memory, and the processor executes the at least one program to implement the multi-task image classification method described above. The electronic device can be any smart terminal including a mobile phone, a tablet computer, a personal digital assistant (PDA), an in-vehicle computer, etc.

[0206] See also Figure 18 , Figure 18 The hardware structure of an electronic device according to another embodiment is shown. The electronic device includes:

[0207] The processor 1801 can be implemented as a general-purpose central processing unit (CPU), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application;

[0208] The memory 1802 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 1802 can store an operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program codes are stored in the memory 1802 and are called by the processor 1801 to execute the multi-task image classification method of the embodiments of this application.

[0209] Input / output interface 1803, used to implement information input and output;

[0210] Communication interface 1804, used to implement communication interaction between this device and other devices, which can be achieved through wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, WiFi, Bluetooth, etc.); and

[0211] Bus 1805 , which transmits information between various components of the device (e.g., processor 1801 , memory 1802 , input / output interface 1803 , and communication interface 1804 );

[0212] The processor 1801 , the memory 1802 , the input / output interface 1803 and the communication interface 1804 are connected to each other in communication within the device via the bus 1805 .

[0213] An embodiment of the present application further provides a storage medium, which is a storage medium storing a computer program. When the computer program is executed by a processor, the multi-task image classification method is implemented.

[0214] The memory, as a non-transient storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely arranged relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0215] The multi-task image classification method, apparatus, device, and storage medium proposed in the embodiments of the present application utilize at least one cascaded spiking neural module in a task encoder to extract spike features from a sample image sequence in a support dataset, obtaining a task encoding vector. A parameter generator then generates parameters based on the task encoding vector to obtain model adjustment parameters, and uses the model adjustment parameters to adjust the parameters of the basic spiking neural modules in a base network, obtaining a modulated base network, wherein the base network has the same structure as the task encoder. The modulated base network then performs feature mapping on the sample image sequence to obtain a sample embedding vector. The embedding vectors of samples belonging to the same class in the support dataset are averaged to obtain a class prototype vector. Finally, a to-be-processed image sequence in a query dataset is obtained, and a classification result for the to-be-processed image sequence is determined based on the vector distance between the to-be-processed image sequence and each class prototype vector. In the embodiments of the present application, for each sample image sequence in each meta-learning task in a small number of sample support datasets, a task encoder and parameter generator are used to generate different model adjustment parameters for each sample image sequence. The base network is then parameterized based on these model adjustment parameters. The adjusted base network is then used to extract feature vectors of the sample image sequence, thereby obtaining a meta-learning task prototype. Finally, based on these task prototypes, the image sequences in the query dataset are classified. This approach requires only a single, integrated image classification model to classify image data from different tasks and modalities. Furthermore, because the time steps of each sample image sequence are essentially the same, training time and overhead are reduced, improving overall classification performance and efficiency.

[0216] The embodiments described in the embodiments of this application are intended to more clearly illustrate the technical solutions of the embodiments of this application and do not constitute a limitation on the technical solutions provided by the embodiments of this application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of this application are also applicable to similar technical problems.

[0217] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0218] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, i.e., they may be located in one place or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of this embodiment.

[0219] Those skilled in the art will appreciate that all or some of the steps in the methods, systems, and functional modules / units in the devices disclosed above may be implemented as software, firmware, hardware, or appropriate combinations thereof.

[0220] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequential order. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0221] It should be understood that in this application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following items" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0222] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of the above-mentioned units is only a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0223] The units described above as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0224] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0225] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including multiple instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: various media that can store programs, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0226] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but are not intended to limit the scope of the present invention. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and essence of the present invention should be within the scope of the present invention.

Claims

1. A multi-task image classification method, characterized in that: include: Using at least one spiking neural module cascaded in the task encoder to extract spiking features from the sample image sequence in the support dataset to obtain a task encoding vector; Using a parameter generator to generate parameters based on the task encoding vector to obtain model adjustment parameters, and using the model adjustment parameters to adjust parameters of a basic spiking neural module in a basic network to obtain a modulated basic network, wherein the structure of the basic network is consistent with that of the task encoder; Performing feature mapping on the sample image sequence using the modulation base network to obtain a sample embedding vector, and averaging the sample embedding vectors belonging to the same class in the supporting data set to obtain a class prototype vector; An image sequence to be processed in a query data set is obtained, and a classification result of the image sequence to be processed is determined according to a vector distance between the image sequence to be processed and each of the class prototype vectors.

2. The multi-task image classification method according to claim 1, characterized in that: The at least one cascaded pulse neural module includes a first pulse convolution module, a second pulse convolution module, a third pulse convolution module, and a fourth pulse convolution module; the at least one cascaded pulse neural module in the task encoder is used to extract pulse features from a sample image sequence in a support data set to obtain a task encoding vector, including: Using the first pulse convolution module to perform pulse feature extraction on the sample image sequence to obtain a first image feature; performing pulse feature extraction on the first image feature using the second pulse convolution module to obtain a second image feature; performing pulse feature extraction on the second image feature using the third pulse convolution module to obtain a third image feature; The fourth pulse convolution module is used to perform pulse feature extraction on the third image feature to obtain the task coding vector.

3. The multi-task image classification method according to claim 2, characterized in that: The first pulse convolution module, the second pulse convolution module, the third pulse convolution module and the fourth pulse convolution module all include cascaded convolution layers, a first batch of normalization layers and a first pulse neuron, and the output channels of the convolution layers in the first pulse convolution module, the second pulse convolution module, the third pulse convolution module and the fourth pulse convolution module gradually increase.

4. The multi-task image classification method according to claim 1, characterized in that The parameter generator includes a first linear module and a second linear module, and the parameter generator is used to generate parameters based on the task encoding vector to obtain model adjustment parameters, including: Using the first linear module to perform vector dimensionality reduction on the task encoding vector to obtain a task embedding vector; The second linear module is used to perform parameter prediction on the task embedding vector to obtain the model adjustment parameters.

5. The multi-task image classification method according to claim 4, characterized in that: The second linear module includes: a first fully connected layer, a second fully connected layer, and a third fully connected layer. The model adjustment parameters include a synaptic weight modulation matrix and a synaptic bias modulation matrix. The second linear module is used to perform parameter prediction on the task embedding vector to obtain the model adjustment parameters, including: Performing feature extraction on the task embedding vector using the first fully connected layer to obtain a first parameter vector, performing feature extraction on the task embedding vector using the second fully connected layer to obtain a second parameter vector, and performing outer product calculation on the first parameter vector and the second parameter vector to obtain the synaptic weight modulation matrix; The third fully connected layer is used to perform feature extraction on the task embedding vector to obtain the synaptic bias modulation matrix.

6. The multi-task image classification method according to claim 5, characterized in that: The basic network includes cascaded basic pulse neural modules, each of which includes a cascaded basic convolutional layer, a basic normalization layer, and a basic pulse neuron. The basic pulse neural modules in the basic network are adjusted using the model adjustment parameters to obtain a modulated basic network, including: Performing parameter adjustment on the basic convolutional layer using the synaptic weight modulation matrix and the synaptic bias modulation matrix to obtain an adjusted convolutional layer; Using the synaptic weight modulation matrix to adjust the parameters of the basic spiking neuron to obtain an adjusted spiking neuron; The basic convolutional layer is replaced by the adjusted convolutional layer, and / or the basic spiking neuron is replaced by the adjusted spiking neuron to obtain the modulated basic network.

7. The multi-task image classification method according to claim 6, characterized in that: The step of adjusting parameters of the basic convolutional layer by using the synaptic weight modulation matrix and the synaptic bias modulation matrix to obtain an adjusted convolutional layer includes: Adding the identity matrix to the synaptic weight modulation matrix to obtain a synaptic unit weight matrix; Performing element-by-element multiplication on the synaptic unit weight matrix and the initial synaptic weight matrix of the base convolutional layer to obtain an adjusted synaptic weight matrix; Adding the synaptic bias modulation matrix and the initial synaptic bias matrix of the base convolutional layer to obtain an adjusted synaptic bias matrix; Parameters of the basic convolutional layer are adjusted based on the adjusted synaptic weight matrix and the adjusted synaptic bias matrix to obtain the adjusted convolutional layer.

8. The multi-task image classification method according to claim 6, characterized in that: The step of adjusting the parameters of the basic spiking neuron using the synaptic weight modulation matrix to obtain an adjusted spiking neuron includes: Obtaining the initial membrane potential voltage matrix of the basic spiking neuron; Performing element-by-element multiplication on the synaptic weight modulation matrix and the initial membrane potential voltage matrix to obtain an adjusted membrane potential voltage matrix; The parameters of the basic pulse neuron are adjusted according to the adjusted membrane potential voltage matrix to obtain the adjusted pulse neuron.

9. The multi-task image classification method according to claim 4, characterized in that: The first linear module includes a linear layer, a second batch of normalization layers, and a second spiking neuron; Inputting the task encoding vector into the first linear module to perform vector dimensionality reduction to obtain a task embedding vector, including: Performing linear feature extraction on the task encoding vector using the linear layer to obtain a first embedding vector; performing batch normalization on the first embedding vector input using the second batch normalization layer to obtain a second embedding vector; The second spiking neuron is used to perform spiking feature extraction on the second embedding vector to obtain the task embedding vector.

10. A multi-task image classification device, characterized in that: include: Pulse feature extraction module: used to extract pulse features from the sample image sequence in the support dataset using at least one pulse neural module cascaded in the task encoder to obtain a task encoding vector; A basic parameter adjustment module is configured to generate parameters based on the task encoding vector using a parameter generator to obtain model adjustment parameters, and to adjust parameters of a basic pulse neural module in a basic network using the model adjustment parameters to obtain a modulated basic network, wherein the structure of the basic network is consistent with that of the task encoder; A class prototype vector calculation module is configured to perform feature mapping on the sample image sequence using the modulation base network to obtain a sample embedding vector, and average the sample embedding vectors belonging to the same class in the support data set to obtain a class prototype vector; Classification module: used to obtain the image sequence to be processed in the query data set, and determine the classification result of the image sequence to be processed according to the vector distance between the image sequence to be processed and each class prototype vector.

11. An electronic device, characterized in that: The electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the multi-task image classification method according to any one of claims 1 to 9 when executing the computer program.

12. A storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the multi-task image classification method according to any one of claims 1 to 9 is implemented.

Images