Neural signal generation method and system based on self-supervised learning, and terminal

Abstract

Disclosed in the present application are a neural signal generation method and system based on self-supervised learning, and a terminal. The method comprises: acquiring an original neural signal, and encoding the original neural signal to obtain a first neural signal set, wherein the first neural signal set comprises: first neural signals, first behavioral signals and first time signals; performing signal enhancement on the first neural signals corresponding to all the first time signals, so as to obtain a plurality of second enhanced neural signals; inputting the first neural signal set and all the second enhanced neural signals into an encoder based on a multi-head self-attention mechanism for encoding, so as to obtain a first latent variable; inputting the first latent variable into a neural ordinary differential equation model for training, so as to obtain a trained second latent variable; and acquiring Gaussian noise, combining the second latent variable with the Gaussian noise to obtain a combined variable, and inputting the combined variable into a generative adversarial network model to generate a high-dimensional pseudo-neural signal. Therefore, the robustness of a decoder can be improved.

Description

A neural signal generation method and system based on self-supervised learning and a terminal TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, in particular to a neural signal generation method and system based on self-supervised learning, a terminal and a computer readable storage medium. BACKGROUND

[0002] The limitations of neural electrophysiological recording acquisition result in less available data, and it is highly challenging to map limited neural signals to complex behavioral spaces (behavioral signals).

[0003] In the prior art, the neural signal generation method is to extract neural dynamics features by mapping neural signals to behavioral signals, and further input certain Gaussian noise to a generative adversarial network based on the neural dynamics features to generate pseudo neural signals. In the aspect of neural dynamics feature extraction, the encoding information of high-dimensional neuron group activity is not considered, and the complex changes in high-dimensional neuron group activity cannot be learned.

[0004] SUMMARY

[0005] Therefore, the present application provides a neural signal generation method and system based on self-supervised learning, a terminal and a computer readable storage medium to solve the problem that in the prior art, the encoding information of high-dimensional neuron group activity is not considered when extracting neural dynamics features, and the complex changes in high-dimensional neuron group activity cannot be learned.

[0006] The present application provides a neural signal generation method based on self-supervised learning, which comprises:

[0007] Obtain an original neural signal, encode the original neural signal to obtain a first neural signal set; wherein the first neural signal set comprises a plurality of first neural signals, a plurality of first behavioral signals and a plurality of first time signals, each first time signal is the time information of each corresponding first neural signal, and each first behavioral signal is the behavioral space information of the first neural signal to be decoded corresponding to the first neural signal;

[0008] Signal enhancement is performed on the first neural signals corresponding to all the first time signals to obtain a plurality of second enhanced neural signals;

[0009] The first neural signal set and all the second enhanced neural signals are input into an encoder based on a multi-head self-attention mechanism for encoding and reparameterization to obtain a low-dimensional first latent variable;

[0010] inputting the first latent variable into a neural ordinary differential equation model to obtain a trained second latent variable;

[0011] obtaining Gaussian noise, combining the second latent variable and the Gaussian noise to obtain a combined variable, and inputting the combined variable into a generative adversarial network model to generate a high-dimensional pseudo neural signal.

[0012] Optionally, the first neural signals corresponding to all the first time signals are subjected to signal enhancement to obtain a plurality of second enhanced neural signals, specifically including:

[0013] In the first neural signals corresponding to all the first time signals, the first neural signals of the discharge are converted into the first neural signals of the non-discharge, and the first neural signals of the non-discharge are converted into the first neural signals of the discharge, and all the first neural signals after the conversion are taken as the second enhanced neural signals.

[0014] Optionally, the first neural signal set and all the second enhanced neural signals are inputted into an encoder based on a multi-head self-attention mechanism to be encoded and re-parameterized to obtain a low-dimensional first latent variable, specifically including:

[0015] The first neural signal set and all the second enhanced neural signals are inputted into an encoder based on a multi-head self-attention mechanism to be encoded, and a first encoding vector and a second encoding vector are outputted;

[0016] The first encoding vector and the second encoding vector are re-parameterized to obtain a low-dimensional first latent variable.

[0017] Optionally, the first encoding vector and the second encoding vector are re-parameterized to obtain a first latent variable, specifically including:

[0018] A first sampling value is sampled from a standard Gaussian distribution, the first sampling value is multiplied by the first encoding vector to obtain an intermediate vector;

[0019] The intermediate vector is added to the second encoding vector to obtain the first latent variable.

[0020] Optionally, the first latent variable is inputted into a neural ordinary differential equation model to obtain a trained second latent variable, specifically including:

[0021] The first latent variable is inputted into a neural ordinary differential equation model to obtain an initial second latent variable and a change relationship between the initial second latent variable and time;

[0022] inputting the initial second latent variable into an output layer of a variational autoencoder or an autoencoder for training to obtain the trained second latent variable.

[0023] Optionally, the inputting the initial second latent variable into an output layer of a variational autoencoder or an autoencoder for training to obtain the trained second latent variable specifically comprises:

[0024] inputting the initial second latent variable into an output layer of a variational autoencoder or an autoencoder for training;

[0025] the output layer performing mutual information calculation on the second enhanced neural signal and the original neural signal, and performing neural signal reconstruction on the input first neural signal and the first behavioral signal at each time point until training is completed;

[0026] the trained second latent variable in the output layer is taken as the final trained second latent variable;

[0027] wherein, the optimization objective of the output layer training is minimization of reconstruction loss, minimization of KL divergence and maximization of mutual information, or minimization of reconstruction loss and maximization of mutual information.

[0028] Optionally, the obtaining of the Gaussian noise, the combining of the second latent variable and the Gaussian noise to obtain a combined variable, and the inputting of the combined variable into a generative adversarial network model to generate a high-dimensional pseudo neural signal specifically comprises:

[0029] obtaining a Gaussian noise, combining the second latent variable and the Gaussian noise to obtain a combined variable, and inputting the combined variable into a generator in the generative adversarial network model to generate a pseudo neural signal;

[0030] inputting the original neural signal and the pseudo neural signal into a discriminator in the generative adversarial network model for binary classification learning to determine whether the input original neural signal and the pseudo neural signal are the pseudo neural signal or the original neural signal;

[0031] when the input original neural signal and the pseudo neural signal are the pseudo neural signal, outputting the pseudo neural signal as the high-dimensional pseudo neural signal.

[0032] The application also proposes a neural signal generation system based on self-supervised learning, which comprises:

[0033] An original neural signal encoding module is configured to acquire original neural signals, encode the original neural signals, and obtain a first neural signal set; wherein the first neural signal set includes a plurality of first neural signals, a plurality of first behavior signals, and a plurality of first time signals, each of the first time signals is time information of each of the first neural signals, and each of the first behavior signals is behavior space information of each of the first neural signals to be decoded;

[0034] A first neural signal enhancement module is configured to perform signal enhancement on the first neural signals corresponding to all the first time signals, and obtain a plurality of second enhanced neural signals.

[0035] A neural signal dimension reduction module is configured to input the first neural signal set and all the second enhanced neural signals into an encoder based on a multi-head self-attention mechanism to perform encoding, and obtain low-dimensional first latent variables.

[0036] A second latent variable training module is configured to input the first latent variables into a neural ordinary differential equation model to perform training, and obtain trained second latent variables.

[0037] A high-dimensional pseudo neural signal generation module is configured to acquire Gaussian noise, combine the second latent variables and the Gaussian noise to obtain combined variables, and input the combined variables into a generative adversarial network model to generate high-dimensional pseudo neural signals.

[0038] The application further provides a terminal, which comprises a memory, a processor, and a self-supervised learning-based neural signal generation program stored in the memory and executable on the processor, and the self-supervised learning-based neural signal generation program is used to implement the steps of the self-supervised learning-based neural signal generation method.

[0039] The application further provides a computer readable storage medium, which stores a self-supervised learning-based neural signal generation program, and the self-supervised learning-based neural signal generation program is used to implement the steps of the self-supervised learning-based neural signal generation method when executed by a processor.

[0040] The beneficial effects of this application are: unlike the prior art, this application obtains a first neural signal set by acquiring the original neural signals and encoding them; wherein, the first neural signal set includes: multiple first neural signals, This application employs multiple first behavioral signals and multiple first temporal signals. Each first temporal signal represents the temporal information of a corresponding first neural signal, and each first behavioral signal represents the behavioral spatial information of a corresponding first neural signal to be decoded. This facilitates the learning of complex changing characteristics in the activity of high-dimensional neuronal groups. Secondly, this application enhances the first neural signals corresponding to all first temporal signals to obtain multiple second enhanced neural signals, which facilitates mutual information optimization in subsequent steps. Thirdly, this application encodes a low-dimensional first latent variable by inputting the set of first neural signals and all second enhanced neural signals into an encoder based on a multi-head self-attention mechanism. Furthermore, this application trains a second latent variable by inputting the first latent variable into a neural frequent differential equation model. Additionally, this application obtains a combined variable by acquiring Gaussian noise and combining the second latent variable with Gaussian noise. This combined variable is then input into a generative adversarial network model to generate a high-dimensional pseudo-neural signal, which can approximate the real neural signal as closely as possible, providing more data input for downstream decoding tasks and improving the robustness of the decoder.

[0041] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this application. Attached Figure Description

[0042] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0043] Figure 1 is a flowchart of a preferred embodiment of the neural signal generation method based on self-supervised learning of this application;

[0044] Figure 2 is a learning model diagram of the neurodynamic features in the neural signal generation method based on self-supervised learning in this application;

[0045] Figure 3 is a diagram of the generative adversarial network model for neural signal generation in the self-supervised learning-based neural signal generation method of this application;

[0046] Figure 4 is a schematic diagram of a preferred embodiment of the neural signal generation system based on self-supervised learning of this application;

[0047] Figure 5 is a schematic diagram of the operating environment of a preferred embodiment of the terminal of this application. DETAILED DESCRIPTION

[0048] In order for those skilled in the art to better understand the technical solutions of the present application, the self-supervised learning-based neural signal generation method, system, terminal and computer readable storage medium provided by the present application are further described in detail below in combination with the drawings and specific embodiments. It can be understood that the described embodiments are only part of the embodiments of the present application, not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0049] The terms "first", "second", and the like in the present application are used to distinguish different objects, not to describe a specific order. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but can optionally include steps or units not listed, or can optionally include other steps or units inherent to the process, method, product or device.

[0050] The present application provides a self-supervised learning-based neural signal generation method, system, terminal and computer readable storage medium to solve the problem that in the prior art, when neural dynamics features are extracted, the encoding information of high-dimensional neuron group activity is not considered, and complex change features in high-dimensional neuron group activity cannot be learned.

[0051] Please refer to FIG. 1 to FIG. 3, FIG. 1 is a flowchart of a preferred embodiment of the self-supervised learning-based neural signal generation method of the present application; FIG. 2 is a learning model diagram of neural dynamics features in the self-supervised learning-based neural signal generation method of the present application; and FIG. 3 is a generative adversarial network model diagram for neural signal generation in the self-supervised learning-based neural signal generation method of the present application.

[0052] The present application provides a self-supervised learning-based neural signal generation method, system, terminal and computer readable storage medium to solve the problem that in the prior art, when neural dynamics features are extracted, the encoding information of high-dimensional neuron group activity is not considered, and complex change features in high-dimensional neuron group activity cannot be learned.

[0053] Step S100: obtaining an original neural signal, encoding the original neural signal to obtain a first neural signal set; wherein the first neural signal set includes a plurality of first neural signals, a plurality of first behavior signals and a plurality of first time signals, each first time signal is time information of each corresponding first neural signal, and each first behavior signal is behavior space information of the first neural signal to be decoded corresponding to the first neural signal.

[0054] Specifically, the original neural signals are acquired, and each original neural signal at a time point is coded to obtain a first neural signal set; the first neural signal set includes a plurality of first neural signals, a plurality of first behavior signals and a plurality of first time signals, the first neural signal refers to a neuron fired at a current time point, the first time signal is time information of each first neural signal, and the first behavior signal is spatial information of the first neural signal to be decoded at the current time point, wherein the spatial information includes direction information, two-dimensional coordinate motion trajectory information and the like, so as to learn complex change characteristics in high-dimensional neuron group activity.

[0055] Step S200: Signal enhancement is performed on the first neural signals corresponding to all the first time signals to obtain a plurality of second enhanced neural signals.

[0056] Specifically, an enhanced signal is designed for the neural signal at each time point, that is, signal enhancement is performed on the first neural signals corresponding to all the first time signals, while the first behavior signals and the first time signals remain unchanged, to obtain a plurality of second enhanced neural signals, so as to facilitate mutual information optimization in subsequent steps.

[0057] The step S200 of performing signal enhancement on the first neural signals corresponding to all the first time signals to obtain a plurality of second enhanced neural signals specifically includes:

[0058] In all the first neural signals corresponding to the first time signals, the first neural signals that fire are converted into the first neural signals that do not fire, and the first neural signals that do not fire are converted into the first neural signals that fire, and all the converted first neural signals are taken as the second enhanced neural signals.

[0059] Specifically, there are only two cases of firing and not firing for neurons, that is, in all the first neural signals corresponding to the first time signals, the first neural signals that fire are converted into the first neural signals that do not fire, and the first neural signals that do not fire are converted into the first neural signals that fire, while the corresponding first behavior signals and the first time signals remain unchanged, and all the converted first neural signals are the second enhanced neural signals.

[0060] When the neural signal is a local field potential signal, the first neural signal enhancement action in step S200 can be directly cancelled, and the other dynamic characteristic learning is the same as that of the neural pulse signal. When step S200 is cancelled, the calculation of mutual information in the optimization target of step S400 is also cancelled.

[0061] Wherein, when the neural signal is a local field potential signal, the first neural signal enhancement in step S200 can choose other enhancement methods, which can be mutation enhancement, splicing enhancement, stretching enhancement, etc. If the first neural signal enhancement action step S200 is adopted, the mutual information needs to be optimized, and if the signal enhancement action step S200 is cancelled, the mutual information does not need to be optimized.

[0062] Step S300: input the first neural signal set and all the second enhanced neural signals into the encoder based on the multi-head self-attention mechanism for encoding and reparameterization, to obtain a low-dimensional first latent variable.

[0063] Specifically, the first neural signal and the first behavior signal at each time in step S100 and all the second enhanced neural signals are input into the encoder based on the multi-head self-attention mechanism for encoding and reparameterization, so as to obtain a low-dimensional first latent variable, and the encoder based on the multi-head self-attention mechanism is used as the encoding head of the variational autoencoder.

[0064] Wherein, the step S300: input the first neural signal set and all the second enhanced neural signals into the encoder based on the multi-head self-attention mechanism for encoding and reparameterization, to obtain a low-dimensional first latent variable, specifically comprising:

[0065] The first neural signal set and all the second enhanced neural signals are input into the encoder based on the multi-head self-attention mechanism for encoding, to output a first encoding vector and a second encoding vector;

[0066] The first encoding vector and the second encoding vector are reparameterized to obtain a low-dimensional first latent variable.

[0067] Specifically, the first neural signal and the first behavior signal at each time in step S100 and all the second enhanced neural signals in step S200 are input into the encoder based on the multi-head self-attention mechanism for encoding, as shown in FIG. 2, the first encoding vector μ and the second encoding vector σ are output by the encoder based on the multi-head self-attention mechanism, the first encoding vector μ and the second encoding vector σ are reparameterized to obtain the first latent variable.

[0068] Wherein, the step: reparameterizing the first encoding vector and the second encoding vector to obtain a low-dimensional first latent variable, specifically comprising:

[0069] A first sampling value is sampled from a standard Gaussian distribution, the first sampling value is multiplied by the first encoding vector to obtain an intermediate vector;

[0070] The intermediate vector is added to the second encoding vector to obtain the first latent variable.

[0071] Specifically, reparameterization refers to sampling from a standard Gaussian distribution (mean 0, variance 1) to obtain a first sampling value, multiplying the first sampling value by a first encoding vector μ to obtain an intermediate vector, and adding the intermediate vector to a second encoding vector σ to obtain a first latent variable.

[0072] When the neural signal is a local field potential signal, the difference is that the first neural signal and the first behavioral signal at each time point in step S100 are input into an encoder based on a multi-head self-attention mechanism for dimension reduction, and the remaining steps are the same as those for neural spike signals.

[0073] Step S400: inputting the first latent variable into a neural ordinary differential equation model for training to obtain a trained second latent variable.

[0074] Specifically, the first latent variable is input into the neural ordinary differential equation model for training to obtain the trained second latent variable, which facilitates combination with Gaussian noise.

[0075] The step S400: inputting the first latent variable into a neural ordinary differential equation model for training to obtain a trained second latent variable, specifically includes:

[0076] The first latent variable is input into a neural ordinary differential equation model to obtain an initial second latent variable and a change relationship between the initial second latent variable and time.

[0077] The initial second latent variable is input into the output layer of the variational autoencoder or the autoencoder for training to obtain the trained second latent variable.

[0078] Specifically, the first latent variable is input into an encoder based on a neural ordinary differential equation model to obtain an initial second latent variable and a change relationship between the initial second latent variable and time. The initial second latent variable is input into the output layer of the variational autoencoder or the autoencoder for training to reconstruct the input and make the distribution of the latent variable close to the Gaussian distribution, and to maximize the mutual information between the enhanced signal and the original signal, to obtain the trained second latent variable.

[0079] The step of inputting the initial second latent variable into the output layer of the variational autoencoder or the autoencoder for training to obtain the trained second latent variable specifically includes:

[0080] The initial second latent variable is input into the output layer of the variational autoencoder or the autoencoder for training.

[0081] The output layer performs mutual information calculation on the second enhanced neural signal and the original neural signal, and performs neural signal reconstruction on the input first neural signal and the first behavioral signal at each time point until the training is completed.

[0082] The trained second latent variable in the output layer is taken as the final trained second latent variable.

[0083] The optimization objective of the output layer training is reconstruction loss minimization, KL divergence minimization, and mutual information maximization, or reconstruction loss minimization and mutual information maximization.

[0084] Specifically, as shown in FIG. 2, the obtained initial second latent variable z is input to the output layer of the variational autoencoder in the neural ordinary differential equation model for training. The neural ordinary differential equation model can solve the change relationship of the initial second latent variable z over time. The output layer reconstructs the first neural signal and the first behavioral signal at each time point of the input, and the output layer performs mutual information calculation on the second enhanced neural signal and the original neural signal, and performs neural signal reconstruction on the first neural signal and the first behavioral signal at each time point of the input until the training is completed. The trained second latent variable in the output layer is taken as the final trained second latent variable. The optimization objective of the output layer training is reconstruction loss minimization, KL divergence minimization, and mutual information maximization.

[0085] The variational autoencoder can also be replaced by an alternative solution of an autoencoder. Specifically, the initial second latent variable is input to the output layer of the autoencoder for training. The output layer performs mutual information calculation on the second enhanced neural signal and the original neural signal, and performs neural signal reconstruction on the first neural signal and the first behavioral signal at each time point of the input until the training is completed. The trained second latent variable in the output layer is taken as the final trained second latent variable. The optimization objective of the output layer training is reconstruction loss minimization and mutual information maximization. Compared with the solution of the variational autoencoder, only one module of KL divergence calculation is less in the optimization objective, that is, the output vector of the encoder based on the multi-head self-attention mechanism is directly set as the first latent variable, and the subsequent training mode is consistent.

[0086] Steps S100-S400 are a training structure of a variational autoencoder. Through training of the present application, the neural dynamics characteristics combined with the behavioral signal and the neural signal can be learned to obtain the second latent variable z.

[0087] Step S500: Obtain Gaussian noise, combine the second latent variable and the Gaussian noise to obtain a combined variable, and input the combined variable into a generative adversarial network model to generate a high-dimensional pseudo neural signal.

[0088] Specifically, the trained second latent variable z and Gaussian noise are combined to obtain a combined variable, and the combined variable is input into the generative adversarial network model to generate a high-dimensional pseudo neural signal, which can be as close as possible to the real neural signal, to provide more data input for the downstream neural decoder and improve the robustness of the decoder.

[0089] The Gaussian noise can also be replaced by noise of other distributions, which can be uniform distribution noise, Poisson distribution noise, etc.

[0090] The step S500 of obtaining Gaussian noise, combining the second latent variable and the Gaussian noise to obtain a combined variable, and inputting the combined variable into the generative adversarial network model to generate a high-dimensional pseudo neural signal specifically includes:

[0091] The Gaussian noise is obtained, the second latent variable and the Gaussian noise are combined to obtain a combined variable, and the combined variable is input into the generator in the generative adversarial network model to generate a pseudo neural signal.

[0092] The original neural signal and the pseudo neural signal are input into the discriminator in the generative adversarial network model for binary classification learning to determine whether the input original neural signal and the pseudo neural signal are the pseudo neural signal or the original neural signal.

[0093] When the input original neural signal and the pseudo neural signal are the pseudo neural signal, the pseudo neural signal is output as the high-dimensional pseudo neural signal.

[0094] Specifically, as shown in FIG. 3, Gaussian noise is obtained, the neural dynamics feature (i.e., the trained second latent variable z) and the Gaussian noise are combined to obtain a combined variable, and the generative adversarial network model includes a generator and a discriminator. The present scheme adopts a bidirectional long short-term memory network as the generator and a simple fully connected network as the discriminator. The combined variable is input into the generator in the generative adversarial network model to generate a pseudo neural signal. The original neural signal and the pseudo neural signal are input into the discriminator in the generative adversarial network model for binary classification learning to determine whether the input original neural signal and the pseudo neural signal are the pseudo neural signal or the original neural signal. When the input original neural signal and the pseudo neural signal are the pseudo neural signal, the pseudo neural signal is output as the high-dimensional pseudo neural signal. Through adversarial learning, the pseudo neural signal generated by the generator can be as close as possible to the real neural signal, which can provide more data input for the downstream decoding task.

[0095] In the generative adversarial network, the generator and the discriminator are replaced by other artificial neural networks as alternatives. These other artificial neural networks include general recurrent neural networks, gated recurrent unit recurrent neural networks, fully connected neural networks, and convolutional neural networks, attention mechanism-based neural networks (Transformer), etc. These networks can be used as the generator or the discriminator in the generative adversarial network, and all belong to the protection scope of the present application.

[0096] As shown in FIG. 3, the downstream decoder is a process of mapping the neural signal to the behavioral signal. This process can be implemented by building a transformer model (transformer decoder) or a recurrent neural network model (including general recurrent neural network, long short-term memory recurrent neural network, and gated recurrent unit recurrent neural network).

[0097] Therefore, the present application combines the neural dynamics characteristics of the first neural signal (high-dimensional neuron population activity) and the first behavioral signal, and further adds certain Gaussian noise input to the generative adversarial network based on the learned good neural dynamics characteristics to generate pseudo neural signal data. Not only the encoding characteristics of the neuron population activity are included, but also the neural dynamics characteristics related to the behavioral signal. This method allows to capture the complexity and dynamics of neuron activity, provides a rich information basis for the decoding process, provides more data support for the decoder, and improves the robustness of the decoder.

[0098] In addition, the present application has conducted experiments and tests on a neural signal data set, with the training set accounting for 80% and the test set accounting for 20%. In the test set, the fitting degree of the decoding result and the observed behavioral signal is 0.63. If the pseudo neural signal generated by the generation model in the present application is input into the decoder for training, the fitting degree of the decoding result and the observed behavioral signal is 0.78, and the training efficiency of the decoder is improved. Meanwhile, the decoding performance in different sessions is also tested. The neural signal and the pseudo neural signal collected in the first session on the first day are input into the decoder for training. The decoding performance of the decoder in the sessions on the second day and the third day does not change basically, which shows that the method proposed in the present application has certain robustness.

[0099] Please refer to FIG. 4 to FIG. 5. FIG. 4 is a principle schematic diagram of a preferred embodiment of the neural signal generation system based on self-supervised learning of the present application; and FIG. 5 is a running environment schematic diagram of a preferred embodiment of the terminal of the present application.

[0100] In some embodiments, as shown in FIG. 4, based on the above-mentioned neural signal generation method based on self-supervised learning, the present application also proposes a neural signal generation system based on self-supervised learning, which comprises:

[0101] An original neural signal encoding module 51 is configured to acquire an original neural signal, encode the original neural signal, and obtain a first neural signal set; wherein the first neural signal set comprises a plurality of first neural signals, a plurality of first behavior signals, and a plurality of first time signals, each first time signal is time information of each corresponding first neural signal, and each first behavior signal is behavior space information of each corresponding first neural signal to be decoded;

[0102] A first neural signal enhancement module 52 is configured to perform signal enhancement on the first neural signals corresponding to all the first time signals, and obtain a plurality of second enhanced neural signals;

[0103] A neural signal dimension reduction module 53 is configured to input the first neural signal set and all the second enhanced neural signals into an encoder based on a multi-head self-attention mechanism to perform encoding, and obtain a low-dimensional first latent variable;

[0104] A second latent variable training module 54 is configured to input the first latent variable into a neural ordinary differential equation model to perform training, and obtain a trained second latent variable;

[0105] A high-dimensional pseudo neural signal generation module 55 is configured to acquire Gaussian noise, combine the second latent variable and the Gaussian noise to obtain a combined variable, input the combined variable into a generative adversarial network model, and generate a high-dimensional pseudo neural signal.

[0106] In some embodiments, as shown in FIG. 5, based on the above-mentioned neural signal generation method and system based on self-supervised learning, the present application also correspondingly proposes a terminal, which comprises a memory 20, a processor 10, and a display 30. FIG. 5 only shows part of the components of the terminal, but it should be understood that it is not required to implement all the shown components, and more or fewer components can be alternatively implemented.

[0107] The memory 20 may, in some embodiments, be an internal storage unit of the terminal, such as a hard disk or a memory of the terminal. The memory 20 may, in other embodiments, also be an external storage device of the terminal, such as a plug-in hard disk, a Smart Media Card (SMC), a Secure Digital (SD) card, a Flash Card, etc. Further, the memory 20 may also include both an internal storage unit and an external storage device of the terminal. The memory 20 is used to store application software installed on the terminal and various types of data, such as program codes of the installed terminal, etc. The memory 20 may also be used to temporarily store data that has been output or is to be output.

[0108] In an embodiment, the memory 20 stores a neural signal generation program based on self-supervised learning 40, which can be executed by the processor 10 to implement the neural signal generation method based on self-supervised learning in the present application.

[0109] The processor 10 may, in some embodiments, be a central processing unit (CPU), a microprocessor or other data processing chip, used to run program codes or process data stored in the memory 20, such as to execute the neural signal generation method based on self-supervised learning, etc.

[0110] The display 30 may, in some embodiments, be an LED display, a liquid crystal display, a touch liquid crystal display, an OLED (Organic Light-Emitting Diode) touch, etc. The display 30 is used to display information of the terminal and to display a visualized user interface. The components 10-30 of the terminal communicate with each other through a system bus.

[0111] The present application also proposes a computer readable storage medium storing a neural signal generation program based on self-supervised learning, which, when executed by a processor, implements the steps of the neural signal generation method based on self-supervised learning as described above.

[0112] To sum up, the application obtains the original neural signal, encodes the original neural signal to obtain a first neural signal set; wherein the first neural signal set includes: a plurality of first neural signals, a plurality of first behavior signals and a plurality of first time signals, each first time signal is time information of each corresponding first neural signal, and each first behavior signal is behavior space information of a corresponding first neural signal to be decoded, so as to facilitate learning of complex change characteristics in a high-dimensional neuron group activity; secondly, the application enhances the first neural signals corresponding to all the first time signals to obtain a plurality of second enhanced neural signals, so as to facilitate mutual information optimization in a subsequent step; thirdly, the application encodes the first neural signal set and all the second enhanced neural signals in an encoder based on a multi-head self-attention mechanism to obtain a low-dimensional first latent variable; in addition, the application inputs the first latent variable into a neural ordinary differential equation model to obtain a trained second latent variable; in addition, the application obtains Gaussian noise, combines the second latent variable and the Gaussian noise to obtain a combined variable, and inputs the combined variable into a generative adversarial network model to generate a high-dimensional pseudo neural signal, which can be as close as possible to a real neural signal, so as to provide more data input for a downstream decoding task and improve robustness of a decoder.

[0113] It should be noted that the various optional embodiments introduced in the embodiments of the application can be combined with each other to be implemented, or can be implemented alone, and the embodiments of the application are not limited thereto.

[0114] In the description of the application, it should be understood that the terms "upper", "lower", "left", "right" and the like indicate the orientation or positional relationship based on the orientation or positional relationship shown in the drawings, and are only for the convenience of describing the application and simplifying the description, and do not indicate or imply that the devices or elements referred to must have a particular orientation, and a particular orientation configuration and operation. Therefore, it cannot be understood as a limitation on the application. In addition, "first", "second" are only for the purpose of description, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of indicated technical features. Therefore, the features limited by "first", "second" can explicitly or implicitly include one or more features. In the description of the application, unless otherwise specified, the meaning of "a plurality of" is two or more.

[0115] In the description of the application, it should be noted that unless otherwise explicitly specified and limited, the terms "mounting", "connection", "connecting" and the like should be interpreted broadly, for example, can be fixed connection, can also be detachable connection, or integrally connected; can be mechanical connection, can also be electrical connection; can be directly connected, can also be indirectly connected through an intermediate medium, can be the communication inside two elements. For those skilled in the art, the specific meaning of the above terms in the application can be understood according to the specific circumstances.

[0116] The above embodiments are described with reference to the accompanying drawings, other different forms and embodiments are also possible without departing from the principles of the application, therefore the application should not be construed as being limited to the embodiments presented herein. Rather, these embodiments are provided so that the application will be thorough and complete, and will convey the scope of the application to those skilled in the art. In the drawings, the sizes and relative sizes of components can be exaggerated for clarity. The terms used herein are based on the purpose of describing specific embodiments only, and are not intended to be limiting. The terms "comprising" and / or "including" used in the specification indicate the presence of the described features, integers, components and / or components, but do not exclude the presence or addition of one or more other features, integers, components, components and / or groups thereof. Unless otherwise indicated, when stated, numerical ranges include the upper and lower limits of the range and any sub-range therebetween.

[0117] The above is only part of the embodiments of the application, and does not limit the protection scope of the application, any equivalent device or equivalent process transformation using the content of the specification and drawings, or directly or indirectly applied to other related technical fields, are also included in the patent protection scope of the application.

Claims

1. A neural signal generation method based on self-supervised learning, characterized by, The method comprises the following steps: obtaining original neural signals, encoding the original neural signals to obtain a first neural signal set; wherein the first neural signal set comprises a plurality of first neural signals, a plurality of first behavior signals and a plurality of first time signals, each first time signal is time information of each corresponding first neural signal, and each first behavior signal is behavior space information of each corresponding first neural signal to be decoded; signal enhancement is performed on the first neural signals corresponding to all the first time signals to obtain a plurality of second enhanced neural signals; the first neural signal set and all the second enhanced neural signals are input into an encoder based on a multi-head self-attention mechanism for encoding and reparameterization to obtain a low-dimensional first latent variable; the first latent variable is input into a neural ordinary differential equation model for training to obtain a trained second latent variable; gaussian noise is obtained, the second latent variable and the gaussian noise are combined to obtain a combined variable, and the combined variable is input into a generative adversarial network model to generate a high-dimensional pseudo neural signal.

2. The neural signal generation method based on self-supervised learning according to claim 1, characterized in that, The signal enhancement on the first neural signals corresponding to all the first time signals to obtain a plurality of second enhanced neural signals specifically comprises: in all the first neural signals corresponding to the first time signals, the first neural signals with discharges are converted into the first neural signals without discharges, and the first neural signals without discharges are converted into the first neural signals with discharges, and all the first neural signals after conversion are taken as the second enhanced neural signals.

3. The neural signal generation method based on self-supervised learning according to claim 1, wherein, The first neural signal set and all the second enhanced neural signals are input into an encoder based on a multi-head self-attention mechanism for encoding and reparameterization to obtain a low-dimensional first latent variable, specifically comprising: the first neural signal set and all the second enhanced neural signals are input into an encoder based on a multi-head self-attention mechanism for encoding to output a first encoding vector and a second encoding vector; the first encoding vector and the second encoding vector are reparameterized to obtain a low-dimensional first latent variable.

4. The neural signal generation method based on self-supervised learning according to claim 3, characterized in that, The reparameterization of the first encoding vector and the second encoding vector to obtain a first latent variable specifically comprises: a first sampling value is sampled from a standard gaussian distribution, the first sampling value is multiplied by the first encoding vector to obtain an intermediate vector; the intermediate vector is added to the second encoding vector to obtain the first latent variable.

5. The self-supervised learning based neural signal generation method of claim 1, wherein, The input of the first latent variable into a neural ordinary differential equation model for training to obtain a trained second latent variable specifically comprises: the first latent variable is input into a neural ordinary differential equation model to obtain an initial second latent variable and a change relationship between the initial second latent variable and time; the initial second latent variable is input into an output layer of a variational autoencoder or an autoencoder for training to obtain a trained second latent variable.

6. The neural signal generation method based on self-supervised learning according to claim 5, characterized in that, The initial second latent variable is input to an output layer of a variational autoencoder or an autoencoder for training, to obtain a trained second latent variable, and the training specifically comprises: The initial second latent variable is input to an output layer of a variational autoencoder or an autoencoder for training; The output layer calculates mutual information of the second enhanced neural signal and the original neural signal, and reconstructs the first neural signal and the first behavior signal at each time point input until the training is completed; The trained second latent variable in the output layer is taken as a final trained second latent variable; The optimization objective of the output layer training is minimization of reconstruction loss, minimization of KL divergence, and maximization of mutual information, or minimization of reconstruction loss and maximization of mutual information.

7. The self-supervised learning based neural signal generation method of claim 1, wherein, The second latent variable and the Gaussian noise are combined to obtain a combined variable, and the combined variable is input into a generative adversarial network model to generate a high-dimensional pseudo neural signal, and the obtaining of the Gaussian noise specifically comprises: The second latent variable and the Gaussian noise are combined to obtain a combined variable, and the combined variable is input into a generator in the generative adversarial network model to generate a pseudo neural signal; The original neural signal and the pseudo neural signal are input into a discriminator in the generative adversarial network model for binary classification learning, to determine whether the input original neural signal and the pseudo neural signal are the pseudo neural signal or the original neural signal; When the input original neural signal and the pseudo neural signal are the pseudo neural signal, the pseudo neural signal is output as the high-dimensional pseudo neural signal.

8. A neural signal generation system based on self-supervised learning, characterized by, The neural signal generation system based on self-supervised learning comprises: An original neural signal encoding module is configured to obtain an original neural signal, encode the original neural signal, and obtain a first neural signal set; the first neural signal set comprises a plurality of first neural signals, a plurality of first behavior signals, and a plurality of first time signals; each first time signal is time information of a corresponding first neural signal; and each first behavior signal is behavior space information of a corresponding first neural signal to be decoded. A first neural signal enhancement module is configured to enhance the first neural signals corresponding to all the first time signals, and obtain a plurality of second enhanced neural signals. A neural signal dimension reduction module is configured to input the first neural signal set and all the second enhanced neural signals into an encoder based on a multi-head self-attention mechanism for encoding, to obtain a low-dimensional first latent variable. A second latent variable training module is configured to input the first latent variable into a neural ordinary differential equation model for training, to obtain a trained second latent variable. A high-dimensional pseudo neural signal generation module is configured to obtain a Gaussian noise, combine the second latent variable and the Gaussian noise to obtain a combined variable, and input the combined variable into a generative adversarial network model to generate a high-dimensional pseudo neural signal.

9. A terminal, characterized by comprising: The terminal comprises a memory, a processor, and a self-supervised learning based neural signal generation program stored on the memory and executable on the processor, the self-supervised learning based neural signal generation program, when executed by the processor, implements the steps of the self-supervised learning based neural signal generation method according to any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer readable storage medium stores a self-supervised learning based neural signal generation program, the self-supervised learning based neural signal generation program, when executed by the processor, implements the steps of the self-supervised learning based neural signal generation method according to any one of claims 1-7.

Images