A method, device, storage medium and terminal for visual decoding of biological neural signals

Abstract

The present invention discloses a method, device, storage medium and terminal for visual decoding of biological neural signals, the method comprising: obtaining a data array of a two-photon calcium imaging signal; inputting the data array of the two-photon calcium imaging signal into a pre-trained V1 visual cortex visual information neural decoding model; wherein the V1 visual cortex visual information neural decoding model is composed of a neural signal conversion module and an automatic encoding module; the automatic encoding module includes a skip-layer structure, and the skip-layer structure is used to fuse the features extracted by the entire neural network at different layers between neural network layers of different depths; and outputs a reconstructed image corresponding to the two-photon calcium imaging signal. Since the present application performs signal data processing through a neural signal conversion module and an automatic encoding module, and fuses the features extracted by the entire neural network at different layers through the skip-layer structure in the automatic encoding module, the accuracy of the reconstructed image is improved.

Description

Technical Field

[0001] The present invention relates to the field of machine vision technology, and in particular to a method, device, storage medium and terminal for visual decoding of biological neural signals. Background Art

[0002] In daily life, a large amount of sensory information is transmitted to the brain and then processed by neurons to produce a series of coping behaviors. Among these different types of sensory sources of external information, vision can be said to be the main contributor to the interaction between the external environment and the brain. Understanding how the brain detects, interprets and responds to visual information representing the external environment is a major problem in neuroscience.

[0003] Decoding specific visual features, such as faces, from the responses of neurons in the visual system of monkeys or humans has become a hot topic for researchers in the field of brain-like vision and computational neuroscience in recent years. However, existing technologies cannot parse a wider range of visual features, so how to obtain high linearity and accuracy is an urgent problem to be solved. Summary of the invention

[0004] The embodiments of the present application provide a method, device, storage medium and terminal for visual decoding of biological neural signals. In order to have a basic understanding of some aspects of the disclosed embodiments, a simple summary is given below. This summary is not a general review, nor is it intended to identify key / important components or describe the scope of protection of these embodiments. Its only purpose is to present some concepts in a simple form as a preface to the detailed description that follows.

[0005] In a first aspect, an embodiment of the present application provides a method for visual decoding of a biological neural signal, the method comprising:

[0006] Acquire data arrays of two-photon calcium imaging signals;

[0007] Inputting the data array of the two-photon calcium imaging signal into a pre-trained neural decoding model of visual information of the V1 visual cortex; wherein the neural decoding model of visual information of the V1 visual cortex is composed of a neural signal conversion module and an automatic encoding module; the automatic encoding module includes a skip-layer structure, and the skip-layer structure is used to fuse the features extracted at different layers of the entire neural network between neural network layers of different depths;

[0008] Output the reconstructed image corresponding to the two-photon calcium imaging signal.

[0009] Optionally, the two-photon calcium imaging signal is fed into a pre-trained neural decoding model of visual information in V1 visual cortex, including:

[0010] The neural signal conversion module converts the data array of the input two-photon calcium imaging signal into an intermediate calculation amount of the same size as the reconstructed target image;

[0011] The intermediate calculations are input into the automatic encoding module to generate a visual stimulus reconstructed image.

[0012] Optionally, the automatic encoding module includes a first network, a skip layer structure and a second network;

[0013] The intermediate calculations are input into the automatic encoding module to generate a visual stimulus reconstructed image, including:

[0014] The first network performs a convolution operation on the vector of the intermediate computational amount and extracts features from the convolution result by downsampling;

[0015] The skip-layer structure fuses the extracted features to obtain fused features;

[0016] The second network upsamples and convolves the fused features to restore a reconstructed image of the same size as the visual stimulus.

[0017] Optionally, generate a pre-trained neural decoding model of visual information in V1 visual cortex by following the steps below, including:

[0018] Create a neural decoding model of visual information in V1 visual cortex;

[0019] Construct model samples and divide the model samples according to preset ratios to obtain training sets and test sets;

[0020] The training set is input into the V1 visual cortex visual information neural decoding model, and the error back propagation algorithm is used to perform end-to-end training and output the loss value;

[0021] Generate a pre-trained neural decoding model of visual information in V1 visual cortex according to the loss value.

[0022] Optionally, build a sample model, including:

[0023] Obtain multiple basic visual stimulus pictures;

[0024] Each basic visual stimulus image is rotated and translated at different angles to obtain a set of artificial scene visual stimulus images;

[0025] Randomly intercepting natural scene visual stimulation images multiple times from the documentary images to obtain a natural scene visual stimulation image set;

[0026] Model samples are generated according to a set of artificial scene visual stimulation images and a set of natural scene visual stimulation images.

[0027] Optionally, generating a model sample according to the artificial scene visual stimulation image set and the natural scene visual stimulation image set includes:

[0028] Performing visual stimulation on the organism according to a set of artificial scene visual stimulation images, and recording the response of neurons in the V1 visual cortex of the organism to obtain a first quantity of calcium imaging signals;

[0029] Performing visual stimulation on the organism according to a set of natural scene visual stimulation images, and recording the response of neurons in the V1 visual cortex of the organism to obtain a second amount of calcium imaging signals;

[0030] determining the first number of calcium imaging signals and the second number of calcium imaging signals as two-photon calcium imaging signals;

[0031] The two-photon calcium imaging signals were mapped and associated with the corresponding visual stimulus images to generate model samples.

[0032] Optionally, visually stimulating an organism according to a natural scene visual stimulation image set, and recording a response of neurons in the V1 visual cortex of the organism to obtain a second number of calcium imaging signals, comprises:

[0033] Displaying a blank display screen to the organism until a preset time;

[0034] The first neural images in living organisms were acquired through electron microscopy using two-photon brightness capture capabilities with single-neuron recording precision;

[0035] The first natural scene visual stimulation image in the natural scene visual stimulation image set is displayed to the organism repeatedly until a preset time;

[0036] Acquire second neural images of living organisms through electron microscopy with two-photon brightness capture capability with single-neuron recording precision;

[0037] determining a difference image based on the first neuroimage and the second neuroimage;

[0038] The ratio of the change in brightness of the calcium indicator on activated neurons was calculated from the difference image;

[0039] The ratio of the brightness change of the calcium indicator on the activated neurons was determined as the calcium imaging signal;

[0040] After the presentation of each natural scene visual stimulation image in the natural scene visual stimulation image set is completed, a second number of calcium imaging signals is generated.

[0041] In a second aspect, an embodiment of the present application provides a visual decoding device for a biological neural signal, the device comprising:

[0042] A signal data array acquisition module, used for acquiring a data array of a two-photon calcium imaging signal;

[0043] A signal data array input module is used to input the data array of the two-photon calcium imaging signal into a pre-trained V1 visual cortex visual information neural decoding model; wherein the V1 visual cortex visual information neural decoding model is composed of a neural signal conversion module and an automatic encoding module; the automatic encoding module includes a skip-layer structure, and the skip-layer structure is used to fuse the features extracted from different layers of the entire neural network between neural network layers of different depths;

[0044] The reconstructed image output module is used to output the reconstructed image corresponding to the two-photon calcium imaging signal.

[0045] In a third aspect, an embodiment of the present application provides a computer storage medium, which stores a plurality of instructions, and the instructions are suitable for being loaded by a processor and executing the above-mentioned method steps.

[0046] In a fourth aspect, an embodiment of the present application provides a terminal, which may include: a processor and a memory; wherein the memory stores a computer program, and the computer program is suitable for being loaded by the processor and executing the above-mentioned method steps.

[0047] The technical solution provided by the embodiments of the present application may have the following beneficial effects:

[0048] In an embodiment of the present application, a visual decoding device for biological neural signals first obtains a data array of a two-photon calcium imaging signal, and then inputs the data array of the two-photon calcium imaging signal into a pre-trained V1 visual cortex visual information neural decoding model; wherein the V1 visual cortex visual information neural decoding model is composed of a neural signal conversion module and an automatic encoding module; the automatic encoding module includes a skip-layer structure, and the skip-layer structure is used to fuse the features extracted by the entire neural network at different layers between neural network layers of different depths, and finally outputs a reconstructed image corresponding to the two-photon calcium imaging signal. Since the present application performs signal data processing through a neural signal conversion module and an automatic encoding module, and fuses the features extracted by the entire neural network at different layers through the skip-layer structure in the automatic encoding module, the accuracy of the reconstructed image is improved.

[0049] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0051] Figure 1It is a flowchart of a visual decoding method of a biological neural signal provided in an embodiment of the present application;

[0052] Figure 2 It is a schematic diagram of the network structure of a pre-trained V1 visual cortex visual information neural decoding model provided in an embodiment of the present application;

[0053] Figure 3 It is a schematic diagram of reconstruction effect and indicator information in a decoding model provided in an embodiment of the present application;

[0054] Figure 4 It is a flowchart of a method for training a neural decoding model of visual information of V1 visual cortex provided in an embodiment of the present application;

[0055] Figure 5 It is a structural schematic diagram of a visual decoding device for biological neural signals provided in an embodiment of the present application;

[0056] Figure 6 It is a structural diagram of a terminal provided in an embodiment of the present application. DETAILED DESCRIPTION

[0057] The following description and the drawings sufficiently illustrate specific embodiments of the invention to enable those skilled in the art to practice them.

[0058] It should be clear that the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0059] When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present invention. Instead, they are only examples of devices and methods consistent with some aspects of the present invention as detailed in the attached claims.

[0060] In the description of the present invention, it should be understood that the terms "first", "second", etc. are only used for descriptive purposes and cannot be understood as indicating or implying relative importance. For those of ordinary skill in the art, the specific meanings of the above terms in the present invention can be understood according to specific circumstances. In addition, in the description of the present invention, unless otherwise specified, "plurality" refers to two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the previously associated objects are in an "or" relationship.

[0061] The present application provides a method, device, storage medium and terminal for visual decoding of biological neural signals to solve the problems existing in the above-mentioned related technical problems. In the technical solution provided by the present application, since the present application processes signal data through a neural signal conversion module and an automatic encoding module, and fuses the features extracted from different layers of the entire neural network through a skip-layer structure in the automatic encoding module, the accuracy of the reconstructed image is improved, and an exemplary embodiment is used for detailed description below.

[0062] The following will be combined with the attached Figure 1 -Attached Figure 4 , a method for visual decoding of biological neural signals provided in an embodiment of the present application is introduced in detail. The method can be implemented by a computer program and can be run on a visual decoding device for biological neural signals based on a von Neumann system. The computer program can be integrated into an application or run as an independent tool application.

[0063] See also Figure 1 , which is a flow chart of a method for visual decoding of biological neural signals provided in an embodiment of the present application. Figure 1 As shown, the method of the embodiment of the present application may include the following steps:

[0064] S101, acquiring a data array of two-photon calcium imaging signals;

[0065] The data array of the two-photon calcium imaging signal is a neural signal obtained in real time, or it can be a neural signal obtained from a test set.

[0066] In a possible implementation, when performing visual decoding of biological neural signals, it is first necessary to obtain real-time neural signals or neural signals in a test set.

[0067] S102, inputting the data array of the two-photon calcium imaging signal into a pre-trained neural decoding model of visual information in the V1 visual cortex;

[0068] The V1 visual cortex visual information neural decoding model is composed of a neural signal conversion module and an automatic encoding module; the automatic encoding module includes a skip-layer structure, which is used to fuse the features extracted from different layers of the entire neural network between neural network layers of different depths;

[0069] In an embodiment of the present application, when generating a pre-trained V1 visual cortex visual information neural decoding model, the V1 visual cortex visual information neural decoding model is first created, and then a model sample is constructed, and the model sample is divided according to a preset ratio to obtain a training set and a test set, and then the training set is input into the V1 visual cortex visual information neural decoding model, and an error back propagation algorithm is used to perform end-to-end training and then output a loss value, and finally a pre-trained V1 visual cortex visual information neural decoding model is generated according to the loss value.

[0070] In one possible implementation, the neural signal conversion module converts the data array of the input two-photon calcium imaging signal into an intermediate calculation quantity of the same size as the reconstructed target image, and then inputs the intermediate calculation quantity into the automatic encoding module to generate a visual stimulus reconstructed image.

[0071] Specifically, the automatic encoding module includes a first network, a skip-layer structure and a second network. When the intermediate calculation amount is input into the automatic encoding module to generate a visual stimulus reconstructed image, the first network performs a convolution operation on the vector of the intermediate calculation amount and extracts features from the convolution result by downsampling. The skip-layer structure performs feature fusion on the extracted features to obtain fused features. The second network upsamples and convolves the fused features to restore a reconstructed image of the same size as the visual stimulus.

[0072] It should be noted that the visual stimulus image was reconstructed from the two-photon calcium imaging neural signals recorded from the 2nd / 3rd layer V1 neurons of awake macaques. Traditional convolutional neural network utilization mostly tends to be decision-oriented tasks, the purpose of which is to extract key knowledge. In contrast, visual stimulus decoding reconstruction is the process of restoring the original information, which requires supplementing the features extracted from the input. The automatic encoding module (U-Net structure) has been widely used in medical image segmentation, providing high-precision pixel-level segmentation. Inspired by this, we added skip-layer connections to the proposed neural decoding model of visual information in the V1 visual cortex, allowing the deeper layers behind to access more complete information in the shallower layers in front.

[0073] S103, outputting a reconstructed image corresponding to the two-photon calcium imaging signal.

[0074] In one possible implementation, after being processed by a pre-trained V1 visual cortex visual information neural decoding model, a reconstructed image corresponding to a two-photon calcium imaging signal can be output.

[0075] For example Figure 2 As shown in the figure, the pre-trained V1 visual cortex visual information neural decoding model can be mainly seen as two parts: a neural signal conversion module (two-photon calcium imaging signal converter) and a reconstruction part based on an automatic encoding module (U-Net automatic encoder).

[0076] For the signal converter part, the size of the model input is set to be equal to the number of two-photon calcium imaging signal neurons in the corresponding artificially designed visual stimulation physiological data set or the natural scene visual stimulation physiological data test set. The output size of the signal converter is set to be consistent with the size of the target visual stimulus reconstructed image to facilitate the reconstruction of visual information in the subsequent autoencoder part. For the visual reconstruction part based on the U-Net autoencoder, this information processing flow can be divided into two stages.

[0077] In the first stage, convolution and downsampling operations are used to process the input from the signal converter in the previous stage, reducing the size of the input to the same size as the target reconstructed visual stimulus image. The other four network layers in the second stage are roughly the "reverse arrangement" of the four network layers in the first stage. The function of the second stage is to restore the downsampled image to the size of the target reconstructed visual stimulus image by upsampling.

[0078] It should be noted that in the network structure of this automatic encoder part, a "skip connection" structure similar to that in U-Net has been added. The function of the skip connection structure is to fuse the features extracted from different layers of the entire neural network between neural network layers of different depths. Specifically, the low-level features in the shallower network are fused with the high-level features in the deeper network in the entire neural decoding and reconstruction process. This process means that both the deep and shallow features in the visual stimulation images presented to awake monkeys can be better captured. Thanks to the design of such a visual information neural decoding network structure, more visual details and precise locations can be better reconstructed with only a small-scale visual stimulation image data set, for example. Figure 3 As shown in the figure. Based on the two-photon calcium imaging signal neural data collected from the V1 visual cortex of awake monkeys and the corresponding visual stimulation data set, we combined the data structure characteristics of the calcium imaging neural signal data and the image space complexity of the visual stimulation image. In the neural decoding model of calcium imaging signals of neurons in the entire V1 visual cortex, three skip-layer structures are used in different layers of the neural network to perform feature fusion.

[0079] Furthermore, the downsampling operation in the first stage is implemented by the MaxPooling2D neural network function in the Keras framework. The upsampling operation in the second stage is implemented by the Upsampling2D neural network function. The RelU function is selected as the activation function in the entire visual information neural decoding model network. All convolutional layers in the proposed V1 visual cortex visual information neural decoding model are followed by a batch normalization layer. In addition, the intermediate layer (that is, the network layer with the smallest convolution kernel), the SpatialDropout2D layer is added after the batch normalization layer in order to improve the overfitting phenomenon of the neural decoding model by randomly discarding activated neurons in space.

[0080] In an embodiment of the present application, a visual decoding device for biological neural signals first obtains a data array of a two-photon calcium imaging signal, and then inputs the data array of the two-photon calcium imaging signal into a pre-trained V1 visual cortex visual information neural decoding model; wherein the V1 visual cortex visual information neural decoding model is composed of a neural signal conversion module and an automatic encoding module; the automatic encoding module includes a skip-layer structure, and the skip-layer structure is used to fuse the features extracted by the entire neural network at different layers between neural network layers of different depths, and finally outputs a reconstructed image corresponding to the two-photon calcium imaging signal. Since the present application performs signal data processing through a neural signal conversion module and an automatic encoding module, and fuses the features extracted by the entire neural network at different layers through the skip-layer structure in the automatic encoding module, the accuracy of the reconstructed image is improved.

[0081] See also Figure 4 , which is a flow chart of a method for visual decoding of biological neural signals provided in an embodiment of the present application. Figure 4 As shown, the method of the embodiment of the present application may include the following steps:

[0082] S201, creating a neural decoding model of visual information in V1 visual cortex;

[0083] S202, constructing a model sample, and dividing the model sample according to a preset ratio to obtain a training set and a test set;

[0084] Usually, in the two-photon imaging calcium signal dataset, there are a total of two types of visual stimulation image datasets, which are used as visual stimulation image sets in visual stimulation experiments. These two visual stimulation image sets are artificially designed visual stimulation images (artificial pattern, AP) and natural scene visual stimulation images (natural scene, NS). In the two-photon calcium imaging physiological experiment, a total of 9,500 artificially designed visual stimulation pictures are presented as visual stimuli. These artificially designed visual stimulation pictures are actually generated by rotating and translating 138 basic visual stimulation pictures at different angles. These operations are to ensure that the visual stimulation can be fully perceived by the "receptive field" of V1 cortical neurons during the display period. There are a total of 2,250 visual stimulation pictures of natural scenes, which are randomly intercepted and generated from documentary images.

[0085] In an embodiment of the present application, a plurality of basic visual stimulation pictures are first obtained, and each basic visual stimulation picture is then rotated and translated at different angles to obtain a set of artificial scene visual stimulation images. Secondly, natural scene visual stimulation images are randomly captured multiple times from documentary images to obtain a set of natural scene visual stimulation images. Finally, a model sample is generated based on the set of artificial scene visual stimulation images and the set of natural scene visual stimulation images.

[0086] Specifically, when generating model samples based on a set of artificial scene visual stimulation images and a set of natural scene visual stimulation images, first, visual stimulation is performed on the organism according to the set of artificial scene visual stimulation images, and the response of the V1 visual cortex neurons of the organism is recorded to obtain a first number of calcium imaging signals. Then, visual stimulation is performed on the organism according to the set of natural scene visual stimulation images, and the response of the V1 visual cortex neurons of the organism is recorded to obtain a second number of calcium imaging signals. Then, the first number of calcium imaging signals and the second number of calcium imaging signals are determined as two-photon calcium imaging signals. Finally, the two-photon calcium imaging signals are mapped and associated with the corresponding visual stimulation images to generate model samples.

[0087] Specifically, when visual stimulation is performed on an organism according to the natural scene visual stimulation image set, and the response of the V1 visual cortex neurons of the organism is recorded to obtain a second number of calcium imaging signals, a blank display screen is first displayed to the organism until a preset time, and then a first neural image of the organism is collected through an electron microscope with a two-photon brightness capture capability with single neuron recording accuracy, and then the first natural scene visual stimulation image in the natural scene visual stimulation image set is displayed to the organism for multiple times until a preset time, and then a second neural image of the organism is collected through an electron microscope with a two-photon brightness capture capability with single neuron recording accuracy, and then a differential image is determined based on the first neural image and the second neural image, and finally, the ratio of the brightness change of the calcium indicator on the activated neuron is calculated based on the differential image, and the ratio of the brightness change of the calcium indicator on the activated neuron is determined as the calcium imaging signal, and when the display of each natural scene visual stimulation image in the natural scene visual stimulation image set is completed, a second number of calcium imaging signals are generated.

[0088] In a possible implementation, the two-photon calcium imaging signal data used in this application is collected from the 2 / 3 cortex of awake monkeys, and the neuroimaging acquisition device used is an electron microscope with two-photon brightness capture capability with single neuron recording accuracy. In the two-photon calcium imaging signal acquisition experiment, some strict experimental settings are used to ensure that the visual stimulus image can be accepted by the accurate retinopic position of the subject awake monkey when it is displayed. When the visual stimulus image is displayed, the subject awake monkey will perform a fixed task. Each artificially designed visual image will be repeated five to six times, and the natural scene visual stimulus image will be repeated three times. V1 visual cortical neurons undergo the following process during the display of a visual stimulus, and the neuronal response during the whole process will be recorded. A blank display screen will be displayed for one second, and then the next second will be used to display the visual stimulus. In order to quantify the response of each activated neuron, the ratio of the brightness change of the calcium indicator on the activated neuron (ΔF / F0) is calculated by the differential image of the above two processes (blank screen display process and visual stimulus image display process), expressed as ΔF=F-F0. F0 is the neuronal activity response during the presentation of a blank screen, and ΔF is the change in light activity during the presentation of visual stimulus images in each trial. Finally, in the two-photon imaging calcium signal data acquisition experiment of awake monkeys, a total of 1,142 activated neurons were recorded under artificially designed visual image stimulation; a total of 1,225 activated neurons were collected under natural scene visual stimulus images. The calcium imaging signals recorded from all these neurons, together with the visual stimulus images that caused these responses, were used to train and test the decoding model.

[0089] S203, inputting the training set into the V1 visual cortex visual information neural decoding model, and outputting the loss value after end-to-end training using an error back propagation algorithm;

[0090] Generally, for the two visual stimulation two-photon calcium imaging signal data sets of artificially designed visual stimulation images and natural scene visual stimulation images, the Adam algorithm is used for gradient descent when training the corresponding visual information neural decoding model. The parameter update learning rate of the neural decoding model corresponding to the artificially designed visual stimulation images is set to 0.003, and the network parameter learning rate of the neural decoding model of the natural scene visual stimulation images is set to 0.001. The batch size of the training data is set to 80. The number of training iterations (epoches) of the neural decoding model of the artificially designed visual stimulation images is controlled by the early-stop algorithm. That is, when the error (validation error) of the neural decoding reconstruction model on the validation set no longer decreases (here is the minimum square error MSE), the model parameters of the neural decoding model stop updating. The number of training iterations of the neural decoding model of the natural scene visual stimulation images is controlled according to the reconstruction results of the neural decoding model. This is because the natural scene visual stimulation image data set has only 2250 natural pictures, and the neural decoding model of the natural scene visual stimulation images is relatively difficult to converge.

[0091] It should be noted that the entire V1 visual cortex neuron visual information neural decoding model runs on the Keras deep learning framework, and its backend is the TensorFlow deep learning platform. The calculation process runs on the NVIDIA v100 computing graphics card. The Adam error gradient descent algorithm with a set learning rate is used to train the neural decoding model.

[0092] In one possible implementation, after obtaining the training data, the training set can be input into the V1 visual cortex visual information neural decoding model, and the error back propagation algorithm is used to perform end-to-end training and output the loss value.

[0093] S204, generating a pre-trained V1 visual cortex visual information neural decoding model according to the loss value.

[0094] In a possible implementation, when the loss value reaches a preset threshold, a pre-trained V1 visual cortex visual information neural decoding model is generated. When the loss value does not reach the preset threshold, the model parameters are continuously updated to perform model training.

[0095] In an embodiment of the present application, a visual decoding device for biological neural signals first obtains a data array of a two-photon calcium imaging signal, and then inputs the data array of the two-photon calcium imaging signal into a pre-trained V1 visual cortex visual information neural decoding model; wherein the V1 visual cortex visual information neural decoding model is composed of a neural signal conversion module and an automatic encoding module; the automatic encoding module includes a skip-layer structure, and the skip-layer structure is used to fuse the features extracted by the entire neural network at different layers between neural network layers of different depths, and finally outputs a reconstructed image corresponding to the two-photon calcium imaging signal. Since the present application performs signal data processing through a neural signal conversion module and an automatic encoding module, and fuses the features extracted by the entire neural network at different layers through the skip-layer structure in the automatic encoding module, the accuracy of the reconstructed image is improved.

[0096] The following are embodiments of the device of the present invention, which can be used to implement the method embodiments of the present invention. For details not disclosed in the device embodiments of the present invention, please refer to the method embodiments of the present invention.

[0097] See also Figure 5 , which shows a schematic diagram of the structure of a visual decoding device for biological neural signals provided by an exemplary embodiment of the present invention. The visual decoding device for biological neural signals can be implemented as all or part of a terminal through software, hardware, or a combination of both. The device 1 includes a signal data array acquisition module 10, a signal data array input module 20, and a reconstructed image output module 30.

[0098] A data array acquisition module 10, used to acquire a data array of a two-photon calcium imaging signal;

[0099] A data array input module 20, used to input the data array of the two-photon calcium imaging signal into a pre-trained V1 visual cortex visual information neural decoding model;

[0100] The V1 visual cortex visual information neural decoding model is composed of a neural signal conversion module and an automatic encoding module; the automatic encoding module includes a skip-layer structure, which is used to fuse the features extracted from different layers of the entire neural network between neural network layers of different depths;

[0101] The reconstructed image output module 30 is used to output the reconstructed image corresponding to the two-photon calcium imaging signal.

[0102] It should be noted that the visual decoding device for biological neural signals provided in the above-mentioned embodiments only uses the division of the above-mentioned functional modules as an example when executing the visual decoding method for biological neural signals. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. In addition, the visual decoding device for biological neural signals provided in the above-mentioned embodiments and the visual decoding method embodiment for biological neural signals belong to the same concept, and the implementation process thereof is detailed in the method embodiment, which will not be repeated here.

[0103] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0104] In an embodiment of the present application, a visual decoding device for biological neural signals first obtains a data array of a two-photon calcium imaging signal, and then inputs the data array of the two-photon calcium imaging signal into a pre-trained V1 visual cortex visual information neural decoding model; wherein the V1 visual cortex visual information neural decoding model is composed of a neural signal conversion module and an automatic encoding module; the automatic encoding module includes a skip-layer structure, and the skip-layer structure is used to fuse the features extracted by the entire neural network at different layers between neural network layers of different depths, and finally outputs a reconstructed image corresponding to the two-photon calcium imaging signal. Since the present application performs signal data processing through a neural signal conversion module and an automatic encoding module, and fuses the features extracted by the entire neural network at different layers through the skip-layer structure in the automatic encoding module, the accuracy of the reconstructed image is improved.

[0105] The present invention also provides a computer-readable medium having program instructions stored thereon, which, when executed by a processor, implement the visual decoding method of biological neural signals provided by the above-mentioned various method embodiments. The present invention also provides a computer program product containing instructions, which, when executed on a computer, enables the computer to execute the visual decoding method of biological neural signals of the above-mentioned various method embodiments.

[0106] See also Figure 6 , which is a schematic diagram of the structure of a terminal provided in an embodiment of the present application. Figure 6 As shown, the terminal 1000 may include: at least one processor 1001 , at least one network interface 1004 , a user interface 1003 , a memory 1005 , and at least one communication bus 1002 .

[0107] The communication bus 1002 is used to realize the connection and communication between these components.

[0108] The user interface 1003 may include a display screen (Display) and a camera (Camera), and the optional user interface 1003 may also include a standard wired interface and a wireless interface.

[0109] The network interface 1004 may optionally include a standard wired interface or a wireless interface (such as a WI-FI interface).

[0110] Among them, the processor 1001 may include one or more processing cores. The processor 1001 uses various interfaces and lines to connect various parts within the entire electronic device 1000, and executes various functions and processes data of the electronic device 1000 by running or executing instructions, programs, code sets or instruction sets stored in the memory 1005, and calling data stored in the memory 1005. Optionally, the processor 1001 can be implemented in at least one hardware form of digital signal processing (Digital Signal Processing, DSP), field programmable gate array (Field-Programmable Gate Array, FPGA), and programmable logic array (Programmable Logic Array, PLA). The processor 1001 can integrate one or a combination of a central processing unit (Central Processing Unit, CPU), a graphics processing unit (Graphics Processing Unit, GPU) and a modem. Among them, the CPU mainly processes the operating system, user interface and application programs; the GPU is responsible for rendering and drawing the content to be displayed on the display screen; the modem is used to process wireless communications. It can be understood that the above-mentioned modem may not be integrated into the processor 1001, and it can be implemented separately through a chip.

[0111] Among them, the memory 1005 may include a random access memory (Random Access Memory, RAM) and may also include a read-only memory (Read-Only Memory). Optionally, the memory 1005 includes a non-transitory computer-readable storage medium. The memory 1005 can be used to store instructions, programs, codes, code sets or instruction sets. The memory 1005 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for at least one function (such as a touch function, a sound playback function, an image playback function, etc.), instructions for implementing the above-mentioned various method embodiments, etc.; the data storage area may store data involved in the above-mentioned various method embodiments, etc. The memory 1005 may optionally be at least one storage device located away from the aforementioned processor 1001. As Figure 6 As shown, the memory 1005 as a computer storage medium may include an operating system, a network communication module, a user interface module, and a visual decoding application for biological neural signals.

[0112] exist Figure 6 In the terminal 1000 shown, the user interface 1003 is mainly used to provide an input interface for the user and obtain the data input by the user; and the processor 1001 can be used to call the visual decoding application of the biological neural signal stored in the memory 1005, and specifically perform the following operations:

[0113] Acquire data arrays of two-photon calcium imaging signals;

[0114] The data array of two-photon calcium imaging signals is input into a pre-trained neural decoding model of visual information in the V1 visual cortex;

[0115] The V1 visual cortex visual information neural decoding model is composed of a neural signal conversion module and an automatic encoding module; the automatic encoding module includes a skip-layer structure, which is used to fuse the features extracted from different layers of the entire neural network between neural network layers of different depths;

[0116] Output the reconstructed image corresponding to the two-photon calcium imaging signal.

[0117] In one embodiment, when the processor 1001 inputs the two-photon calcium imaging signal into the pre-trained V1 visual cortex visual information neural decoding model, the processor 1001 specifically performs the following operations:

[0118] The neural signal conversion module converts the data array of the input two-photon calcium imaging signal into an intermediate calculation amount of the same size as the reconstructed target image;

[0119] The intermediate calculations are input into the automatic encoding module to generate a visual stimulus reconstructed image.

[0120] In one embodiment, when the processor 1001 inputs the intermediate calculation amount into the automatic encoding module to generate the visual stimulus reconstructed image, the processor 1001 specifically performs the following operations:

[0121] The first network performs a convolution operation on the vector of the intermediate computational amount and extracts features from the convolution result by downsampling;

[0122] The skip-layer structure fuses the extracted features to obtain fused features;

[0123] The second network upsamples and convolves the fused features to restore a reconstructed image of the same size as the visual stimulus.

[0124] In one embodiment, when the processor 1001 generates a pre-trained V1 visual cortex visual information neural decoding model, the following operations are specifically performed:

[0125] Create a neural decoding model of visual information in V1 visual cortex;

[0126] Construct model samples and divide the model samples according to preset ratios to obtain training sets and test sets;

[0127] The training set is input into the V1 visual cortex visual information neural decoding model, and the error back propagation algorithm is used to perform end-to-end training and output the loss value;

[0128] Generate a pre-trained neural decoding model of visual information in V1 visual cortex according to the loss value.

[0129] In one embodiment, when executing the construction model sample, the processor 1001 specifically performs the following operations:

[0130] Obtain multiple basic visual stimulus pictures;

[0131] Each basic visual stimulus image is rotated and translated at different angles to obtain a set of artificial scene visual stimulus images;

[0132] Randomly intercepting natural scene visual stimulation images multiple times from the documentary images to obtain a natural scene visual stimulation image set;

[0133] Model samples are generated according to a set of artificial scene visual stimulation images and a set of natural scene visual stimulation images.

[0134] In one embodiment, when the processor 1001 generates a model sample according to the artificial scene visual stimulation image set and the natural scene visual stimulation image set, the processor 1001 specifically performs the following operations:

[0135] Performing visual stimulation on the organism according to a set of artificial scene visual stimulation images, and recording the response of neurons in the V1 visual cortex of the organism to obtain a first quantity of calcium imaging signals;

[0136] Performing visual stimulation on the organism according to a set of natural scene visual stimulation images, and recording the response of neurons in the V1 visual cortex of the organism to obtain a second amount of calcium imaging signals;

[0137] determining the first number of calcium imaging signals and the second number of calcium imaging signals as two-photon calcium imaging signals;

[0138] The two-photon calcium imaging signals were mapped and associated with the corresponding visual stimulus images to generate model samples.

[0139] In one embodiment, when the processor 1001 performs visual stimulation on the organism according to the natural scene visual stimulation image set and records the response of the V1 visual cortex neurons of the organism to obtain the second number of calcium imaging signals, the processor 1001 specifically performs the following operations:

[0140] Displaying a blank display screen to the organism until a preset time;

[0141] The first neural images in living organisms were acquired through electron microscopy using two-photon brightness capture capabilities with single-neuron recording precision;

[0142] The first natural scene visual stimulation image in the natural scene visual stimulation image set is displayed to the organism repeatedly until a preset time;

[0143] Acquire second neural images of living organisms through electron microscopy with two-photon brightness capture capability with single-neuron recording precision;

[0144] determining a difference image based on the first neuroimage and the second neuroimage;

[0145] The ratio of the change in brightness of the calcium indicator on activated neurons was calculated from the difference image;

[0146] The ratio of the brightness change of the calcium indicator on the activated neurons was determined as the calcium imaging signal;

[0147] After the presentation of each natural scene visual stimulation image in the natural scene visual stimulation image set is completed, a second number of calcium imaging signals is generated.

[0148] In an embodiment of the present application, a visual decoding device for biological neural signals first obtains a data array of a two-photon calcium imaging signal, and then inputs the data array of the two-photon calcium imaging signal into a pre-trained V1 visual cortex visual information neural decoding model; wherein the V1 visual cortex visual information neural decoding model is composed of a neural signal conversion module and an automatic encoding module; the automatic encoding module includes a skip-layer structure, and the skip-layer structure is used to fuse the features extracted by the entire neural network at different layers between neural network layers of different depths, and finally outputs a reconstructed image corresponding to the two-photon calcium imaging signal. Since the present application performs signal data processing through a neural signal conversion module and an automatic encoding module, and fuses the features extracted by the entire neural network at different layers through the skip-layer structure in the automatic encoding module, the accuracy of the reconstructed image is improved.

[0149] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program, and the program for visual decoding of biological neural signals can be stored in a computer-readable storage medium, and when the program is executed, it can include the processes of the embodiments of the above-mentioned methods. The storage medium can be a disk, an optical disk, a read-only storage memory, or a random access memory, etc.

[0150] The above disclosure is only the preferred embodiment of the present application, which certainly cannot be used to limit the scope of rights of the present application. Therefore, equivalent changes made according to the claims of the present application are still within the scope covered by the present application.

Claims

1. A visual decoding method for biological neural signals, characterized in that: The method comprises: Acquire data arrays of two-photon calcium imaging signals; The data array of the two-photon calcium imaging signal is input into a pre-trained V1 visual cortex visual information neural decoding model; wherein the V1 visual cortex visual information neural decoding model is composed of a neural signal conversion module and an automatic encoding module; the automatic encoding module includes a skip layer structure, and the skip layer structure is used to fuse the features extracted from different layers of the entire neural network between neural network layers of different depths; wherein, The step of inputting the two-photon calcium imaging signal into a pre-trained V1 visual cortex visual information neural decoding model comprises: The neural signal conversion module converts the data array of the input two-photon calcium imaging signal into an intermediate calculation amount of the same size as the reconstructed target image; The intermediate calculation amount is input into the automatic encoding module to generate a visual stimulus reconstructed image; wherein, The automatic encoding module includes a first network, a skip layer structure and a second network; The step of inputting the intermediate calculation amount into the automatic encoding module to generate a visual stimulation reconstructed image comprises: The first network performs a convolution operation on the vector of the intermediate calculation amount, and extracts features from the convolution result by downsampling; The skip-layer structure fuses the extracted features to obtain fused features; The second network upsamples and convolves the fused features to restore a reconstructed image of the same size as the visual stimulus; Output a reconstructed image corresponding to the two-photon calcium imaging signal.

2. The method according to claim 1, characterized in that Follow these steps to generate a pre-trained neural decoding model of visual information in V1 visual cortex, including: Create a neural decoding model of visual information in V1 visual cortex; Constructing a model sample, and dividing the model sample according to a preset ratio to obtain a training set and a test set; Inputting the training set into the V1 visual cortex visual information neural decoding model, and outputting a loss value after end-to-end training using an error back propagation algorithm; Generate a pre-trained neural decoding model of visual information in V1 visual cortex according to the loss value.

3. The method according to claim 2, characterized in that The constructing model sample comprises: Obtain multiple basic visual stimulus pictures; Each basic visual stimulus image is rotated and translated at different angles to obtain a set of artificial scene visual stimulus images; Randomly intercepting natural scene visual stimulation images multiple times from the documentary images to obtain a natural scene visual stimulation image set; Model samples are generated according to the artificial scene visual stimulation image set and the natural scene visual stimulation image set.

4. The method according to claim 3, characterized in that The generating of the model sample according to the artificial scene visual stimulation image set and the natural scene visual stimulation image set comprises: Performing visual stimulation on an organism according to the artificial scene visual stimulation image set, and recording the response of neurons in the V1 visual cortex of the organism to obtain a first number of calcium imaging signals; Performing visual stimulation on an organism according to the natural scene visual stimulation image set, and recording the response of neurons in the V1 visual cortex of the organism to obtain a second number of calcium imaging signals; determining the first number of calcium imaging signals and the second number of calcium imaging signals as two-photon calcium imaging signals; The two-photon calcium imaging signal is associated with the corresponding visual stimulation image mapping to generate a model sample.

5. The method according to claim 4, characterized in that The step of visually stimulating the organism according to the natural scene visual stimulation image set and recording the response of neurons in the V1 visual cortex of the organism to obtain a second number of calcium imaging signals includes: Displaying a blank display screen to the organism until a preset time; The first neural images in living organisms were acquired through electron microscopy using two-photon brightness capture capabilities with single-neuron recording precision; Displaying the first natural scene visual stimulation image in the natural scene visual stimulation image set to the organism multiple times until a preset time; Acquire second neural images of living organisms through electron microscopy with two-photon brightness capture capability with single-neuron recording precision; determining a difference image based on the first neural image and the second neural image; Calculating the ratio of the brightness change of the calcium indicator on the activated neurons according to the difference image; Determining the ratio of the brightness change of the calcium indicator on the activated neuron as a calcium imaging signal; After the display of each natural scene visual stimulation image in the natural scene visual stimulation image set is completed, a second number of calcium imaging signals are generated.

6. A visual decoding device for biological neural signals, characterized in that: The device comprises: A data array acquisition module, used for acquiring a data array of a two-photon calcium imaging signal; A data array input module is used to input the data array of the two-photon calcium imaging signal into a pre-trained V1 visual cortex visual information neural decoding model; wherein the V1 visual cortex visual information neural decoding model is composed of a neural signal conversion module and an automatic encoding module; the automatic encoding module includes a skip layer structure, and the skip layer structure is used to fuse the features extracted from different layers of the entire neural network between neural network layers of different depths; wherein, The step of inputting the two-photon calcium imaging signal into a pre-trained V1 visual cortex visual information neural decoding model comprises: The neural signal conversion module converts the data array of the input two-photon calcium imaging signal into an intermediate calculation amount of the same size as the reconstructed target image; The intermediate calculation amount is input into the automatic encoding module to generate a visual stimulus reconstructed image; wherein, The automatic encoding module includes a first network, a skip layer structure and a second network; The step of inputting the intermediate calculation amount into the automatic encoding module to generate a visual stimulation reconstructed image comprises: The first network performs a convolution operation on the vector of the intermediate calculation amount, and extracts features from the convolution result by downsampling; The skip-layer structure fuses the extracted features to obtain fused features; The second network upsamples and convolves the fused features to restore a reconstructed image of the same size as the visual stimulus; The reconstructed image output module is used to output the reconstructed image corresponding to the two-photon calcium imaging signal.

7. A computer storage medium, characterized in that: The computer storage medium stores a plurality of instructions, which are suitable for being loaded by a processor and executing the method steps according to any one of claims 1 to 5.

8. A terminal, characterized in that: include: A processor and a memory; wherein the memory stores a computer program, and the computer program is suitable for being loaded by the processor and executing the method steps as claimed in any one of claims 1 to 5.

Images