Mouse retinal signal reconstruction method and device, electronic device and storage medium

Abstract

The present application provides a mouse retinal signal reconstruction method and device, electronic device and storage medium, which belong to the field of image processing technology. A first generator is used to sequentially reconstruct the first stimulation paradigm of each retinal electrical imaging signal of the local, global, flicker, color, shape and video stimulation paradigm to obtain multiple first predicted stimulation paradigms. A discriminator is used to perform image discrimination on the local, global, flicker, color, shape, video stimulation paradigm and each first predicted stimulation paradigm to construct first and second losses. A third loss is determined according to the local, global, flicker, color, shape, video stimulation paradigm, each retinal electrical imaging signal and each first predicted stimulation paradigm. The first generator and the discriminator are alternately updated according to the first, second and third losses to obtain a target generator. The stimulation paradigm of the target retinal electrical imaging signal is reconstructed by the target generator, which can accurately simulate the encoding method of visual stimulation information.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a method and device for reconstructing mouse retinal signals, an electronic device, and a storage medium. Background Art

[0002] Visual information acquisition primarily occurs in the visual system. Light passes through the cornea, pupil, lens, and vitreous humor, focusing on the retina. The retina encodes the light signals into electrical signals, which are then transmitted to the brain via the optic nerve. The brain decodes these signals to produce visual information. Understanding the encoding mechanism of the visual system will not only aid in the development of treatments for optic nerve damage such as glaucoma and macular degeneration, but will also facilitate the development of artificial bionic eyes and machine vision. However, the encoding characteristics of visual stimuli by the visual system are currently unknown. Summary of the Invention

[0003] The main purpose of the embodiments of the present application is to propose a mouse retinal signal reconstruction method and device, electronic equipment and storage medium, aiming to accurately simulate and predict the way the visual system encodes visual stimulus information and understand the response pattern of the visual system to various visual stimuli.

[0004] To achieve the above objectives, a first aspect of the embodiments of the present application provides a method for reconstructing mouse retinal signals, the method comprising:

[0005] Acquire various retinal electrical imaging signals under local stimulation paradigm, global stimulation paradigm, flicker stimulation paradigm, color stimulation paradigm, shape stimulation paradigm and video stimulation paradigm;

[0006] Reconstructing a first stimulation paradigm for each retinal electrical imaging signal in sequence using a first generator to obtain a first predicted stimulation paradigm for each retinal electrical imaging signal; the first generator is composed of a first multi-layer perceptron and a first three-dimensional convolutional neural network connected in series; the first three-dimensional convolutional neural network includes five first three-dimensional convolutional layers;

[0007] The discriminator sequentially performs image discrimination on the local stimulation paradigm, the global stimulation paradigm, the flicker stimulation paradigm, the color stimulation paradigm, the shape stimulation paradigm, and the video stimulation paradigm to obtain a plurality of first target stimulation paradigm convolution features, and constructs a first loss based on the plurality of first target stimulation paradigm convolution features; the first loss is the sum of the losses of the local stimulation paradigm, the global stimulation paradigm, the flicker stimulation paradigm, the color stimulation paradigm, the shape stimulation paradigm, and the video stimulation paradigm; the discriminator includes four second three-dimensional convolution layers;

[0008] Performing image discrimination on each of the first predicted stimulus paradigms in sequence by the discriminator to obtain a plurality of second target stimulus paradigm convolution features, and constructing a second loss based on the plurality of second target stimulus paradigm convolution features; the second loss is the sum of the losses of all the first predicted stimulus paradigms;

[0009] determining a third loss based on the local stimulation paradigm, the global stimulation paradigm, the flicker stimulation paradigm, the color stimulation paradigm, the shape stimulation paradigm, the video stimulation paradigm, each of the retinal electrographic signals, and each of the first predicted stimulation paradigms;

[0010] Fixing the model parameters of the first generator, and updating the model parameters of the discriminator according to the first loss and the second loss;

[0011] Fixing the model parameters of the discriminator, and updating the model parameters of the first generator according to the second loss and the third loss;

[0012] Alternatingly updating the first generator and the discriminator until the first generator converges to obtain a target generator;

[0013] A target retinal electrical imaging signal is acquired, and a stimulation paradigm is reconstructed for the target retinal electrical imaging signal by the target generator.

[0014] To achieve the above-mentioned purpose, a second aspect of the embodiments of the present application provides a mouse retinal signal reconstruction device, comprising:

[0015] The first acquisition module is used to acquire various retinal electrical imaging signals under the local stimulation paradigm, the global stimulation paradigm, the flicker stimulation paradigm, the color stimulation paradigm, the shape stimulation paradigm and the video stimulation paradigm;

[0016] a reconstruction module, configured to sequentially reconstruct a first stimulation paradigm for each retinal electrical imaging signal using a first generator, to obtain a first predicted stimulation paradigm for each retinal electrical imaging signal; the first generator being composed of a first multilayer perceptron and a first three-dimensional convolutional neural network connected in series;

[0017] a first discrimination module, configured to perform image discrimination on the local stimulation paradigm, the global stimulation paradigm, the flicker stimulation paradigm, the color stimulation paradigm, the shape stimulation paradigm, and the video stimulation paradigm in sequence through a discriminator to obtain a plurality of first target stimulation paradigm convolution features, and construct a first loss based on the plurality of first target stimulation paradigm convolution features; the first loss is the sum of the losses of the local stimulation paradigm, the global stimulation paradigm, the flicker stimulation paradigm, the color stimulation paradigm, the shape stimulation paradigm, and the video stimulation paradigm;

[0018] a second discrimination module, configured to perform image discrimination on each of the first predicted stimulation paradigms in sequence through the discriminator to obtain a plurality of second target stimulation paradigm convolution features, and construct a second loss based on the plurality of second target stimulation paradigm convolution features; the second loss being the sum of the losses of all the first predicted stimulation paradigms;

[0019] a calculation module, configured to determine a third loss based on the local stimulation paradigm, the global stimulation paradigm, the flicker stimulation paradigm, the color stimulation paradigm, the shape stimulation paradigm, the video stimulation paradigm, each of the retinal electroimaging signals, and each of the first predicted stimulation paradigms;

[0020] A first updating module, configured to fix the model parameters of the first generator and update the model parameters of the discriminator according to the first loss and the second loss;

[0021] A second updating module is used to fix the model parameters of the discriminator and update the model parameters of the first generator according to the second loss and the third loss;

[0022] A third updating module is used to alternately update the first generator and the discriminator until the first generator converges to obtain a target generator;

[0023] The second acquisition module is used to acquire the target retinal electrical imaging signal and reconstruct the stimulation paradigm of the target retinal electrical imaging signal through the target generator.

[0024] To achieve the above-mentioned purpose, the third aspect of an embodiment of the present application proposes an electronic device, which includes a memory and a processor. The memory stores a computer program, and when the processor executes the computer program, the mouse retinal signal reconstruction method of the first aspect mentioned above is implemented.

[0025] To achieve the above objectives, the fourth aspect of the embodiments of the present application proposes a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the mouse retinal signal reconstruction method of the first aspect.

[0026] The mouse retinal signal reconstruction method, mouse retinal signal reconstruction device, electronic device and computer-readable storage medium proposed in this application adopt retinal electrical imaging signals under six visual stimuli, namely local stimulation paradigm, global stimulation paradigm, flicker stimulation paradigm, color stimulation paradigm, shape stimulation paradigm and video stimulation paradigm, in order to obtain the encoding method of the visual system for different visual stimuli. In order to obtain the encoding method from electrical signals to visual information, the first generator is used to reconstruct the first stimulation paradigm of each retinal electrical imaging signal in turn to obtain the first predicted stimulation paradigm of each retinal electrical imaging signal. The first generator is composed of a first multi-layer perceptron and a first three-dimensional convolutional neural network connected in series. The first multi-layer perceptron can encode the retinal electrical imaging signal into a higher-level feature representation, and the first three-dimensional convolutional neural network can retain the spatial characteristics of the visual stimulus to ensure that the reconstructed first predicted stimulation paradigm can accurately reflect the bioelectric activity. In order to improve the reconstruction quality of the stimulation paradigm by the first generator, a discriminator is introduced to adjust the first generator according to the output feedback of the discriminator. The discriminator sequentially performs image discrimination on the local, global, flicker, color, shape, and video stimulus paradigms to obtain a first loss. The discriminator then sequentially performs image discrimination on each of the first predicted stimulus paradigms to obtain a second loss. To ensure that the first predicted stimulus paradigms under different visual stimuli are close to their corresponding real visual stimuli, a third loss is determined based on the local, global, flicker, color, shape, and video stimulus paradigms, the individual electroretinogram signals, and each of the first predicted stimulus paradigms. To improve the discriminator's accuracy, the model parameters of the first generator are fixed and updated based on the first and second losses. The model parameters of the discriminator are also fixed and updated based on the second and third losses to ensure that the first predicted stimulus paradigm is close to the real visual stimulus, thereby improving the accuracy of the first generator's reconstruction of the stimulus paradigm. The first generator and the discriminator are updated alternately, using an alternating training strategy that accelerates training while improving the model's generalization performance, resulting in the target generator. Acquiring the target retinal electrical imaging signal and reconstructing the stimulation paradigm of the target retinal electrical imaging signal through the target generator can accurately simulate the way the visual system encodes different visual stimuli. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 This is a flow chart of the mouse retinal signal reconstruction method provided in the embodiments of the present application;

[0028] Figure 2a is a schematic diagram of the local stimulation paradigm provided in the embodiments of the present application;

[0029] Figure 2bis a schematic diagram of the global stimulation paradigm provided in the embodiments of the present application;

[0030] Figure 2c is a schematic diagram of the flicker stimulation paradigm provided in the embodiments of the present application;

[0031] Figure 2d is a schematic diagram of the color stimulation paradigm provided in the embodiments of the present application;

[0032] Figure 2e is a schematic diagram of the shape stimulation paradigm provided in the embodiments of the present application;

[0033] Figure 2f is a schematic diagram of the video stimulation paradigm provided in the embodiments of the present application;

[0034] Figure 2g is a waveform diagram of the original retinal electroencephalography signal provided in an embodiment of the present application;

[0035] Figure 2h is a waveform diagram of the downsampled retinal electroencephalography signal provided in an embodiment of the present application;

[0036] Figure 3 yes Figure 1 Flowchart of step S120 in FIG.

[0037] Figure 4 yes Figure 1 Flowchart of step S130 in FIG.

[0038] Figure 5 yes Figure 1 Flowchart of step S150 in FIG.

[0039] Figure 6 yes Figure 5 Flowchart of step S510 in FIG.

[0040] Figure 7 yes Figure 5 Flowchart of step S540 in FIG.

[0041] Figure 8 yes Figure 1 Flowchart of step S160 in FIG.

[0042] Figure 9a This is a rendering of the reconstruction stimulation paradigm provided in the embodiments of the present application;

[0043] Figure 9b This is a rendering of the real stimulation paradigm provided in the embodiments of the present application;

[0044] Figure 9c This is another effect diagram of the reconstruction stimulation paradigm provided in an embodiment of the present application;

[0045] Figure 9d This is another effect diagram of the real stimulation paradigm provided by the embodiment of the present application;

[0046] Figure 9e This is another effect diagram of the reconstruction stimulation paradigm provided in an embodiment of the present application;

[0047] Figure 9f This is another effect diagram of the reconstruction stimulation paradigm provided in an embodiment of the present application;

[0048] Figure 9g This is another effect diagram of the real stimulation paradigm provided by the embodiment of the present application;

[0049] Figure 10a is an image of a dataset sample provided in an embodiment of the present application;

[0050] Figure 10b is an image of another dataset sample provided by an embodiment of the present application;

[0051] Figure 10c This is another effect diagram of the reconstruction stimulation paradigm provided in an embodiment of the present application;

[0052] Figure 10d This is another effect diagram of the reconstruction stimulation paradigm provided in an embodiment of the present application;

[0053] Figure 10e This is an effect diagram of the reconstructed ERG signal provided by an embodiment of the present application;

[0054] Figure 10f This is another effect diagram of the reconstructed ERG signal provided by an embodiment of the present application;

[0055] Figure 11 Schematic diagram of the structure of the mouse retinal signal reconstruction device provided in the embodiment of the present application;

[0056] Figure 12 This is a schematic diagram of the hardware structure of the electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0057] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0058] It should be noted that although the device schematics illustrate functional module divisions and the flowcharts illustrate logical sequences, in certain circumstances, the steps shown or described may be performed in a sequence that differs from the module divisions in the device or the sequence in the flowcharts. The terms "first," "second," and so on, in the specification, claims, and drawings, are used to distinguish similar items and are not necessarily used to describe a specific sequence or precedence.

[0059] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which this application pertains. The terms used herein are for the purpose of describing the embodiments of this application only and are not intended to limit this application.

[0060] Over 70% of sensory information comes from vision. Visual information acquisition primarily occurs in the visual system. Light passes through the cornea, pupil, lens, and vitreous humor, focusing on the retina. The retina encodes the light signals into electrical signals, which are then transmitted to the brain via the optic nerve. The brain decodes these signals to produce visual information. Understanding the encoding mechanism of the visual system will not only aid in the development of treatments for optic nerve damage such as glaucoma and macular degeneration, but will also facilitate the development of artificial bionic eyes and machine vision. However, the encoding characteristics of visual stimuli by the visual system are currently unknown.

[0061] Based on this, the embodiments of the present application provide a mouse retinal signal reconstruction method, a mouse retinal signal reconstruction device, an electronic device and a computer-readable storage medium, which aim to accurately simulate and predict the way the visual system encodes visual stimulus information and understand the response pattern of the visual system to various visual stimuli.

[0062] The mouse retinal signal reconstruction method, mouse retinal signal reconstruction device, electronic device and computer-readable storage medium provided in the embodiments of the present application are specifically illustrated by the following examples. First, the mouse retinal signal reconstruction method in the embodiments of the present application is described.

[0063] The mouse retinal signal reconstruction method provided in the embodiment of the present application relates to the field of image processing technology. The mouse retinal signal reconstruction method provided in the embodiment of the present application can be applied to a terminal, can also be applied to a server side, and can also be software running in a terminal or a server side. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, etc.; the server side can be configured as an independent physical server, or can be configured as a server cluster or a distributed system composed of multiple physical servers, or can be configured as a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms; the software can be an application that implements the mouse retinal signal reconstruction method, etc., but is not limited to the above forms.

[0064] Figure 1 This is an optional flow chart of the mouse retinal signal reconstruction method provided in the embodiments of the present application. Figure 1 The method may include but is not limited to steps S110 to S160.

[0065] Step S110, acquiring various retinal electrical imaging signals under the local stimulation paradigm, the global stimulation paradigm, the flicker stimulation paradigm, the color stimulation paradigm, the shape stimulation paradigm, and the video stimulation paradigm;

[0066] Step S120: reconstructing a first stimulation paradigm for each retinal electrical imaging signal in sequence using a first generator to obtain a first predicted stimulation paradigm for each retinal electrical imaging signal; the first generator is composed of a first multilayer perceptron and a first three-dimensional convolutional neural network connected in series; the first three-dimensional convolutional neural network includes five first three-dimensional convolutional layers;

[0067] Step S130: performing image discrimination on the local stimulation paradigm, the global stimulation paradigm, the flicker stimulation paradigm, the color stimulation paradigm, the shape stimulation paradigm, and the video stimulation paradigm in sequence through a discriminator to obtain a plurality of first target stimulation paradigm convolution features, and constructing a first loss based on the plurality of first target stimulation paradigm convolution features; the first loss is the sum of the losses of the local stimulation paradigm, the global stimulation paradigm, the flicker stimulation paradigm, the color stimulation paradigm, the shape stimulation paradigm, and the video stimulation paradigm; the discriminator includes four second three-dimensional convolution layers;

[0068] Step S140: performing image discrimination on each of the first predicted stimulus paradigms in sequence through a discriminator to obtain a plurality of second target stimulus paradigm convolution features, and constructing a second loss based on the plurality of second target stimulus paradigm convolution features; the second loss is the sum of the losses of all the first predicted stimulus paradigms;

[0069] Step S150, determining a third loss based on the local stimulation paradigm, the global stimulation paradigm, the flicker stimulation paradigm, the color stimulation paradigm, the shape stimulation paradigm, the video stimulation paradigm, each retinal electrographic signal, and each first predicted stimulation paradigm;

[0070] Step S160, fixing the model parameters of the first generator, and updating the model parameters of the discriminator according to the first loss and the second loss;

[0071] Step S170, fixing the model parameters of the discriminator, and updating the model parameters of the first generator according to the second loss and the third loss;

[0072] Step S180, alternately updating the first generator and the discriminator until the first generator converges to obtain the target generator;

[0073] Step S190 , obtaining a target retinal electrical imaging signal, and reconstructing a stimulation paradigm for the target retinal electrical imaging signal through a target generator.

[0074] In step S110 of some embodiments, to simulate how visual information is encoded in the visual system, the present application designs a stimulation paradigm and applies it to mouse retinal cells. The stimulation paradigm is a visual stimulus signal used to simulate visual information that might be encountered in the natural environment, activating retinal neurons and causing them to generate bioelectric signals. To simulate and accurately predict how images are encoded in the visual system, the stimulation paradigm can employ an image sequence. These stimulation paradigms are projected directly onto retinal cells using a high-definition projection device. This requires precise calibration and control of the projection device to ensure that the projected signal is positioned at the intended location for a specific cell and that each cell receives the correct stimulus amount and type. The stimulus amount can be stimulus parameters such as brightness level, color saturation, and shape and size, which can vary over time. The high customizability of Psychtoolbox enables these stimulus parameters to be adjusted as needed. It should be noted that the resolution of the projection device needs to be adjusted to accommodate the size of the mouse retina to ensure that the stimulation paradigm covers the retinal region of interest, and different lighting conditions must be set to ensure the accuracy and reproducibility of the experimental data.

[0075] The non-invasive multi-electrode arrays (MEA) combined with microelectronic chip manufacturing technology have the characteristics of high density, high sensitivity and portability. MEA technology can be used to record the activity of retinal neurons in real time to collect retinal electroretinography (ERG) signals of the mouse retina, providing high-quality data for subsequent retinal signal reconstruction. By attaching mouse retinal tissue to MEA chip electrodes, the electrophysiological activity of any site of the tissue can be recorded and stimulated. This type of electrophysiological activity data is called retinal electroretinography signal. The stimulation paradigm is single and the bioelectric signals are few, which makes it impossible to accurately model the encoding process of visual information, hindering the in-depth exploration of advanced cognitive functions such as vision. This application uses six stimulation paradigms to activate retinal cells and collect retinal electroretinography signals under these six stimulation paradigms. These six stimulation paradigms are local stimulation paradigm, global stimulation paradigm, flickering stimulation paradigm, color stimulation paradigm, shape stimulation paradigm and video stimulation paradigm. The Psychtoolbox software package can be used to design stimulation paradigms in the MATLAB environment to generate visual stimulation paradigms with different change frequencies, colors and shapes. The Psychtoolbox software package is a widely used toolbox in visual and auditory psychology experiments. It provides a series of powerful functions for precisely controlling the timing and presentation of complex visual and auditory stimuli. Among them, the local stimulation paradigm and the global stimulation paradigm are used to analyze the response of the retina to changes in light intensity waveforms. The local stimulation paradigm is used to analyze the response of the retina to changes in light intensity waveforms. Figure 2a As shown in the global stimulation paradigm Figure 2b As shown. Figure 2a and Figure 2b In the experiment, both the local stimulation paradigm and the global stimulation paradigm used images of white circles on a black background. The size of the white circles in the global stimulation paradigm was larger than that in the local stimulation paradigm. The flicker stimulation paradigm is used to simulate dynamically changing visual scenes in the natural environment. The retina's response to different flicker speeds is tested by adjusting the image presentation frequency. When determining the optimal frequency range (1Hz to 20Hz), the frequency is gradually adjusted and the changes in retinal activity are monitored to avoid frequencies that are too fast or too slow to be effectively detected by the retina. Figure 2c As shown in the figure, a flickering stimulation paradigm is constructed by constantly alternating black and white images. The color stimulation paradigm is used to test the sensitivity of the retina to different spectral components in order to analyze the response of the retina to different colors. The stimulation adjustment of color changes can be performed by adjusting the color intensity and hue. Taking into account the limited perception of certain colors by the mouse retina, the three primary colors (red, green, and blue) are selected to explore the color vision characteristics and analyze the differences in the effects of these colors on the retina, which is of great significance for understanding the visual signal processing mechanism of mice and potential color vision disorders. Figure 2dAs shown in , the color stimulation paradigm is constructed by constantly alternating images of different colors. The shape stimulation paradigm includes geometric figures of different shapes, covering the transition from simple geometric figures to complex patterns, and is used to analyze the retinal response to shape changes. Figure 2e As shown in the figure, the shape stimulation paradigm is constructed by alternating geometric figures of different shapes. The video stimulation paradigm is constructed by multiple consecutive video frames and is used to analyze the retinal response to video changes. Figure 2f shown.

[0076] MEAs record bioelectrical activity at a sampling rate of up to 25,000 times per second to capture subtle changes in retinal cells under various stimulation paradigms. High-frequency sampling provides extremely high temporal resolution, but it also generates a large amount of data. This large amount of data consumes storage space, placing pressure on storage devices in large-scale experiments. Furthermore, processing and analyzing large amounts of data requires significant computing resources, increasing both computational and time costs, and imposing a computational burden. In machine learning and deep learning, a large number of data points poses the risk of overfitting, potentially causing the model to learn noise rather than underlying, more general patterns. Excessively detailed data does not always improve model performance; instead, it may reduce training and inference efficiency. To address the storage pressure, computational burden, and overfitting risks associated with large amounts of data, and to improve overall processing efficiency, this application downsamples the retinal electroencephalography signals to a more reasonable size while retaining sufficient information to accurately reconstruct the stimulation paradigm. Downsampling aims to effectively convert the original signal to make it more suitable for model training. It should be noted that the sampling rate for downsampling can be customized based on actual circumstances. If the sampling rate is 1000, then the original 25,000 ERG signals per second will be compressed to 25 ERG signals per second. The original ERG signal is as follows: Figure 2g As shown, the downsampled ERG signal is as follows Figure 2h shown.

[0077] The stimulation paradigm can take the form of a video. Videos typically consist of a series of consecutive video frames, each of which is a separate image. When working with video-related machine learning tasks, especially when the goal is to correlate video frames with ERG signals, a series of steps are required to ensure data validity and model feasibility. In the retinal signal reconstruction scenario of this application, the video is segmented into one-second frames, with each second of video content converted into a separate image. This simplifies the complexity of the video data while retaining sufficient information to capture key visual content. The following details the process of segmenting a video into time-series images, one frame per second. The specific steps include: reading a video file and decoding it into a series of video frames. Video decoding can be implemented using a library such as OpenCV; determining the number of video frames in the video file based on the video frame rate and duration; extracting one frame per second from the video stream with this number of frames per second; and cropping the extracted frames into images of a preset size, which can be 512x512. Using these images as labels and the ERG signals as input, a dataset is created using the PyTorch Dataset class.

[0078] This application uses a generative adversarial network architecture to learn the mapping of bioelectric signals to the original stimulus paradigm in an end-to-end manner, eliminating the need for manually designed feature extraction and reconstruction steps to simulate the encoding and transformation processes of visual information. This not only helps us understand how visual information is encoded and transformed in the retina, but the successful implementation of this method also demonstrates the advancement of neural interface technology and has potential application value in the development of new visual prostheses that can restore or enhance human sensory function. Furthermore, precise stimulation and data acquisition help validate and improve theoretical models of visual signal processing, providing new strategies for the diagnosis and treatment of visual impairments. For example, in the study of retinal degenerative diseases, by analyzing ERG signals, functional changes and pathological states of the retina can be inferred, providing guidance for the design and optimization of clinical interventions. Furthermore, it provides a new experimental platform for high-throughput drug screening and neurological disease research based on MEA technology. Through this reverse engineering, we can not only gain a deeper understanding of the working mechanisms of biological sensory systems but also gain a more comprehensive understanding of the information processing capabilities of the human brain, opening up new avenues for future biomedical research and applications. The present application reconstructs the stimulation paradigm of each retinal electrical imaging signal obtained under six stimulation paradigms in sequence through the first generator, and the first generator is a reconstructed stimulation signal model. Retinal electrical imaging signals are usually manifested as one-dimensional electrical signal waveforms. Converting a one-dimensional signal into an image sequence that can represent the timing of visual stimulation requires not only encoding the temporal characteristics of the signal, but also retaining the spatial characteristics of the visual stimulation. Therefore, the constructed model must have a high degree of complexity and accuracy to ensure that the reconstructed image can accurately reflect the original bioelectric activity. In this scenario, the overall framework of the first generator consists of two main parts: a first multi-layer perceptron (MLP) and a first three-dimensional convolutional neural network. The first multi-layer perceptron is a multi-layer perceptron within the reconstructed stimulation signal model, and the first three-dimensional convolutional neural network is a three-dimensional convolutional neural network within the reconstructed stimulation signal model, which is used to create a time series image that is as similar as possible to the real ERG signal. The first three-dimensional convolutional neural network includes five first three-dimensional convolutional layers, and the first three-dimensional convolutional layer is a three-dimensional convolutional layer of the first three-dimensional convolutional neural network. Each first three-dimensional convolutional layer is followed by a normalization layer and a ReLU activation function. These five first three-dimensional convolutional layers are all deconvolution layers. Each layer further abstracts and transforms the extracted signal features to generate complex spatiotemporal data. By stacking them layer by layer, the generator is able to produce high-quality image sequences from the high-dimensional features extracted by the multilayer perceptron. The following is a detailed description of the specific processing of the first generator.

[0079] See also Figure 3 In some embodiments, step S120 may include but is not limited to steps S310 to S360:

[0080] Step S310, performing feature extraction on each retinal electrical imaging signal by a first multi-layer perceptron to obtain retinal signal features of each retinal electrical imaging signal;

[0081] Step S320: performing a first three-dimensional convolution on each retinal signal feature through a first first three-dimensional convolution layer to obtain multiple first retinal convolution features; the first first three-dimensional convolution layer has 100 input channels, 256 output channels, a convolution kernel size of 4, a sliding step size of 1, and a padding value of 0;

[0082] Step S330: Perform a second three-dimensional convolution on each first retinal convolution feature through a second first three-dimensional convolution layer to obtain multiple second retinal convolution features; the second first three-dimensional convolution layer has 256 input channels, 128 output channels, a convolution kernel size of 4, a sliding step size of 2, and a padding value of 1;

[0083] Step S340: Perform a third three-dimensional convolution on each second retinal convolution feature through a third first three-dimensional convolution layer to obtain multiple third retinal convolution features; the convolution kernel size includes a convolution width, a convolution height, and a convolution depth, and the sliding step includes a first sliding sub-step in the convolution width direction, a second sliding sub-step in the convolution height direction, and a third sliding sub-step in the convolution depth direction; the third first three-dimensional convolution layer has 128 input channels, 64 output channels, 1 convolution width, 4 convolution height, 4 convolution depth, a first sliding sub-step of 1, a second sliding sub-step of 2, a third sliding sub-step of 2, and a padding value of 1;

[0084] Step S350: Perform a fourth three-dimensional convolution on each third retinal convolution feature through a fourth first three-dimensional convolution layer to obtain multiple fourth retinal convolution features; the fourth first three-dimensional convolution layer has 64 input channels, 32 output channels, a convolution width of 1, a convolution height of 4, a convolution depth of 4, a first sliding sub-step of 1, a second sliding sub-step of 2, a third sliding sub-step of 2, and a padding value of 1;

[0085] Step S360, performing a fifth three-dimensional convolution on each fourth retinal convolution feature through the fifth first three-dimensional convolution layer to obtain a first predicted stimulation paradigm for each retinal electrical imaging signal; the number of input channels of the fifth first three-dimensional convolution layer is 32, the number of output channels is 160, the convolution width is 2, the convolution height is 4, the convolution depth is 4, the first sliding sub-step is 1, the second sliding sub-step is 2, the third sliding sub-step is 2, and the padding value is 1.

[0086] In step S310 of some embodiments, the first multilayer perceptron includes a series of fully connected layers for extracting and converting features in the retinal electrographic imaging signal. The first multilayer perceptron sequentially extracts features from the retinal electrographic imaging signals under the six stimulation paradigms, encodes the original one-dimensional signal waveform into a higher-level feature representation, and obtains the retinal signal features of each retinal electrographic imaging signal. Retinal signal features are waveform features used to characterize retinal bioelectrical activity, which can be oscillatory potentials, time-domain Gaussian parameters, ratio parameters, etc. Oscillatory potentials are high-frequency components in ERG signals. Time-domain Gaussian parameters are parameters obtained by performing Gaussian fitting on the waveform after the envelope function is fitted after the oscillatory potential is Hilbert transformed in the time domain. The ratio parameter is the ratio of the oscillatory potential amplitude to the b-wave amplitude in the ERG signal. The process of feature extraction by the first multilayer perceptron can be expressed as: E=MLP(ERG), where ERG∈R 25000 Represents the original ERG signal, E∈R 1000 represents the feature representation output after MLP feature extraction, R represents the real feature space, 25000 represents the feature dimension of the original ERG signal, and 1000 represents the feature dimension of the output feature.

[0087] In step S320 of some embodiments, the retinal signal feature is multi-channel data and can be expressed as (C, W, H, D), where C is the number of characteristic channels of the retinal signal feature, W (width) is the characteristic width of the retinal signal feature, H (height) is the characteristic height of the retinal signal feature, and D (depth) is the characteristic depth of the retinal signal feature. The retinal signal feature is used as the input of the first three-dimensional convolutional neural network to generate a time series image. The input and output of the three-dimensional convolution layer satisfy the following parameter relationship:

[0088]

[0089]

[0090]

[0091] C out =K

[0092] Among them, w in 、H in 、D in Respectively represent the feature width, feature height and feature depth of the input feature, w out 、H out 、D out 、C outThey represent the feature width, feature height, feature depth and number of feature channels of the output features respectively; K is the number of convolution kernels in the three-dimensional convolution layer; p represents the padding value; s represents the sliding step size; w*h*d represents the convolution kernel dimension, w represents the width of the convolution kernel, h represents the height of the convolution kernel, and d represents the depth of the convolution kernel.

[0093] Each retinal signal feature undergoes a first 3D convolution through the first 3D convolution layer. These convolved retinal signal features are then normalized to obtain multiple normalized features. These normalized features are then activated using the ReLU activation function to obtain multiple first retinal convolution features, with each retinal signal feature corresponding to a first retinal convolution feature. Deconvolution is used in feature extraction to restore image size and extract richer features. First retinal convolution features are image features reconstructed based on retinal signal features, and have higher resolution than retinal signal features. Normalization is performed using a 3D normalization function, which can be batch normalization, layer normalization, or instance normalization. The first 3D convolution layer has 100 input channels and 256 output channels, a convolution kernel size of 4, a sliding stride of 1, and zero padding. The convolution kernel dimensions include convolution width, convolution height, and convolution depth. Convolution width refers to the width of the convolution kernel, convolution height refers to the height of the convolution kernel, and convolution depth refers to the depth of the convolution kernel. The sliding step size includes the first sliding sub-step size in the convolution width direction, the second sliding sub-step size in the convolution height direction, and the third sliding sub-step size in the convolution depth direction. It should be noted that a convolution kernel size of 4 means that the width, height, and depth of the convolution kernel are all 4, which can be expressed as (4,4,4). A sliding step size of 1 means that the first sliding sub-step size, the second sliding sub-step size, and the third sliding sub-step size are all 1, which can be expressed as (1,1,1).

[0094] In step S330 of some embodiments, each first retinal convolution feature corresponds to a second retinal convolution feature. The second retinal convolution feature is obtained by sequentially processing the first retinal convolution feature through a second first three-dimensional convolution layer, a normalization layer, and a ReLU activation function. The extraction process of the second retinal convolution feature can refer to the first retinal convolution feature and will not be repeated here. The second first three-dimensional convolution layer has 256 input channels, 128 output channels, a convolution kernel size of (4,4,4), a sliding step size of (2,2,2), and a padding value of 1.

[0095] In step S340 of some embodiments, each second retinal convolution feature corresponds to a third retinal convolution feature, and the third retinal convolution feature is obtained by sequentially processing the second retinal convolution feature through the third first three-dimensional convolution layer, the normalization layer, and the ReLU activation function. The extraction process of the third retinal convolution feature can refer to the first retinal convolution feature and will not be repeated here. The number of input channels of the third first three-dimensional convolution layer is 128, the number of output channels is 64, the convolution kernel size is (1,4,4), the sliding step is (1,2,2), and the padding value is 1. The convolution kernel size is (1,4,4), that is, the convolution width is 1, the convolution height is 4, and the convolution depth is 4. The sliding step is (1,2,2), that is, the first sliding sub-step is 1, the second sliding sub-step is 2, and the third sliding sub-step is 2.

[0096] In step S350 of some embodiments, the fourth first three-dimensional convolution layer has 64 input channels, 32 output channels, a convolution kernel size of (1, 4, 4), a sliding step size of (1, 2, 2), and a padding value of 1. Each third retinal convolution feature is input into the fourth first three-dimensional convolution layer, the normalization layer, and the ReLU activation layer for a fourth three-dimensional convolution to obtain a fourth retinal convolution feature corresponding to each third retinal convolution feature. The feature depths of the first retinal convolution feature, the second retinal convolution feature, the third retinal convolution feature, and the fourth retinal convolution feature increase in sequence, representing a deep feature representation.

[0097] In step S360 of some embodiments, the fifth first three-dimensional convolutional layer has 32 input channels, 160 output channels, a convolution kernel size of (2, 4, 4), a sliding step size of (1, 2, 2), and a padding value of 1. Each fourth retinal convolution feature is input into the fifth first three-dimensional convolutional layer, the normalization layer, and the ReLU activation layer for a fifth three-dimensional convolution to obtain a first predicted stimulus paradigm corresponding to each retinal electrographic imaging signal. The first predicted stimulus paradigm is a temporal image sequence reconstructed by the first generator based on the retinal electrographic imaging signal.

[0098] In the above steps S310 to S360, the first multi-layer perceptron can extract high-level feature representations of retinal electrical imaging signals, and the three-dimensional deconvolution layers stacked layer by layer can reconstruct the stimulation paradigm from the high-level feature representations to simulate the encoding process of the visual system for bioelectric signals and understand the response patterns of the visual system to different stimulation paradigms.

[0099] In the generative adversarial network framework, a discriminator is included in addition to the generator. The discriminator evaluates the similarity between images generated by the generator and real ERG signal recordings, distinguishing whether the input image is generated by the generator or from a real ERG signal set. During training, the generator and the discriminator engage in a game of negotiation, with the generator striving to produce increasingly realistic images while the discriminator attempts to improve its ability to distinguish between real and fake. This dynamic generative adversarial process forces the generator to learn how to improve its output until it can ultimately deceive the discriminator. In this way, the model not only learns how to reconstruct visual stimuli from ERG signals during training, but also improves the realism and accuracy of generated images. This enables the 3D convolutional model combining MLP and GAN to transform complex 1D electrical signals into meaningful visual representations. The discriminator consists of four second 3D convolutional layers, all of which are convolutional layers, each followed by a 3D normalization function and a LeakyRelu activation function.

[0100] See also Figure 4 In some embodiments, step S130 may include but is not limited to steps S410 to S460:

[0101] Step S410, performing a first three-dimensional convolution on the local stimulation paradigm, the global stimulation paradigm, the flicker stimulation paradigm, the color stimulation paradigm, the shape stimulation paradigm, and the video stimulation paradigm in sequence through the first second three-dimensional convolution layer to obtain multiple first stimulation paradigm convolution features; the convolution kernel size includes convolution width, convolution height, and convolution depth; the first second three-dimensional convolution layer has 160 input channels, 128 output channels, 1 convolution width, 4 convolution height, 4 convolution depth, a sliding step size of 2, and a padding value of 1;

[0102] Step S420, performing a second three-dimensional convolution on each first stimulus paradigm convolution feature through a second second three-dimensional convolution layer to obtain multiple second stimulus paradigm convolution features; the second second three-dimensional convolution layer has an input channel number of 128, an output channel number of 256, a convolution width of 1, a convolution height of 4, a convolution depth of 4, a sliding step size of 2, and a padding value of 1;

[0103] Step S430, performing a third three-dimensional convolution on each second stimulus paradigm convolution feature through a third second three-dimensional convolution layer to obtain multiple third stimulus paradigm convolution features; the third second three-dimensional convolution layer has an input channel number of 256, an output channel number of 512, a convolution width of 1, a convolution height of 4, a convolution depth of 4, a sliding step size of 2, and a padding value of 1;

[0104] Step S440: Perform a fourth three-dimensional convolution on each third stimulus paradigm convolution feature through a fourth second three-dimensional convolution layer to obtain multiple first target stimulus paradigm convolution features; the fourth second three-dimensional convolution layer has 512 input channels, 160 output channels, a convolution width of 2, a convolution height of 8, a convolution depth of 8, a sliding step of 1, and a padding value of 0;

[0105] Step S450, calculating the expected log-likelihood of each first target stimulus paradigm convolution feature to obtain the loss of each first target stimulus paradigm convolution feature;

[0106] In step S460 , the losses of the convolutional features of each first target stimulus paradigm are summed to obtain a first loss.

[0107] In step S410 of some embodiments, the number of input channels of the first second three-dimensional convolution layer is 160, the number of output channels is 128, the convolution kernel size is (1, 4, 4), the sliding step size is (2, 2, 2), and the padding value is 1. The first second three-dimensional convolution layer sequentially performs the first three-dimensional convolution on the local stimulation paradigm, the global stimulation paradigm, the flickering stimulation paradigm, the color stimulation paradigm, the shape stimulation paradigm, and the video stimulation paradigm to obtain the convolution feature under each stimulation paradigm. The first stimulation paradigm convolution feature is a temporal image feature extracted from the stimulation paradigm. The convolution feature under each stimulation paradigm is processed respectively by the three-dimensional normalization function and the LeakyRelu activation function to obtain the first stimulation paradigm convolution feature corresponding to each convolution feature.

[0108] In step S420 of some embodiments, the number of input channels of the second second three-dimensional convolution layer is 128, the number of output channels is 256, the convolution kernel size is (1, 4, 4), the sliding step size is (2, 2, 2), and the padding value is 1. The second second three-dimensional convolution layer is used to perform a second three-dimensional convolution on each first stimulus paradigm convolution feature, and each first stimulus paradigm convolution feature after the second three-dimensional convolution is processed by a three-dimensional normalization function and a LeakyRelu activation function to obtain multiple second stimulus paradigm convolution features, each of which corresponds to a first stimulus paradigm convolution feature.

[0109] In step S430 of some embodiments, the number of input channels of the third second three-dimensional convolution layer is 256, the number of output channels is 512, the convolution kernel size is (1, 4, 4), the sliding step size is (2, 2, 2), and the padding value is 1. A third three-dimensional convolution is performed on each second stimulus paradigm convolution feature through the third second three-dimensional convolution layer, and each second stimulus paradigm convolution feature after the third three-dimensional convolution is processed by a three-dimensional normalization function and a LeakyRelu activation function to obtain multiple third stimulus paradigm convolution features. Each third stimulus paradigm convolution feature corresponds to a second stimulus paradigm convolution feature. The feature depths of the first stimulus paradigm convolution feature, the second stimulus paradigm convolution feature, and the third stimulus paradigm convolution feature increase in sequence.

[0110] In step S440 of some embodiments, the number of input channels of the fourth second three-dimensional convolution layer is 512, the number of output channels is 160, the convolution kernel size is (2, 8, 8), the sliding step size is (1, 1, 1), and the padding value is 0. The fourth three-dimensional convolution is performed on each third stimulation paradigm convolution feature through the fourth second three-dimensional convolution layer, and each third stimulation paradigm convolution feature after the fourth three-dimensional convolution is processed by the three-dimensional normalization function and the LeakyRelu activation function, and each processed feature is activated by the sigmoid function to obtain multiple first target stimulation paradigm convolution features, and each first target stimulation paradigm convolution feature corresponds to a third stimulation paradigm convolution feature. The first target stimulation paradigm convolution feature is used to characterize the true or false probability of the stimulation paradigm. Multiple first target stimulation paradigm convolution features can reflect the true or false probability of six stimulation paradigms.

[0111] In step S450 of some embodiments, the loss function of the discriminator is used to evaluate the difference between the generated image and the real image. The present application quantifies the uncertainty of the discriminator in distinguishing between true and false based on the cross-entropy loss function. Based on the spatial distribution of each real stimulus paradigm, the expectation of the log-likelihood of the convolution feature of each first target stimulus paradigm is calculated to obtain the loss of the convolution feature of each first target stimulus paradigm. This loss is the expectation of the log-likelihood predicted by the discriminator for the real stimulus paradigm, which measures the ability of the discriminator to correctly identify real samples as true. The purpose of the discriminator is to maximize the loss, that is, to identify all real samples as correctly as possible so that D(x) is close to 1. The loss is expressed as follows:

[0112] E x～pdata(x) [logD(x)]

[0113] Where D represents the discriminator; x represents the real stimulus paradigm, that is, one of the six stimulus paradigms. The convolution feature of the first target stimulus paradigm is expressed as D(x); E represents the expectation; pdata represents the spatial distribution of the real stimulus paradigm x.

[0114] In step S460 of some embodiments, the losses of the convolution features of the first target stimulation paradigm under the six real stimulation paradigms are summed to obtain a first loss.

[0115] Through the above steps S410 to S460, the first losses under the six stimulation paradigms can be obtained, and the model parameters of the discriminator can be adjusted based on the first losses, so that the discriminator can correctly identify the six stimulation paradigms as true, thereby improving the discriminator's ability to judge real samples.

[0116] In step S140 of some embodiments, each first predicted stimulation paradigm is a stimulation paradigm obtained by reconstructing the retinal electrographic imaging signal under six stimulation paradigms, and each first predicted stimulation paradigm corresponds to a real stimulation paradigm. Referring to steps S410 to S440, the discriminator performs image discrimination on each first predicted stimulation paradigm in turn to obtain a plurality of second target stimulation paradigm convolution features, and the second target stimulation paradigm convolution features are used to characterize the true or false probability of the first predicted stimulation paradigm output by the discriminator. The log-likelihood of each second target stimulation paradigm convolution feature is calculated, and the loss of the first predicted stimulation paradigm is determined according to the spatial distribution and log-likelihood of the first predicted stimulation paradigm. The losses of each first predicted stimulation paradigm obtained under the six stimulation paradigms are summed to obtain the second loss. The loss of the first predicted stimulation paradigm is expressed as follows:

[0117] E z～pz(z) [1-logD(G(z))]

[0118] Where G represents the first generator; G(z) represents the first predicted stimulus pattern; D represents the discriminator; p z Represents the latent space distribution of generated data.

[0119] It should be noted that the first generator generates fake samples G(z), and the discriminator needs to judge that the generated samples are not real, that is, D(G(z)) should be as close to 0 as possible. The goal of the discriminator is to maximize the second loss.

[0120] In generative adversarial networks, the design of the loss function is the core of the training process. Figure 5 In some embodiments, step S150 may include but is not limited to steps S510 to S540:

[0121] Step S510: Retinal signals are sequentially reconstructed for a local stimulation paradigm, a global stimulation paradigm, a flicker stimulation paradigm, a color stimulation paradigm, a shape stimulation paradigm, and a video stimulation paradigm using a second generator to obtain a plurality of first predicted retinal signals; the second generator is formed by serially connecting a second three-dimensional convolutional neural network and a second multilayer perceptron; the second three-dimensional convolutional neural network includes a third three-dimensional convolutional layer, a residual layer, and a self-attention layer;

[0122] Step S520, reconstructing the stimulation paradigm of each first predicted retinal signal by the first generator to obtain multiple second predicted stimulation paradigms;

[0123] Step S530, reconstructing retinal signals for each first predicted stimulation paradigm by a second generator to obtain a plurality of second predicted retinal signals;

[0124] Step S540, determining the third loss based on the local stimulation paradigm, the global stimulation paradigm, the flicker stimulation paradigm, the color stimulation paradigm, the shape stimulation paradigm, the video stimulation paradigm, each retinal electrographic imaging signal, each first predicted stimulation paradigm, each first predicted retinal signal, each second predicted stimulation paradigm, and each second predicted retinal signal.

[0125] In step S510 of some embodiments, the visual stimulus (stimulation paradigm) is converted into neural activity data (bioelectric signal), which helps to model the encoding and processing of visual information in the visual system. For retinal signal reconstruction, most of the existing technologies are to recover the original bioelectric signal from the lost or degraded bioelectric signal, and rarely reconstruct the stimulation paradigm from the bioelectric signal. The reason is that it is impossible to explicitly model the encoding process of the light stimulation paradigm in the retinal cells into electrical signals from a mathematical or physical perspective. By analyzing and reconstructing the stimulation signal in the mouse retinal electroretinography (ERG) signal, we can deeply understand the response pattern of retinal neurons to various visual stimuli. This not only helps to test and optimize the visual information processing model, but also lays the foundation for the development of cognitive science. In addition, reconstructing the ERG signal from the stimulation signal is to convert the visual stimulation signal into predicted neural activity data, so as to study how visual information is processed and encoded in the nervous system. This application introduces a second generator to establish a mapping from the stimulation paradigm to the retinal electrical imaging signal. The second generator sequentially reconstructs the retinal signals for six real stimulation paradigms, namely the local stimulation paradigm, the global stimulation paradigm, the flicker stimulation paradigm, the color stimulation paradigm, the shape stimulation paradigm, and the video stimulation paradigm, and obtains the first predicted retinal signal corresponding to each real stimulation paradigm, so as to derive the bioelectrical signal of the retinal cells from the known visual stimulation. The second generator is an ERG signal reconstruction model, and the first predicted retinal signal is the ERG signal reconstructed according to the real stimulation paradigm. The three-dimensional convolutional neural network can capture the feature changes in the time dimension and is an ideal choice for processing time series data such as video and dynamic image sequences. The second generator is composed of a second three-dimensional convolutional neural network and a second multi-layer perceptron connected in series. The second three-dimensional convolutional neural network and the second multi-layer perceptron are respectively the three-dimensional convolutional neural network and the multi-layer perceptron of the ERG signal reconstruction model. The second three-dimensional convolutional neural network is used to extract spatial features and temporal features from the stimulation paradigm, and can analyze the feature changes of the stimulation paradigm over time, which is crucial for understanding how the retina responds to visual stimulation that changes over time. The second multilayer perceptron is used to convert the one-dimensional feature vectors extracted by the second three-dimensional convolutional neural network into ERG signals. The second three-dimensional convolutional neural network includes five third-dimensional convolutional layers, each of which is followed by a residual layer and a self-attention layer. The third-dimensional convolutional layer can adopt a convolutional layer. By integrating the residual network and self-attention mechanism into the second three-dimensional convolutional neural network, the model's learning and generalization capabilities can be enhanced.

[0126] In step S520 of some embodiments, referring to steps S310 to S360, the first generator performs stimulation paradigm reconstruction on each first predicted retinal signal obtained by reconstructing six different real stimulation paradigms to obtain multiple second predicted stimulation paradigms. The first predicted retinal signal under each real stimulation paradigm corresponds to a second predicted stimulation paradigm, and the second predicted stimulation paradigm is a stimulation signal reconstructed using the reconstructed ERG signal.

[0127] In step S530 of some embodiments, a second generator reconstructs retinal signals for each first predicted stimulation paradigm (reconstructed stimulation signal) to obtain a second predicted retinal signal corresponding to each first predicted stimulation paradigm. The second predicted retinal signal is an ERG signal reconstructed using the reconstructed stimulation signal. The process of retinal signal reconstruction by the second generator can be referred to in step S510 and will not be repeated here.

[0128] In step S540 of some embodiments, the local stimulation paradigm, the global stimulation paradigm, the flicker stimulation paradigm, the color stimulation paradigm, the shape stimulation paradigm, and the video stimulation paradigm are compared with each first predicted stimulation paradigm, each retinal electrical imaging signal is compared with each first predicted retinal signal, the local stimulation paradigm, the global stimulation paradigm, the flicker stimulation paradigm, the color stimulation paradigm, the shape stimulation paradigm, and the video stimulation paradigm are compared with each second predicted stimulation paradigm, each retinal electrical imaging signal is compared with each second predicted retinal signal, and the third loss is determined.

[0129] The above steps S510 to S540 can obtain the third loss, so as to adjust the model parameters of the first generator based on the third loss, reduce the difference between the real ERG signal and the predicted ERG signal, and between the real stimulation paradigm and the predicted stimulation paradigm, improve the accuracy of visual information reconstruction, and accurately simulate the visual system's encoding process of visual information.

[0130] See also Figure 6 In some embodiments, step S510 may include but is not limited to steps S610 to S640:

[0131] Step S610, performing three-dimensional convolution on the local stimulation paradigm, the global stimulation paradigm, the flicker stimulation paradigm, the color stimulation paradigm, the shape stimulation paradigm, and the video stimulation paradigm in sequence through a third three-dimensional convolution layer to obtain a plurality of fourth stimulation paradigm convolution features;

[0132] Step S620, performing feature extraction on each fourth stimulation paradigm convolution feature through a residual layer to obtain a plurality of stimulation paradigm residual features;

[0133] Step S630: Perform feature conversion on the residual features of each stimulus paradigm through the self-attention layer to obtain a query vector, a key vector, and a value vector; determine an attention score based on the query vector and the key vector; and perform weighted calculation on the value vector based on the attention score to obtain multiple stimulus paradigm attention features;

[0134] Step S640: Perform feature conversion on the attention features of each stimulation paradigm through a second multi-layer perceptron to obtain a plurality of first predicted retinal signals.

[0135] In step S610 of some embodiments, three-dimensional convolution is sequentially performed on six real stimulus paradigms: the local stimulus paradigm, the global stimulus paradigm, the flicker stimulus paradigm, the color stimulus paradigm, the shape stimulus paradigm, and the video stimulus paradigm, through the first third three-dimensional convolution layer to obtain fourth stimulus paradigm convolution features corresponding to each real stimulus paradigm. The fourth stimulus paradigm convolution features are temporal image features extracted from the real stimulus paradigm.

[0136] In step S620 of some embodiments, deep neural networks often encounter performance saturation or even degradation problems after adding more layers. This is mainly because as the network layers increase, the gradient may gradually become smaller after back propagation through multiple layers due to the product chain rule, resulting in slow or even stopped updating of network weights during training, which will ultimately affect the learning ability and performance of the model. In order to alleviate the gradient vanishing problem that occurs during the training of deep neural networks, this application introduces a residual layer, and the residual layer adopts a residual network. The residual network solves this problem by introducing a structure called skip connection, which allows data to skip one or more layers to a certain extent and directly pass the input to the subsequent layers. In the residual network, the input is not only passed to the next layer, but also added to the output of the subsequent layer through the skip connection, so that even in the deep network, information can be effectively transmitted, thereby reducing the risk of gradient vanishing. This structure allows the network to maintain a more stable gradient flow during training, thereby allowing the network to perform deeper feature learning. Integrating a residual network into a 3D convolutional neural network significantly improves the model's training and generalization capabilities, particularly when processing complex spatiotemporal data, such as reconstructing ERG signals from stimulus signals. The residual network ensures that even the deeper parts of the network learn effective features, which is crucial for capturing subtle changes in visual stimulus signals, which can have a significant impact on ERG signal prediction. Each fourth stimulus paradigm convolutional feature is extracted through a residual layer, resulting in a stimulus paradigm residual feature corresponding to each fourth stimulus paradigm convolutional feature. The stimulus paradigm residual feature serves as the output feature of the residual layer.

[0137] In step S630 of some embodiments, the self-attention layer adopts a self-attention mechanism, the core idea of ​​which is to enable the neural network to focus on the important parts of the input data rather than treating all inputs equally. The self-attention mechanism focuses on key information by calculating the relationship between different parts of the input sequence. It can identify which parts of the sequence are important and which are secondary, and adjust the model's processing method accordingly. The model determines the importance of the value of each time point in the final output by comparing queries and keys at different time points. In the self-attention mechanism, each input data point is assigned a weight that reflects the importance of the point relative to other points. This means that the model not only learns the characteristics of the input data, but also learns the relationship between data points and their importance to the final task.

[0138] The query vector Q, key vector K and value vector V are obtained by linearly transforming the residual features of the stimulus paradigm through three different fully connected layers in the self-attention layer. These three vectors are expressed as follows:

[0139] Q=XW Q

[0140] K=XW K

[0141] V=XW V

[0142] Among them, X represents the input data, that is, the residual features of the stimulus paradigm, and W Q 、W K 、W V is a learnable weight matrix. Performing a dot product operation on the query vector and the key vector yields an attention score, which represents the similarity or association between different locations in the data. Normalizing the attention score using the softmax function yields a weight. Multiplying the weight by the value vector yields the stimulus paradigm attention feature. Performing the above steps for each stimulus paradigm residual feature yields multiple stimulus paradigm attention features. The stimulus paradigm attention feature is represented as:

[0143]

[0144] Among them, d k represents the dimension of the key vector, divided by This is to prevent the dot product from becoming too large when the dimension is high, causing the softmax function to enter the gradient saturation region and affect model learning.

[0145] When applied to a 3D convolutional network, the self-attention mechanism enables the network to more accurately capture spatiotemporal features relevant to the prediction task. Neural responses are often highly complex and dynamic, and this ability to accurately capture these features is particularly important in neuroscience. For example, when processing visual stimuli, the self-attention mechanism can help the model identify which time periods and spatial regions of activity are most critical for predicting ERG signals. This capability is particularly important for understanding complex visual stimuli and neural responses, as these data typically contain a wealth of information, not all of which is equally important. The self-attention mechanism enables the 3D convolutional network to not only process the spatiotemporal features of visual stimuli but also to identify the complex interactions that may exist between these features, such as the possibility that certain visual patterns may elicit significant neural responses at specific time points. Furthermore, different visual stimuli may influence neural activity in different ways. The self-attention mechanism provides the model with greater flexibility, enabling it to adaptively adjust its focus based on the task requirements and data characteristics of the ERG prediction task. This allows the model to more accurately identify and exploit these influences, thereby improving the accuracy of ERG prediction. The self-attention mechanism also enhances the model's ability to handle complex data. Visual stimulus signals may contain a variety of complex and subtle spatiotemporal patterns that may have a significant impact on neural activity. The self-attention mechanism enables the model to better understand these patterns and their importance in the entire dataset, thereby improving the overall accuracy of ERG signal prediction.

[0146] In step S640 of some embodiments, a second multilayer perceptron is responsible for converting the one-dimensional feature vectors (stimulus paradigm attention features) extracted by the second three-dimensional convolutional network into predicted ERG signals, thereby obtaining a first predicted retinal signal corresponding to each stimulation paradigm attention feature. The multi-layer structure of the second multilayer perceptron enables the model to capture and integrate complex nonlinear relationships, thereby generating accurate neural activity data.

[0147] Through the above steps S610 to S640, the dynamic characteristics of the stimulation signal can be accurately captured, and the neural activity can be reconstructed based on the dynamic characteristics to accurately establish the mapping from visual stimulation to bioelectric signals.

[0148] See also Figure 7 In some embodiments, step S540 may include but is not limited to steps S710 to S750:

[0149] Step S710 , determining a first sub-loss according to the local stimulation paradigm, the global stimulation paradigm, the flicker stimulation paradigm, the color stimulation paradigm, the shape stimulation paradigm, the video stimulation paradigm, and each first predicted stimulation paradigm;

[0150] Step S720, determining a second sub-loss according to each retinal electrical imaging signal and each first predicted retinal signal;

[0151] Step S730 , determining a third sub-loss according to the local stimulation paradigm, the global stimulation paradigm, the flicker stimulation paradigm, the color stimulation paradigm, the shape stimulation paradigm, the video stimulation paradigm, and each second predicted stimulation paradigm;

[0152] Step S740 , determining a fourth sub-loss according to each retinal electrical imaging signal and each second predicted retinal signal;

[0153] Step S750 : summing the first sub-loss, the second sub-loss, the third sub-loss, and the fourth sub-loss to obtain a third loss.

[0154] In step S710 of some embodiments, the mean square error function measures the average square difference between the model prediction value and the true value. The mean square error function is applicable to situations where the prediction error needs to be minimized, and it imposes a greater penalty for larger errors, which makes the model more inclined to avoid large prediction errors. Based on the mean square error function (MSE), the loss calculation is performed on the local stimulation paradigm and the first predicted stimulation paradigm corresponding to the local stimulation paradigm, the loss calculation is performed on the global stimulation paradigm and the first predicted stimulation paradigm corresponding to the global stimulation paradigm, the loss calculation is performed on the flicker stimulation paradigm and the first predicted stimulation paradigm corresponding to the flicker stimulation paradigm, the loss calculation is performed on the color stimulation paradigm and the first predicted stimulation paradigm corresponding to the color stimulation paradigm, the loss calculation is performed on the shape stimulation paradigm and the first predicted stimulation paradigm corresponding to the shape stimulation paradigm, and the loss calculation is performed on the video stimulation paradigm and the first predicted stimulation paradigm corresponding to the video stimulation paradigm. The losses obtained by these loss calculations are summed to obtain the first sub-loss. The mean square error function is expressed as:

[0155]

[0156] Where N represents the number of samples; is the true label of the i-th sample (real stimulation paradigm or real ERG signal); is the predicted result of the i-th sample (predicted stimulation paradigm or predicted retinal signal).

[0157] If the training goal is to enable the model to accurately reconstruct the stimulus signal based on the input ERG signal, the mean squared error function can be used to measure the pixel-level difference between the generated image and the target image. By minimizing this loss, the reconstructed image learned by the generator tends to be consistent with the real image in terms of visual structure and details.

[0158] In step S720 of some embodiments, based on the mean square error function, loss calculation is performed on the retinal electrical imaging signal of the local stimulation paradigm and the first predicted retinal signal corresponding to the local stimulation paradigm, loss calculation is performed on the retinal electrical imaging signal of the global stimulation paradigm and the first predicted retinal signal corresponding to the global stimulation paradigm, loss calculation is performed on the retinal electrical imaging signal of the flicker stimulation paradigm and the first predicted retinal signal corresponding to the flicker stimulation paradigm, loss calculation is performed on the retinal electrical imaging signal of the color stimulation paradigm and the first predicted retinal signal corresponding to the color stimulation paradigm, loss calculation is performed on the retinal electrical imaging signal of the shape stimulation paradigm and the first predicted retinal signal corresponding to the shape stimulation paradigm, loss calculation is performed on the retinal electrical imaging signal of the video stimulation paradigm and the first predicted retinal signal corresponding to the video stimulation paradigm, and the losses obtained by these loss calculations are summed to obtain a second sub-loss.

[0159] In step S730 of some embodiments, referring to step S710, based on the mean square error function, loss calculation is performed on the six real stimulation paradigms, namely the local stimulation paradigm, the global stimulation paradigm, the flicker stimulation paradigm, the color stimulation paradigm, the shape stimulation paradigm, and the video stimulation paradigm, respectively, and the second predicted stimulation paradigm that matches the real stimulation paradigm, and the results of the six loss calculations are summed to obtain a third sub-loss.

[0160] In step S740 of some embodiments, referring to step S720, based on the mean square error function, a loss calculation is performed on the retinal electrical imaging signal of the real stimulation paradigm and the second predicted retinal signal matching the real stimulation paradigm, and the loss calculation results obtained from the six real stimulation paradigms are summed to obtain a fourth sub-loss.

[0161] In some embodiments, in step S750, the first sub-loss, the second sub-loss, the third sub-loss, and the fourth sub-loss are summed to obtain a third loss. Alternatively, a first loss weight for the first sub-loss, a second loss weight for the second sub-loss, a third loss weight for the third sub-loss, and a fourth loss weight for the fourth sub-loss are determined. The first sub-loss is multiplied by the first loss weight, the second sub-loss is multiplied by the second loss weight, the third sub-loss is multiplied by the third loss weight, and the fourth sub-loss is multiplied by the fourth loss weight. The four multiplication results are added to obtain the third loss.

[0162] Through the above steps S710 to S750, a third loss can be obtained to reduce the difference between the real ERG signal and the predicted ERG signal, and the real stimulation paradigm and the predicted stimulation paradigm based on the third loss, so as to accurately simulate the encoding, transmission and conversion process of the visual system for the visual stimulation signal.

[0163] When training a generative adversarial model, the generator and discriminator will be updated alternately, and the Adam optimizer can be used for model training. Root Mean Square Propagation (RMSprop) is an adaptive learning rate method designed specifically for solving non-convex optimization problems in deep learning networks. RMSprop adjusts the learning rate of each parameter by maintaining a moving average, making the adjustment of the learning rate more detailed and stable. For deep learning models, RMSprop can provide a stable and effective parameter update strategy. This application uses RMSprop as the optimizer and adopts an alternating training strategy for model training.

[0164] See also Figure 8 In some embodiments, step S160 may include but is not limited to steps S810 to S840:

[0165] Step S810, obtaining the parameter gradient, initial learning rate, and historical gradient moving average of the model parameters of the discriminator;

[0166] Step S820, adjusting the historical gradient moving average according to the parameter gradient and the preset decay rate to obtain the current gradient moving average;

[0167] Step S830, adjusting the initial learning rate according to the current gradient moving average to obtain a target learning rate;

[0168] Step S840: Based on the target learning rate and the parameter gradient, the model parameters of the discriminator are updated according to the first loss and the second loss.

[0169] In step S810 of some embodiments, the model parameters of the first generator are fixed so that the discriminator optimizes its ability to identify true and false images under the current generator's capabilities. The parameter gradient of the model parameters of the discriminator at the current time, the initial learning rate, and the historical gradient moving average of the model parameters at the historical time are obtained. The parameter gradient is the partial derivative of the discriminator's loss function with respect to the model parameters. The historical time is the time before the current time. The historical gradient moving average is obtained by weighted averaging all parameter gradients at the historical time according to the exponential sliding average method. The initial learning rate can be a relatively small value, such as 0.00001 or 0.0002. A smaller learning rate can slow down the learning process and prevent the model from adapting to the training data too quickly, thereby reducing the risk of overfitting. In complex models, a smaller learning rate helps stabilize the training process, especially in structures such as self-attention and residual networks. For accurate prediction of microelectrode array data, the model needs to be able to perform detailed parameter adjustments, and a smaller learning rate is just right for this need.

[0170] In step S820 of some embodiments, according to the gradient update rule, the historical gradient moving average is adjusted according to the parameter gradient and the preset decay rate to obtain the current gradient moving average. The current gradient moving average is as follows:

[0171]

[0172] Among them, g t is the parameter gradient at time t, v t is the current gradient moving average, β is the preset decay rate, the preset decay rate is greater than 0.5 and less than 0.99, v t-1 is the moving average of historical gradients.

[0173] In step S830 of some embodiments, the current gradient moving average is added to a small constant added for data stability, and a square root operation is performed on the result of the addition, and the ratio of the initial learning rate to the square root result is used as the target learning rate.

[0174] In step S840 of some embodiments, the first loss and the second loss are summed to obtain a target loss. Minimizing the negative of the target loss, i.e., maximizing the target loss, updates the model parameters of the discriminator according to the target learning rate and the parameter gradient. The parameter update rules are as follows:

[0175]

[0176] Where θ is the model parameter, η is the initial learning rate, and ∈ is a small constant added for numerical stability.

[0177] In the above steps S810 to S840, the model parameters of the discriminator are updated through the parameter update rule of the RMSprop optimizer, which helps the discriminator to converge quickly in complex gradient scenarios and prevents the learning rate from falling into a local minimum or saddle point during training, thereby improving the discriminator's ability to judge the stimulus paradigm.

[0178] In step S170 of some embodiments, the present application combines the generation adversarial loss and the mean square error loss to optimize the generator so that the generator generates a predicted stimulus paradigm close to the real stimulus paradigm. The model parameters of the discriminator are fixed. For the first generator, it is necessary to minimize the The second loss is determined, and the second loss and the third loss are summed or weighted to obtain the target loss. The target loss is minimized to update the model parameters of the first generator.

[0179] In step S180 of some embodiments, the first generator and the discriminator are updated alternately until an equilibrium point is reached, at which point the generator can produce high-quality images and the discriminator can accurately identify true and false images. It can be determined that the first generator has converged, and the converged first generator is used as the target generator. The condition for the first generator to converge can be that the training rounds reach an iteration threshold, or that the target loss of the generator remains unchanged for a preset number of consecutive times. The iteration threshold can be 400.

[0180] It should be noted that in the early stages of training, the first generator may produce poor images, but as training progresses, it will gradually learn how to mimic real ERG signals. Because generative adversarial models are prone to mode collapse, that is, the generator produces very similar or identical outputs and cannot continue to learn sample diversity, generative adversarial models may also have excessive discriminative ability, causing the generator to reach a local optimum in the early stages of training and generate poor image quality. For these problems, consider adjusting the learning rate of the generator separately and optimizing the generator multiple times before optimizing the discriminator once to ensure the ability and quality of the generator to generate images.

[0181] The second generator can be trained according to the following steps: a target discriminator is designed to match the second generator. The target discriminator can have the same network structure as the discriminator of the first generator. According to step S720, the second generator sequentially reconstructs retinal signals for the local stimulation paradigm, the global stimulation paradigm, the flicker stimulation paradigm, the color stimulation paradigm, the shape stimulation paradigm, and the video stimulation paradigm to obtain multiple first predicted retinal signals. A second sub-loss is determined based on each retinal electrical imaging signal and each first predicted retinal signal. The target discriminator sequentially discriminates each retinal electrical imaging signal under the local stimulation paradigm, the global stimulation paradigm, the flicker stimulation paradigm, the color stimulation paradigm, the shape stimulation paradigm, and the video stimulation paradigm to obtain a first discrimination probability for each retinal electrical imaging signal. For each retinal electrical imaging signal, the expected log-likelihood of the first discrimination probability is calculated to obtain a first discrimination loss. The first discrimination losses of each retinal electrical imaging signal are summed to obtain a first target discrimination loss. The target discriminator discriminates each first predicted retinal signal to obtain a second discrimination probability for each first predicted retinal signal. For each first predicted retinal signal, the log-likelihood of one minus the second discriminant probability is used, and the expectation of the output result of the subtraction operation is calculated to obtain the second discriminant loss. The second discriminant losses of each first predicted retinal signal are summed to obtain the second target discriminant loss. The model parameters of the second generator are fixed, the first target discriminant loss and the second target discriminant loss are summed to obtain the target discriminant loss, and the target discriminant loss is maximized to adjust the model parameters of the target discriminator. The model parameters of the target discriminator are fixed, the second sub-loss and the second target discriminant loss are summed to obtain the target generation loss, and the target generation loss is minimized to adjust the model parameters of the second generator. The second generator and the target discriminator are alternately trained until the second generator converges to obtain the trained second generator.

[0182] In step S190 of some embodiments, the target retinal electrical imaging signal is a retinal electrical imaging signal under a target stimulation paradigm, and the target stimulation paradigm is a time-series image sequence under a real scene. The target retinal electrical imaging signal is obtained, and the stimulation paradigm is reconstructed on the target retinal electrical imaging signal by a target generator. Alternatively, the target stimulation paradigm is obtained, and the ERG signal is reconstructed on the target stimulation paradigm by a second generator to obtain a predicted ERG signal, and the predicted ERG signal is reconstructed on the stimulation paradigm by the target generator to simulate the encoding process of converting visual information into bioelectric signals and converting bioelectric signals into visual information.

[0183] During the training phase, the data set is divided into a training set and a test set. The training set is used to train the model, and the test set is used to verify the model effect. The training set is used to train the first generator for 400 iterations to obtain the target generator. The model parameters of the fixed target generator are used to verify the reconstruction effect of the stimulation paradigm. The real ERG signal is input into the target generator, and the target generator outputs the reconstructed timing stimulation paradigm. Visualize the reconstructed timing stimulation paradigm and compare it with its corresponding real stimulation paradigm. The experimental results of electrode plate channel 21 and channel 47 (i.e., the ERG signal data detected by the corresponding retinal cells on the electrode plate) are selected as the display. For channel 21, the reconstruction results under different stimulation paradigms are visualized respectively. The reconstruction of the shape change, color change, and graphic area change stimulation paradigms are as follows. Figure 9a As shown in the figure, the real stimulus paradigm of shape change, color change, and graphic area change is as follows Figure 9b As shown, Figure 9a and Figure 9b By comparison, it can be seen that the target generator accurately reconstructs the shape change part, color change part and shape area change in the stimulation paradigm. Figure 9c This shows the reconstruction of channel 21 for the shape change and color change stimulus paradigms. Figure 9d By comparing the real stimulus paradigms of shape change and color change, we can find that the target generator can well predict the stimulus paradigms of color change and shape change, but it still has defects in reconstructing the area of ​​the graphic area. The black and white grid noise stimulus itself requires a high level of fine-grained reconstruction of the model. Figure 9e The reconstruction of the black and white grid noise stimulation paradigm is demonstrated. In order to solve the reconstruction defect problem of the graphic area, more signal features can be extracted in the data processing link, and the signal feature quantities can be used to further help the model improve the image generation effect. In addition, it is also possible to consider designing a new loss function so that the model can take the area reconstruction of the graphic area into consideration during the iterative optimization process to improve the model performance. The initial success of the target generator is not only reflected in the accurate reconstruction of the stimulation paradigm, but also in the effective integration of the receptive field information into the stimulation paradigm. By analyzing the reconstructed stimulation paradigm and receptive field data, we can have a deeper understanding of the visual information processing mechanism. In order to prove that the receptive field information is accurately predicted by the model and applied in the reconstruction process, the results of channel 47 are selected for display. The reconstruction results are shown as follows. Figure 9f As shown in Figure 9g The results show that the model performs better than channel 21 in reconstructing the stimulation paradigm of channel 47. While accurately reconstructing the image information of the stimulation paradigm, it also accurately applies the receptive field and position information of channel 47 in the reconstruction process.

[0184] The second generator is used to reconstruct the ERG signal of the reconstructed stimulation paradigm. Figure 10aThe real ERG signal of channel 21 and the real stimulation paradigm are shown. Figure 10b The real ERG signal of channel 57 and the real stimulation paradigm are shown. The real stimulation paradigm includes local stimulation, global stimulation, shape stimulation and color change stimulation. The target generator reconstructs the stimulation paradigm of the real ERG signals of channel 21 and channel 57 respectively, and the reconstructed stimulation paradigms of channel 21 and channel 57 are shown in 10c and 10d respectively. The second generator reconstructs the ERG signals of the reconstructed stimulation paradigms of channel 21 and channel 57 respectively, and obtains the reconstructed ERG signals. The reconstructed ERG signals of channel 21 and channel 57 are shown in 10e and 10f respectively. Among them, blue represents the real ERG signal and red represents the reconstructed ERG signal. Compared with the real ERG signal, the reconstructed ERG signal is very different from the real ERG signal. The second generator can accurately convert the stimulation paradigm into an ERG signal.

[0185] See also Figure 11 The present application also provides a mouse retinal signal reconstruction device that can implement the above-mentioned mouse retinal signal reconstruction method. The mouse retinal signal reconstruction device includes:

[0186] The first acquisition module 1110 is used to obtain each retinal electrical imaging signal under the local stimulation paradigm, the global stimulation paradigm, the flicker stimulation paradigm, the color stimulation paradigm, the shape stimulation paradigm and the video stimulation paradigm; the reconstruction module 1120 is used to reconstruct the first stimulation paradigm of each retinal electrical imaging signal in turn through the first generator to obtain the first predicted stimulation paradigm of each retinal electrical imaging signal; the first generator is composed of a first multi-layer perceptron and a first three-dimensional convolutional neural network connected in series; the first discrimination module 1130 is used to perform image discrimination on the local stimulation paradigm, the global stimulation paradigm, the flicker stimulation paradigm, the color stimulation paradigm, the shape stimulation paradigm and the video stimulation paradigm in turn through the discriminator to obtain multiple first target stimulation paradigm convolution features, and construct a first loss based on the multiple first target stimulation paradigm convolution features; the first loss is the sum of the losses of the local stimulation paradigm, the global stimulation paradigm, the flicker stimulation paradigm, the color stimulation paradigm, the shape stimulation paradigm and the video stimulation paradigm; the second discrimination module 1140 is used to reconstruct each first predicted stimulation paradigm in turn through the discriminator Perform image discrimination to obtain multiple second target stimulus paradigm convolution features, and construct a second loss based on the multiple second target stimulus paradigm convolution features; the second loss is the sum of the losses of all first predicted stimulus paradigms; a calculation module 1150 is used to determine the third loss based on the local stimulation paradigm, the global stimulation paradigm, the flickering stimulation paradigm, the color stimulation paradigm, the shape stimulation paradigm, the video stimulation paradigm, each retinal electrical imaging signal and each first predicted stimulus paradigm; a first update module 1160 is used to fix the model parameters of the first generator and update the model parameters of the discriminator according to the first loss and the second loss; a second update module 1170 is used to fix the model parameters of the discriminator and update the model parameters of the first generator according to the second loss and the third loss; a third update module 1180 is used to alternately update the first generator and the discriminator until the first generator converges to obtain a target generator; a second acquisition module 1190 is used to acquire the target retinal electrical imaging signal and perform stimulus paradigm reconstruction on the target retinal electrical imaging signal through the target generator.

[0187] The specific implementation of the mouse retinal signal reconstruction device is basically the same as the specific embodiment of the above-mentioned mouse retinal signal reconstruction method, and will not be repeated here.

[0188] The present application also provides an electronic device comprising a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the above-described mouse retinal signal reconstruction method. The electronic device can be any intelligent terminal, such as a tablet computer or an in-vehicle computer.

[0189] See also Figure 12 , Figure 12 The hardware structure of an electronic device according to another embodiment is shown. The electronic device includes:

[0190] The processor 1210 can be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application; the memory 1220 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 1220 can store an operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program codes are stored in the memory 1220 and are called by the processor 1210 to execute the mouse retinal signal reconstruction method of the embodiments of the present application; the input / output interface 1230 is used to realize information input and output; the communication interface 1240 is used to realize communication interaction between this device and other devices, and communication can be realized through wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.); the bus 1250 transmits information between the various components of the device (such as the processor 1210, memory 1220, input / output interface 1230 and communication interface 1240); wherein the processor 1210, memory 1220, input / output interface 1230 and communication interface 1240 realize communication connection with each other within the device through the bus 1250.

[0191] An embodiment of the present application further provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the above-mentioned mouse retinal signal reconstruction method is implemented.

[0192] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely located relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0193] The embodiments described in the embodiments of this application are intended to more clearly illustrate the technical solutions of the embodiments of this application and do not constitute a limitation on the technical solutions provided by the embodiments of this application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of this application are also applicable to similar technical problems.

Claims

1. A method for reconstructing mouse retinal signals, characterized in that: The method comprises: Acquire various retinal electrical imaging signals under local stimulation paradigm, global stimulation paradigm, flicker stimulation paradigm, color stimulation paradigm, shape stimulation paradigm and video stimulation paradigm; Reconstructing a first stimulation paradigm for each retinal electrical imaging signal in sequence using a first generator to obtain a first predicted stimulation paradigm for each retinal electrical imaging signal; the first generator is composed of a first multi-layer perceptron and a first three-dimensional convolutional neural network connected in series; the first three-dimensional convolutional neural network includes five first three-dimensional convolutional layers; The discriminator sequentially performs image discrimination on the local stimulation paradigm, the global stimulation paradigm, the flicker stimulation paradigm, the color stimulation paradigm, the shape stimulation paradigm, and the video stimulation paradigm to obtain a plurality of first target stimulation paradigm convolution features, and constructs a first loss based on the plurality of first target stimulation paradigm convolution features; the first loss is the sum of the losses of the local stimulation paradigm, the global stimulation paradigm, the flicker stimulation paradigm, the color stimulation paradigm, the shape stimulation paradigm, and the video stimulation paradigm; the discriminator includes four second three-dimensional convolution layers; Performing image discrimination on each of the first predicted stimulus paradigms in sequence by the discriminator to obtain a plurality of second target stimulus paradigm convolution features, and constructing a second loss based on the plurality of second target stimulus paradigm convolution features; the second loss is the sum of the losses of all the first predicted stimulus paradigms; determining a third loss based on the local stimulation paradigm, the global stimulation paradigm, the flicker stimulation paradigm, the color stimulation paradigm, the shape stimulation paradigm, the video stimulation paradigm, each of the retinal electrographic signals, and each of the first predicted stimulation paradigms; Fixing the model parameters of the first generator, and updating the model parameters of the discriminator according to the first loss and the second loss; Fixing the model parameters of the discriminator, and updating the model parameters of the first generator according to the second loss and the third loss; Alternatingly updating the first generator and the discriminator until the first generator converges to obtain a target generator; A target retinal electrical imaging signal is acquired, and a stimulation paradigm is reconstructed for the target retinal electrical imaging signal by the target generator.

2. The mouse retinal signal reconstruction method according to claim 1, characterized in that The determining of the third loss according to the local stimulation paradigm, the global stimulation paradigm, the flicker stimulation paradigm, the color stimulation paradigm, the shape stimulation paradigm, the video stimulation paradigm, each of the retinal electroimaging signals, and each of the first predicted stimulation paradigms comprises: Retinal signals are reconstructed in sequence on the local stimulation paradigm, the global stimulation paradigm, the flicker stimulation paradigm, the color stimulation paradigm, the shape stimulation paradigm, and the video stimulation paradigm by a second generator to obtain a plurality of first predicted retinal signals; the second generator is composed of a second three-dimensional convolutional neural network and a second multi-layer perceptron connected in series; the second three-dimensional convolutional neural network includes a third three-dimensional convolutional layer, a residual layer, and a self-attention layer; Reconstructing the stimulation paradigm of each of the first predicted retinal signals by the first generator to obtain a plurality of second predicted stimulation paradigms; Reconstructing retinal signals for each of the first predicted stimulation paradigms by the second generator to obtain a plurality of second predicted retinal signals; A third loss is determined based on the local stimulation paradigm, the global stimulation paradigm, the flicker stimulation paradigm, the color stimulation paradigm, the shape stimulation paradigm, the video stimulation paradigm, each of the retinal electroimaging signals, each of the first predicted stimulation paradigms, each of the first predicted retinal signals, each of the second predicted stimulation paradigms, and each of the second predicted retinal signals.

3. The mouse retinal signal reconstruction method according to claim 2, characterized in that: Determining a third loss according to the local stimulation paradigm, the global stimulation paradigm, the flicker stimulation paradigm, the color stimulation paradigm, the shape stimulation paradigm, the video stimulation paradigm, each of the retinal electroimaging signals, each of the first predicted stimulation paradigms, each of the first predicted retinal signals, each of the second predicted stimulation paradigms, and each of the second predicted retinal signals comprises: determining a first sub-loss according to the local stimulation paradigm, the global stimulation paradigm, the flicker stimulation paradigm, the color stimulation paradigm, the shape stimulation paradigm, the video stimulation paradigm, and each of the first predicted stimulation paradigms; determining a second sub-loss according to each of the retinal electrical imaging signals and each of the first predicted retinal signals; determining a third sub-loss according to the local stimulation paradigm, the global stimulation paradigm, the flicker stimulation paradigm, the color stimulation paradigm, the shape stimulation paradigm, the video stimulation paradigm, and each of the second predicted stimulation paradigms; determining a fourth sub-loss according to each of the retinal electrographic signals and each of the second predicted retinal signals; The first sub-loss, the second sub-loss, the third sub-loss, and the fourth sub-loss are summed to obtain the third loss.

4. The mouse retinal signal reconstruction method according to claim 2, characterized in that: The second generator sequentially reconstructs retinal signals for the local stimulation paradigm, the global stimulation paradigm, the flicker stimulation paradigm, the color stimulation paradigm, the shape stimulation paradigm, and the video stimulation paradigm to obtain a plurality of first predicted retinal signals, including: performing three-dimensional convolution on the local stimulation paradigm, the global stimulation paradigm, the flicker stimulation paradigm, the color stimulation paradigm, the shape stimulation paradigm, and the video stimulation paradigm in sequence through the third three-dimensional convolution layer to obtain a plurality of fourth stimulation paradigm convolution features; Extracting features of each fourth stimulus paradigm convolution feature through the residual layer to obtain a plurality of stimulus paradigm residual features; Performing feature conversion on each of the stimulus paradigm residual features through the self-attention layer to obtain a query vector, a key vector, and a value vector, determining an attention score based on the query vector and the key vector, and performing weighted calculation on the value vector based on the attention score to obtain multiple stimulus paradigm attention features; The second multi-layer perceptron performs feature conversion on each of the stimulation paradigm attention features to obtain a plurality of the first predicted retinal signals.

5. The mouse retinal signal reconstruction method according to any one of claims 1 to 4, characterized in that: The first generator sequentially reconstructs the first stimulation paradigm for each retinal electrical imaging signal to obtain the first predicted stimulation paradigm for each retinal electrical imaging signal, including: Performing feature extraction on each of the retinal electrical imaging signals by using the first multi-layer perceptron to obtain a retinal signal feature of each of the retinal electrical imaging signals; performing a first three-dimensional convolution on each of the retinal signal features through a first first three-dimensional convolution layer to obtain a plurality of first retinal convolution features; the first first three-dimensional convolution layer has an input channel number of 100, an output channel number of 256, a convolution kernel size of 4, a sliding step size of 1, and a padding value of 0; performing a second three-dimensional convolution on each of the first retinal convolution features through a second first three-dimensional convolution layer to obtain multiple second retinal convolution features; the second first three-dimensional convolution layer has an input channel number of 256, an output channel number of 128, a convolution kernel size of 4, a sliding step size of 2, and a padding value of 1; Performing a third three-dimensional convolution on each of the second retinal convolution features through a third first three-dimensional convolution layer to obtain multiple third retinal convolution features; the convolution kernel size includes a convolution width, a convolution height, and a convolution depth, and the sliding step includes a first sliding sub-step in the convolution width direction, a second sliding sub-step in the convolution height direction, and a third sliding sub-step in the convolution depth direction; the third first three-dimensional convolution layer has 128 input channels, 64 output channels, 1 convolution width, 4 convolution height, 4 convolution depth, a first sliding sub-step, 1 second sliding sub-step, 2 third sliding sub-step, and a padding value of 1; performing a fourth three-dimensional convolution on each of the third retinal convolution features through a fourth first three-dimensional convolution layer to obtain a plurality of fourth retinal convolution features; the fourth first three-dimensional convolution layer has 64 input channels, 32 output channels, a convolution width of 1, a convolution height of 4, a convolution depth of 4, a first sliding sub-step of 1, a second sliding sub-step of 2, a third sliding sub-step of 2, and a padding value of 1; A fifth three-dimensional convolution is performed on each of the fourth retinal convolution features through the fifth first three-dimensional convolution layer to obtain the first predicted stimulation paradigm for each retinal electrical imaging signal; the number of input channels of the fifth first three-dimensional convolution layer is 32, the number of output channels is 160, the convolution width is 2, the convolution height is 4, the convolution depth is 4, the first sliding sub-step is 1, the second sliding sub-step is 2, the third sliding sub-step is 2, and the padding value is 1.

6. The mouse retinal signal reconstruction method according to any one of claims 1 to 4, characterized in that: The discriminator sequentially performs image discrimination on the local stimulation paradigm, the global stimulation paradigm, the flicker stimulation paradigm, the color stimulation paradigm, the shape stimulation paradigm, and the video stimulation paradigm to obtain a plurality of first target stimulation paradigm convolution features, and constructs a first loss according to the plurality of first target stimulation paradigm convolution features, including: performing a first three-dimensional convolution on the local stimulation paradigm, the global stimulation paradigm, the flicker stimulation paradigm, the color stimulation paradigm, the shape stimulation paradigm, and the video stimulation paradigm in sequence through the first second three-dimensional convolution layer to obtain a plurality of first stimulation paradigm convolution features; the convolution kernel size includes convolution width, convolution height, and convolution depth; the first second three-dimensional convolution layer has 160 input channels, 128 output channels, 1 convolution width, 4 convolution height, 4 convolution depth, 2 sliding steps, and 1 padding value; performing a second three-dimensional convolution on each of the first stimulus paradigm convolution features through a second second three-dimensional convolution layer to obtain a plurality of second stimulus paradigm convolution features; the second second three-dimensional convolution layer has an input channel number of 128, an output channel number of 256, a convolution width of 1, a convolution height of 4, a convolution depth of 4, a sliding step size of 2, and a padding value of 1; performing a third three-dimensional convolution on each of the second stimulus paradigm convolution features through a third second three-dimensional convolution layer to obtain multiple third stimulus paradigm convolution features; the third second three-dimensional convolution layer has 256 input channels, 512 output channels, 1 convolution width, 4 convolution height, 4 convolution depth, 2 sliding steps, and 1 padding value; performing a fourth three-dimensional convolution on each of the third stimulus paradigm convolution features through a fourth second three-dimensional convolution layer to obtain a plurality of first target stimulus paradigm convolution features; the fourth second three-dimensional convolution layer has 512 input channels, 160 output channels, a convolution width of 2, a convolution height of 8, a convolution depth of 8, a sliding step of 1, and a padding value of 0; Calculating the expected log-likelihood of each convolution feature of the first target stimulation paradigm to obtain the loss of each convolution feature of the first target stimulation paradigm; The losses of each of the first target stimulus paradigm convolutional features are summed to obtain the first loss.

7. The mouse retinal signal reconstruction method according to any one of claims 1 to 4, characterized in that: The updating of the model parameters of the discriminator according to the first loss and the second loss includes: Obtaining the parameter gradient, initial learning rate, and historical gradient moving average of the model parameters of the discriminator; Adjusting the historical gradient moving average according to the parameter gradient and a preset decay rate to obtain a current gradient moving average; Adjusting the initial learning rate according to the current gradient moving average to obtain a target learning rate; Based on the target learning rate and the parameter gradient, the model parameters of the discriminator are updated according to the first loss and the second loss.

8. Mouse retinal signal reconstruction device, characterized in that The device comprises: The first acquisition module is used to acquire various retinal electrical imaging signals under the local stimulation paradigm, the global stimulation paradigm, the flicker stimulation paradigm, the color stimulation paradigm, the shape stimulation paradigm and the video stimulation paradigm; a reconstruction module, configured to sequentially reconstruct a first stimulation paradigm for each retinal electrical imaging signal using a first generator, to obtain a first predicted stimulation paradigm for each retinal electrical imaging signal; the first generator being composed of a first multilayer perceptron and a first three-dimensional convolutional neural network connected in series; a first discrimination module, configured to perform image discrimination on the local stimulation paradigm, the global stimulation paradigm, the flicker stimulation paradigm, the color stimulation paradigm, the shape stimulation paradigm, and the video stimulation paradigm in sequence through a discriminator to obtain a plurality of first target stimulation paradigm convolution features, and construct a first loss based on the plurality of first target stimulation paradigm convolution features; the first loss is the sum of the losses of the local stimulation paradigm, the global stimulation paradigm, the flicker stimulation paradigm, the color stimulation paradigm, the shape stimulation paradigm, and the video stimulation paradigm; a second discrimination module, configured to perform image discrimination on each of the first predicted stimulation paradigms in sequence through the discriminator to obtain a plurality of second target stimulation paradigm convolution features, and construct a second loss based on the plurality of second target stimulation paradigm convolution features; the second loss being the sum of the losses of all the first predicted stimulation paradigms; a calculation module, configured to determine a third loss based on the local stimulation paradigm, the global stimulation paradigm, the flicker stimulation paradigm, the color stimulation paradigm, the shape stimulation paradigm, the video stimulation paradigm, each of the retinal electroimaging signals, and each of the first predicted stimulation paradigms; A first updating module, configured to fix the model parameters of the first generator and update the model parameters of the discriminator according to the first loss and the second loss; A second updating module is used to fix the model parameters of the discriminator and update the model parameters of the first generator according to the second loss and the third loss; A third updating module is used to alternately update the first generator and the discriminator until the first generator converges to obtain a target generator; The second acquisition module is used to acquire the target retinal electrical imaging signal and reconstruct the stimulation paradigm of the target retinal electrical imaging signal through the target generator.

9. An electronic device, characterized in that The electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the mouse retinal signal reconstruction method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the mouse retinal signal reconstruction method according to any one of claims 1 to 7 is implemented.

Images