Multimodal retinal signal reconstruction method and device based on long-range attention mechanism

Abstract

The application provides a kind of multi-modal retinal signal reconstruction method and device based on long-distance attention mechanism, belong to image processing technical field, by first long short-term memory submodel to the first output state of last time step, the second output state of last time step and target retinal electric imaging signal are first feature coding to obtain first output state, and long-distance attention is determined according to the first output state, by second long short-term memory submodel to the second output state of last time step, first output state and long-distance attention are second feature coding to obtain initial retinal signal feature, by signal alignment module, initial retinal signal feature is signal aligned to obtain target alignment signal feature, signal alignment module is based on sample stimulus paradigm feature and sample retinal electric imaging signal feature is trained to obtain, by target generator, stimulus paradigm is generated to target alignment signal feature, can accurately simulate the encoding mode of visual stimulation information.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, in particular to a multi-modal retinal signal reconstruction method and device based on a long-distance attention mechanism. BACKGROUND

[0002] In the related art, the complete visual pathway from the eye to the brain region can be clearly understood through biological and anatomical techniques, but the encoding mode of visual information in the visual pathway cannot be obtained. Studying how visual information is encoded in the visual pathway helps the development of medicine, bioengineering, computer vision, and robotics. SUMMARY

[0003] The main purpose of the embodiments of the present application is to propose a multi-modal retinal signal reconstruction method and device based on a long-distance attention mechanism, aiming to accurately simulate the encoding process of visual information in the visual pathway and understand the response mode of the visual pathway to visual stimulation.

[0004] To achieve the above purpose, a first aspect of the embodiments of the present application proposes a multi-modal retinal signal reconstruction method based on a long-distance attention mechanism, the method comprising:

[0005] obtaining a target retinal electrical imaging signal;

[0006] inputting the target retinal electrical imaging signal into a retinal signal encoder, the retinal signal encoder comprising a first long short-term memory sub-model and a second long short-term memory sub-model;

[0007] for each time step in a plurality of time steps, performing first feature encoding on a first output state output by the first long short-term memory sub-model at a previous time step, a second output state output by the second long short-term memory sub-model at the previous time step, and the target retinal electrical imaging signal by the first long short-term memory sub-model, to obtain a first output state output by the first long short-term memory sub-model, and determining a long-distance attention according to the first output state;

[0008] performing second feature encoding on the second output state at the previous time step, the first output state, and the long-distance attention by the second long short-term memory sub-model to obtain a second output state, and taking the second output state of all the time steps as an initial retinal signal feature;

[0009] performing signal alignment on the initial retinal signal feature by a signal alignment module to obtain a target aligned signal feature; the signal alignment module is trained based on a sample stimulus paradigm feature and a sample retinal electrical imaging signal feature;

[0010] The target stimulus paradigm is generated by a target generator based on the target alignment signal feature.

[0011] In some embodiments, the determining the long-distance attention according to the first output state comprises:

[0012] The retrieval feature vector, the key feature vector and the value feature vector are determined according to the first output state, and the signal relationship feature is determined according to the retrieval feature vector, the key feature vector and the value feature vector.

[0013] The target threshold probability is determined based on a threshold unit, the first output state of the current time step and the signal relationship feature.

[0014] The long-distance attention is determined according to the first output state of the current time step, the signal relationship feature and the target threshold probability.

[0015] In some embodiments, the retinal signal encoder is trained according to the following steps:

[0016] The reference retinal electrical imaging signal sample, the positive retinal electrical imaging signal sample and the negative retinal electrical imaging signal sample are obtained.

[0017] The reference sample retinal signal feature is obtained by performing retinal signal encoding on the reference retinal electrical imaging signal sample through a preset encoder.

[0018] The positive sample retinal signal feature is obtained by performing retinal signal encoding on the positive retinal electrical imaging signal sample through the preset encoder.

[0019] The negative sample retinal signal feature is obtained by performing retinal signal encoding on the negative retinal electrical imaging signal sample through the preset encoder.

[0020] The first loss is determined according to the reference sample retinal signal feature and the positive sample retinal signal feature, and the second loss is determined according to the reference sample retinal signal feature and the negative sample retinal signal feature.

[0021] The margin distance between the positive sample retinal signal feature and the negative sample retinal signal feature is calculated.

[0022] The preset encoder is updated in parameters according to the first loss, the second loss and the margin distance until the first loss is less than the second loss, the second loss is less than the sum of the first loss and the margin distance, and the preset encoder is taken as the retinal signal encoder.

[0023] In some embodiments, the signal alignment module is trained according to the following steps:

[0024] obtain a sample stimulus paradigm and a sample retinal electrical imaging signal;

[0025] encode the sample stimulus paradigm to obtain a sample stimulus paradigm feature;

[0026] encode the sample retinal electrical imaging signal to obtain a sample retinal electrical imaging signal feature;

[0027] encode the sample stimulus paradigm feature to obtain an encoded stimulus paradigm feature through a preset alignment module;

[0028] encode the sample retinal electrical imaging signal feature to obtain an encoded retinal signal feature through the preset alignment module;

[0029] calculate a similarity score between the encoded stimulus paradigm feature and the encoded retinal signal feature;

[0030] determine a third loss based on a preset loss function according to the similarity score and the encoded stimulus paradigm feature;

[0031] update the preset alignment module according to the third loss to obtain the signal alignment module.

[0032] In some embodiments, the target generator comprises a mapping sub-model and a generation sub-model, the generation sub-model comprising a plurality of resolution units with increasing resolutions in sequence, and the encoding the target alignment signal feature to obtain a target stimulus paradigm through the target generator comprises:

[0033] randomly generate a noise signal;

[0034] map the target alignment signal feature and the noise signal to obtain a latent space distribution feature through the mapping sub-model;

[0035] perform affine transformation on the latent space distribution feature to obtain a style vector;

[0036] determine an output feature output by a current resolution unit based on an output feature output by a previous resolution unit, the style vector, and a Gaussian noise component generated for the current resolution unit, based on the current resolution unit;

[0037] take the output feature output by the last resolution unit as the target stimulus paradigm.

[0038] In some embodiments, the target generator is trained according to the following steps:

[0039] obtaining a sample stimulation paradigm and a sample retinal electrical imaging signal;

[0040] generating a first predicted stimulation paradigm by a preset generator based on the sample stimulation paradigm;

[0041] generating a second predicted stimulation paradigm by the preset generator based on the sample retinal electrical imaging signal;

[0042] updating parameters of a preset discriminator based on the sample stimulation paradigm, the first predicted stimulation paradigm and the second predicted stimulation paradigm;

[0043] determining a fourth loss based on a first discrimination feature obtained by discriminating the first predicted stimulation paradigm by the updated preset discriminator;

[0044] determining a fifth loss based on a second discrimination feature obtained by discriminating the second predicted stimulation paradigm by the preset discriminator;

[0045] updating parameters of the preset generator based on the fourth loss and the fifth loss, to obtain the target generator.

[0046] In some embodiments, the updating parameters of the preset discriminator based on the sample stimulation paradigm, the first predicted stimulation paradigm and the second predicted stimulation paradigm comprises:

[0047] determining a sixth loss based on a third discrimination feature obtained by discriminating the sample stimulation paradigm by the preset discriminator;

[0048] determining a seventh loss based on a fourth discrimination feature obtained by discriminating the first predicted stimulation paradigm by the preset discriminator;

[0049] determining an eighth loss based on a fifth discrimination feature obtained by discriminating the second predicted stimulation paradigm by the preset discriminator;

[0050] updating parameters of the preset discriminator based on the sixth loss, the seventh loss and the eighth loss.

[0051] To achieve the above object, a second aspect of the embodiment of the present application proposes a multi-modal retinal signal reconstruction device based on a long-distance attention mechanism, which comprises:

[0052] an obtaining module configured to obtain a target retinal electrical imaging signal;

[0053] The input module is configured to input the target retinal electrical imaging signal into a retinal signal encoder, and the retinal signal encoder includes a first long short-term memory sub-model and a second long short-term memory sub-model.

[0054] The first encoding module is configured to, for each time step in a plurality of time steps, perform first feature encoding on a first output state output by the first long short-term memory sub-model at a previous time step, a second output state output by the second long short-term memory sub-model at the previous time step, and the target retinal electrical imaging signal by the first long short-term memory sub-model, to obtain a first output state output by the first long short-term memory sub-model, and determine long-range attention according to the first output state.

[0055] The second encoding module is configured to perform second feature encoding on the second output state at the previous time step, the first output state, and the long-range attention by the second long short-term memory sub-model to obtain a second output state, and take the second output state of all the time steps as initial retinal signal features.

[0056] The alignment module is configured to perform signal alignment on the initial retinal signal features by a signal alignment module to obtain target aligned signal features, and the signal alignment module is trained based on sample stimulus paradigm features and sample retinal electrical imaging signal features.

[0057] The generation module is configured to perform stimulus paradigm generation on the target aligned signal features by a target generator to obtain a target stimulus paradigm.

[0058] To achieve the above object, a third aspect of embodiments of the present application provides an electronic device, which includes a memory and a processor, the memory stores a computer program, and the processor implements the multi-modal retinal signal reconstruction method based on the long-range attention mechanism of the first aspect when executing the computer program.

[0059] To achieve the above object, a fourth aspect of embodiments of the present application provides a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the multi-modal retinal signal reconstruction method based on the long-range attention mechanism of the first aspect.

[0060] The multi-modal retinal signal reconstruction method based on the long-distance attention mechanism, the multi-modal retinal signal reconstruction device based on the long-distance attention mechanism, the electronic device and the computer readable storage medium provided by the application, by acquiring a target retinal electrical imaging signal, inputting the target retinal electrical imaging signal into a retinal signal encoder to extract key features in the target retinal electrical imaging signal for recovering visual information. The target retinal electrical imaging signal is a time sequence. In order to capture the dependency relationship between the data points far apart in the time sequence, the long-distance dependency relationship is accurately extracted through the stacked long short-term memory network. There is an association relationship between the data points at different time steps. In order to accurately extract the feature representation of each time step data point, the first long short-term memory sub-model is used to perform first feature encoding on the first output state of the previous time step, the second output state of the previous time step and the target retinal electrical imaging signal, and the first output state is obtained. In order to enhance the feature representation ability of the model, the long-distance attention mechanism is introduced, the long-distance attention is determined according to the first output state, and the long-distance attention is used to capture the long-distance dependency relationship. In order to extract deep semantic features, the second long short-term memory sub-model is used to perform second feature encoding on the second output state of the previous time step, the first output state and the long-distance attention, and the initial retinal signal feature is obtained. The signal alignment module is used for signal alignment of the initial retinal signal feature, so that the initial retinal signal feature can be aligned with the visual information, so as to accurately extract the key detail information for recovering the visual information by using the multi-modal information, and obtain the target alignment signal feature. The target generator is used for stimulus paradigm generation of the target alignment signal feature, and the target stimulus paradigm is obtained. The mapping from the retinal electrical imaging signal to the visual information is established, and the encoding process of the visual information in the visual pathway can be accurately simulated. BRIEF DESCRIPTION OF DRAWINGS

[0061] Figure 1 is a flowchart of the multi-modal retinal signal reconstruction method based on the long-distance attention mechanism provided by the application embodiment;

[0062] Figure 2 is a flowchart of the training process of the retinal signal encoder provided by the application embodiment;

[0063] Figure 3 is a schematic diagram of the network structure of the retinal signal encoder provided by the application embodiment;

[0064] Figure 4 is Figure 1 the flowchart of step S130 in

[0065] Figure 5 is a flowchart of the training process of the signal alignment module provided by the application embodiment;

[0066] Figure 6 is Figure 1is a flowchart of step S160 in

[0067] Figure 7 is a flowchart of a training process of the target generator provided by the embodiments of the present application;

[0068] Figure 8 is Figure 7 is a flowchart of step S740 in

[0069] Figure 9A is a reconstruction effect diagram from retinal electrical imaging signals to a stimulation paradigm provided by the embodiments of the present application;

[0070] Figure 9B is another reconstruction effect diagram from retinal electrical imaging signals to a stimulation paradigm provided by the embodiments of the present application;

[0071] Figure 9C is a reconstruction effect diagram from a stimulation paradigm to a stimulation paradigm provided by the embodiments of the present application;

[0072] Figure 10 is a structural schematic diagram of a multi-modal retinal signal reconstruction device based on a long-distance attention mechanism provided by the embodiments of the present application;

[0073] Figure 11 is a hardware structural schematic diagram of an electronic device provided by the embodiments of the present application. DETAILED DESCRIPTION

[0074] In order to make the objects, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application, and are not intended to limit the present application.

[0075] It should be noted that although the functional modules are divided in the device schematic diagram, and the logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in a manner different from the module division in the device or the order in the flowchart. The terms "first", "second", etc. in the specification and claims and the above-described drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0076] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the present application belongs. The terms used herein are only for the purpose of describing the embodiments of the present application, and are not intended to limit the present application.

[0077] In the related art, the complete visual pathway from the eye to the brain area of visual information can be clearly recognized by biological and anatomical techniques, but the encoding manner of visual information in the visual pathway cannot be obtained. Studying how visual information is encoded in the visual pathway helps the development of medicine, bioengineering, computer vision, and robot technology.

[0078] Based on this, the embodiment of the application provides a multi-modal retinal signal reconstruction method based on a long-distance attention mechanism, a multi-modal retinal signal reconstruction device based on a long-distance attention mechanism, an electronic device, and a computer readable storage medium, aiming to accurately simulate and predict the encoding manner of visual information and understand the response mode of the retina to visual information.

[0079] The multi-modal retinal signal reconstruction method based on a long-distance attention mechanism provided by the embodiment of the application, the multi-modal retinal signal reconstruction device based on a long-distance attention mechanism, the electronic device, and the computer readable storage medium are specifically explained through the following embodiments. First, the multi-modal retinal signal reconstruction method based on a long-distance attention mechanism in the embodiment of the application is described.

[0080] The multi-modal retinal signal reconstruction method based on a long-distance attention mechanism provided by the embodiment of the application relates to the technical field of image processing. The multi-modal retinal signal reconstruction method based on a long-distance attention mechanism provided by the embodiment of the application can be applied in a terminal, can be applied in a server end, and can also be software running in a terminal or a server end. In some embodiments, the terminal can be a smart phone, a tablet computer, a notebook computer, a desktop computer, etc.; the server end can be configured as an independent physical server, can be configured as a server cluster or a distributed system composed of multiple physical servers, can also be configured as a cloud server providing basic cloud computing services such as cloud service, cloud database, cloud computing, cloud function, cloud storage, network service, cloud communication, middleware service, domain name service, security service, CDN, and big data and artificial intelligence platform; and the software can be an application for implementing the multi-modal retinal signal reconstruction method based on a long-distance attention mechanism, but is not limited to the above forms.

[0081] The application is operable with numerous general purpose or special purpose computer system environments or configurations. Examples of well-known computing systems, environments, and / or configurations that can be suitable for use with the application include personal computers, server computers, handheld or laptop devices, tablet devices, multiprocessor systems, microprocessor-based systems, set top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments that include any of the above systems or devices, and the like. The application can be described in the general context of computer-executable instructions, such as program modules, being executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, and the like, that perform particular tasks or implement particular abstract data types. The application can also be practiced in distributed computing environments where tasks are performed by remote processing devices that are linked through a communications network. In a distributed computing environment, program modules can be located in local and remote computer storage media including memory storage devices.

[0082] Figure 1 is an optional flowchart of a multi-modal retinal signal reconstruction method based on a long-distance attention mechanism provided by an embodiment of the application, Figure 1 The method in the method can include, but is not limited to, steps S110 to S160.

[0083] Step S110, acquiring a target retinal electrical imaging signal;

[0084] Step S120, inputting the target retinal electrical imaging signal into a retinal signal encoder, the retinal signal encoder including a first long short-term memory sub-model and a second long short-term memory sub-model;

[0085] Step S130, for each time step in a plurality of time steps, performing first feature encoding on a first output state output by the first long short-term memory sub-model at a previous time step, a second output state output by the second long short-term memory sub-model at the previous time step, and the target retinal electrical imaging signal by the first long short-term memory sub-model, to obtain a first output state output by the first long short-term memory sub-model, and determining a long-distance attention according to the first output state.

[0086] Step S140, performing second feature encoding on the second output state at the previous time step, the first output state, and the long-distance attention by the second long short-term memory sub-model to obtain the second output state, and taking the second output state of all time steps as an initial retinal signal feature;

[0087] Step S150, performing signal alignment on the initial retinal signal feature by a signal alignment module to obtain a target aligned signal feature; the signal alignment module is trained based on a sample stimulus paradigm feature and a sample retinal electrical imaging signal feature;

[0088] In step S160, the target generator generates a target stimulation paradigm for the target alignment signal feature.

[0089] In step S110 of some embodiments, a non-invasive multi-electrode array (MEA) combined with microelectronic chip manufacturing technology has the characteristics of high density, high sensitivity, and portability. The MEA technology can be used to record the activity of retinal neurons in real time to collect the electro retinography (ERG) of the mouse retina, thereby providing high-quality data for subsequent retinal signal reconstruction. By attaching the mouse retina tissue to the MEA chip electrode, the electrical activity of any site of the tissue can be recorded and stimulated. This kind of electrical activity data is called electro retinography signal. The target electro retinography signal is the electro retinography signal generated by stimulating the mouse retinal cells with the target stimulation paradigm. The target stimulation paradigm is visual stimulation information. The target stimulation paradigm can use a dynamic image sequence, and the categories of the target stimulation paradigm can be local stimulation, global stimulation, shape stimulation, color stimulation, etc. By converting the visual stimulation signal into neural activity data, it is helpful to study the encoding and processing of visual information in the nervous system.

[0090] In step S120 of some embodiments, in order to study the encoding and processing of visual stimulation information by the retina, it is necessary to establish a mapping between the ERG signal and the stimulation paradigm to promote visual repair, biological bionic eye research, etc. In order to fully extract the ERG signal features, the target electro retinography signal is input into the retinal signal encoder. The retinal signal encoder is constructed by stacking multiple long short-term memory networks (LSTM) in layers. The long short-term memory network has strong generalization ability and can adapt to different data distributions and capture long-term dependencies in sequence data. The number of long short-term memory networks is not limited in the embodiments of the present application, and can be set arbitrarily according to actual conditions, such as being set to 2. The retinal signal encoder includes a cascaded first long short-term memory sub-model and a second long short-term memory sub-model. Both the first long short-term memory sub-model and the second long short-term memory sub-model are LSTM networks.

[0091] Please refer to Figure 2 In some embodiments, the training process of the retinal signal encoder can include, but is not limited to, steps S210 to S270:

[0092] In step S210, reference electro retinography signal samples, positive electro retinography signal samples, and negative electro retinography signal samples are obtained.

[0093] In step S220, the reference retinal electroimaging signal sample is encoded by a preset encoder to obtain reference sample retinal signal features.

[0094] In step S230, the positive retinal electroimaging signal sample is encoded by a preset encoder to obtain positive sample retinal signal features.

[0095] In step S240, the negative retinal electroimaging signal sample is encoded by a preset encoder to obtain negative sample retinal signal features.

[0096] In step S250, a first loss is determined according to the reference sample retinal signal features and the positive sample retinal signal features, and a second loss is determined according to the reference sample retinal signal features and the negative sample retinal signal features.

[0097] In step S260, a margin distance between the positive sample retinal signal features and the negative sample retinal signal features is calculated.

[0098] In step S270, the preset encoder is updated in parameters according to the first loss, the second loss, and the margin distance, until the first loss is less than the second loss, and the second loss is less than the sum of the first loss and the margin distance, and the preset encoder is taken as a retinal signal encoder.

[0099] In step S210 of some embodiments, the reference retinal electroimaging signal sample (reference sample), the positive retinal electroimaging signal sample (positive sample), and the negative retinal electroimaging signal sample (negative sample) are obtained, the reference retinal electroimaging signal sample is a retinal electroimaging signal obtained by stimulating a retinal cell with a stimulation paradigm, the positive retinal electroimaging signal sample is another retinal electroimaging signal obtained by stimulating another retinal cell with the same stimulation paradigm, and the reference retinal electroimaging signal sample and the positive retinal electroimaging signal sample are different ERG signals under the same stimulation paradigm. The negative retinal electroimaging signal sample is a retinal electroimaging signal obtained by stimulating a retinal cell with another stimulation paradigm, i.e., an ERG signal under a stimulation paradigm different from the reference sample and the positive sample.

[0100] In step S220 of some embodiments, the preset encoder has the same network structure as a retinal signal encoder, and the retinal signal encoder is a trained preset encoder. The reference retinal electroimaging signal sample is input to the preset encoder for retinal signal encoding to obtain reference sample retinal signal features. The reference sample retinal signal features are ERG signal features extracted by the preset encoder for the reference sample, which can be a-wave amplitude, b-wave amplitude, oscillatory potential, phase spectrum, power spectrum, etc. The process of retinal signal encoding by the preset encoder can refer to the feature encoding process of the retinal signal encoder on the target retinal electroimaging signal, which is not described here.

[0101] In step S230 of some embodiments, the positive retinal electroretinography signal sample is input to a preset encoder for retinal signal encoding to obtain positive sample retinal signal features. The positive sample retinal signal features are the ERG signal features extracted by the preset encoder for the positive sample.

[0102] In step S240 of some embodiments, the negative electroretinography signal sample is input to a preset encoder for retinal signal encoding to obtain negative sample retinal signal features. The negative sample retinal signal features are the ERG signal features extracted by the preset encoder for the negative sample.

[0103] In step S250 of some embodiments, a semi-hard triplet loss is used as the loss function during preset encoder training. The semi-hard triplet loss attempts to learn a feature space such that the distance between benchmark samples and positive samples of the same class (same stimulus paradigm) is closer, and the distance between benchmark samples and negative samples of different classes (different stimulus paradigms) is farther. The semi-hard triplet loss function is expressed as:

[0104] ||f θ (E a )-f θ (E p )‖<‖f θ (E a )-f θ (E n )‖<‖f θ (E a )-f θ (E p Formula (1)

[0105] Among them, f θ (·) represents the preset encoder, θ represents the model parameters of the preset encoder, and E a It is the baseline sample, E p It is a positive sample, E n δ represents the margin distance between positive and negative samples.

[0106] Based on the semi-hard triplet loss function, the feature error between the retinal signal features of the benchmark sample and the retinal signal features of the positive sample is calculated to obtain the first loss, which is expressed as ||f θ (E a )-f θ (E p The feature error between the baseline sample retinal signal features and the negative sample retinal signal features is calculated to obtain the second loss, which is expressed as ||f. θ (E a )-fθ (E n )||.

[0107] In step S260 of some embodiments, the margin distance between positive sample retinal signal features and negative sample retinal signal features is calculated using a distance metric. The margin distance is the distance between two feature representations in the feature space. The distance metric can be Euclidean distance, Manhattan distance, cosine similarity, etc.

[0108] In step S270 of some embodiments, the model parameters of the preset encoder are updated according to the first loss, the second loss and the margin distance to train the preset encoder until the first loss is less than the second loss and the second loss is less than the sum of the first loss and the margin distance, that is, the preset encoder training is completed, and the preset encoder at this time is used as the retinal signal encoder.

[0109] It should be noted that the distance between the baseline sample and the positive sample is smaller than the distance between the baseline sample and the negative sample, but the difference between these two distances does not satisfy the margin distance condition. The purpose of the semi-hard triplet loss function is to separate positive and negative samples at a certain distance. Through optimization, it ensures that sample points of the same class are close enough, while sample points of different classes are far enough apart. That is, the distance between the baseline sample and the negative sample should be much greater than the distance between the baseline sample and the positive sample.

[0110] Steps S210 to S270 above train the preset encoder so that the feature representations of positive samples and reference samples are closer in the feature space, while the feature representations of reference samples, positive samples and negative samples are farther apart in the feature space. This improves the preset encoder's feature encoding capability for different ERG signals and ensures the accuracy of feature extraction.

[0111] The network structure of the retinal signal encoder is as follows: Figure 3 As shown, E represents the target retinal electroretinography signal. This represents the initial retinal signal features, i.e., the output features extracted by the retinal signal encoder from the target retinal electroretinography signal. This represents the hidden state, i.e., the first output state, of the first long short-term memory sub-model at time step t. This represents the hidden state of the second long short-term memory sub-model at time step t, i.e., the second output state. When the time step is 0, and All are randomly initialized tensors. The number of time steps can be the duration of the target retinal electroretinography (RTG) signal, i.e., the number of data points in the target RTG signal. For each of the multiple time steps, the first output state of the first Long Short-Term Memory (LSTM) submodel at the previous time step is... The second long short-term memory sub-model outputs a second output state at a previous time step The target retinal electric imaging signal is feature spliced with the target retinal electric imaging signal to obtain first spliced features, the first spliced features are first time sequence feature encoded by the first long short-term memory sub-model to obtain a first output state of the first long short-term memory sub-model The calculation process of the first long short-term memory sub-model can be represented as:

[0112]

[0113] The LSTM 1 The first long short-term memory sub-model is represented.

[0114] The time sequence features (the first output state) encoded by the first long short-term memory sub-model are long-distance attention calculated to obtain long-distance attention, the long-distance attention allows the model to capture the dependency between data points far apart in the sequence when processing the sequence data, and the long-distance attention is used to represent the implicit relationship between the data points of the ERG signal.

[0115] Please refer to Figure 4 In some embodiments, step S130 can include but is not limited to steps S410 to S430:

[0116] Step S410, determining a search feature vector, a key feature vector and a value feature vector according to the first output state, and determining a signal relationship feature according to the search feature vector, the key feature vector and the value feature vector;

[0117] Step S420, determining a target threshold probability according to the first output state of the current time step and the signal relationship feature based on a threshold unit;

[0118] Step S430, determining long-distance attention according to the first output state of the current time step, the signal relationship feature and the target threshold probability.

[0119] In step S410 of some embodiments, the first weight matrix is multiplied by the first output state to obtain the search feature vector, the second weight matrix is multiplied by the first output state to obtain the key feature vector, and the third weight matrix is multiplied by the first output state to obtain the value feature vector. The feature dimensions of the search feature vector, the key feature vector and the value feature vector are the same, and the calculation process can be represented as:

[0120]

[0121]

[0122]

[0123] Wherein Q, K, V respectively represent the search feature vector, the key feature vector and the value feature vector, W q , W k , W v respectively represent the first weight matrix, the second weight matrix and the third weight matrix.

[0124] Based on the multi-head self-attention mechanism, the self-attention calculation is performed according to the search feature vector, the key feature vector and the value feature vector, the signal relationship features output by each self-attention head are obtained, and the signal relationship features are spliced. The calculation process of the signal relationship features is as follows:

[0125]

[0126] A = concat (A1, …, A h ) formula (7)

[0127] Wherein A i represents the signal relationship feature output by the i-th self-attention head, T represents the transposition operation; d k represents the feature dimension, and softmax represents the softmax activation function; A represents the spliced feature; h represents the number of self-attention heads; concat represents the splicing operation.

[0128] In step S420 of some embodiments, the target threshold probability is the output feature of the threshold unit, which is used to control the probability of the signal relationship feature A and the first output state at the current time step to pass down. Based on the threshold unit, the first output state at the current time step t and the signal relationship feature A are spliced, the threshold feature extraction is performed on the spliced feature, and the target threshold probability is obtained. The target threshold probability is represented as:

[0129]

[0130] Wherein g t represents the target threshold probability at time step t; W g represents the learnable parameters of the threshold unit; sigmoid represents the activation function.

[0131] In step S430 of some embodiments, the signal relationship feature and the first output state transmitted from the first long short-term memory submodel to the second long short-term memory submodel are controlled according to the target threshold probability. The first output state at the current time step is multiplied by the target threshold probability, the signal relationship feature is multiplied by the probability obtained by subtracting the target threshold probability from 1, and the two multiplication results are added to obtain the long-distance attention. The long-distance attention is used to represent the implicit relationship between different feature channels of the ERG signal, and the long-distance attention is represented as:

[0132] ​

[0133] By the steps S410 to S430, the long-range attention can be obtained to determine the implicit relationship between different feature channels of the ERG signal, and further capture the long-range dependency of the ERG signal, thereby improving the accuracy of the retinal signal reconstruction.

[0134] In step S140 of some embodiments, for each time step, the second output state, the first output state and the long-range attention of the previous time step are spliced to obtain second spliced features. The second spliced features are encoded by a second long short-term memory sub-model to obtain a second output state, and the second output state of each time step is taken as an initial retinal signal feature. The initial retinal signal feature is represented as:

[0135]

[0136] wherein, represents the initial retinal signal feature; LSTM 2 represents the second long short-term memory sub-model.

[0137] In step S150 of some embodiments, the initial retinal signal feature is mapped by a linear layer, and the initial retinal signal feature after the feature mapping is signal-aligned with the stimulus paradigm features in the feature space by a signal alignment module to extract the features related to the stimulus paradigm in the initial retinal signal feature, thereby obtaining a target aligned signal feature. The target aligned signal feature is the feature encoded by the signal alignment module from the initial retinal signal feature. The signal alignment module is trained based on the sample stimulus paradigm features and the sample retinal electrography signal features.

[0138] Please refer to Figure 5 In some embodiments, the training process of the signal alignment module can include, but is not limited to, steps S510 to S580:

[0139] Step S510, obtaining a sample stimulus paradigm and a sample retinal electrography signal;

[0140] Step S520, stimulus paradigm encoding the sample stimulus paradigm to obtain sample stimulus paradigm features;

[0141] Step S530, encoding the sample retinal electrography signal by a retinal signal encoder to obtain sample retinal electrography signal features;

[0142] Step S540, encoding the sample stimulus paradigm features by a preset alignment module to obtain encoded stimulus paradigm features;

[0143] Step S550, the sample retinal electroimaging signal feature is encoded into a retinal signal feature by a preset alignment module;

[0144] Step S560, a similarity score between the encoded stimulus paradigm feature and the encoded retinal signal feature is calculated;

[0145] Step S570, a third loss is determined according to the similarity score and the encoded stimulus paradigm feature based on a preset loss function;

[0146] Step S580, the preset alignment module is updated in parameters according to the third loss, and a signal alignment module is obtained.

[0147] In step S510 of some embodiments, a sample stimulus paradigm and a sample retinal electroimaging signal are obtained from a training set, and the sample retinal electroimaging signal is a retinal electroimaging signal under the sample stimulus paradigm.

[0148] In step S520 of some embodiments, the sample stimulus paradigm is encoded into a stimulus paradigm feature by a pre-trained convolutional neural network. The convolutional neural network can be ResNet50, and the sample stimulus paradigm feature includes image dynamic features, texture features, spatial relationship features, color features, shape structure features, etc. It should be noted that in the training process of the signal alignment module, the model parameters of the convolutional neural network are fixed and do not need to be updated.

[0149] In step S530 of some embodiments, the retinal signal encoder includes a first long short-term memory sub-model and a second long short-term memory sub-model in cascade. The sample retinal electroimaging signal is encoded into a sample retinal electroimaging signal feature by the first long short-term memory sub-model, the second long short-term memory sub-model, and a long-distance attention mechanism, and the sample retinal electroimaging signal feature is an ERG signal feature of the sample retinal electroimaging signal. For specific retinal signal encoding process, refer to steps S130 to S140, which will not be repeated here.

[0150] In step S540 of some embodiments, the sample stimulus paradigm feature is encoded into an encoded stimulus paradigm feature by a preset alignment module. That is wherein V represents the sample stimulus paradigm feature, represents the encoded stimulus paradigm feature, and CL IP represents the preset alignment module.

[0151] In step S550 of some embodiments, the sample retinal electroimaging signal feature is linearly mapped by a linear layer, and the linearly mapped encoding retinal signal features The linear layer is used to convert the sample retinal electrical imaging signal features into the same dimension as the sample stimulus paradigm features.

[0152] In step S560 of some embodiments, a dot product operation is performed on the encoded stimulus paradigm features and the encoded retinal signal features to obtain a similarity score. The similarity score is represented as:

[0153]

[0154] where logits represents the similarity score; T represents the transpose operation; and · represents the dot product operation.

[0155] In step S570 of some embodiments, a preset loss function is a loss function used to train the preset alignment module to achieve cross-modal alignment between the sample retinal electrical imaging signal features and the sample stimulus paradigm features. Based on the preset loss function, a first cross-entropy loss is calculated according to the similarity score and the encoded stimulus paradigm features, a second cross-entropy loss is calculated according to the transpose of the similarity score and the encoded stimulus paradigm features, and a third loss is obtained by weighted average of the first cross-entropy loss and the second cross-entropy loss. The preset loss function is represented as:

[0156]

[0157] where loss align is the third loss; cross_entropy represents the cross-entropy loss; logits represents the similarity score; T represents the transpose operation; and represents the encoded stimulus paradigm features.

[0158] In step S580 of some embodiments, the linear layer and the preset alignment module are updated according to the third loss, and the preset alignment module at the time when the third loss is minimized is taken as the signal alignment module.

[0159] The above steps S510 to S580 train the preset alignment module, so that the signal alignment module can extract features similar to the stimulus paradigm from the retinal electrical imaging signal features, thereby improving the accuracy of retinal signal reconstruction.

[0160] The target generator generates a stimulation paradigm for the target alignment signal feature to obtain a target stimulation paradigm of the retinal electrical imaging signal, so as to simulate the encoding mode of the visual stimulation information in the retina. The target generator adopts StyleGAN. The target generator can not only reconstruct the stimulation paradigm from the ERG signal, but also reconstruct the stimulation paradigm from the stimulation paradigm. In some embodiments, the target stimulation paradigm is obtained, the target stimulation paradigm is encoded into a first stimulation paradigm feature by a pre-trained convolutional neural network, and the target stimulation paradigm feature is obtained. The target stimulation paradigm feature is aligned by the signal alignment module to obtain an aligned stimulation paradigm feature. The aligned stimulation paradigm feature is the feature of the target stimulation paradigm feature after alignment with the ERG signal feature in the feature space. The aligned stimulation paradigm feature is generated by the target generator to obtain a generated stimulation paradigm feature. It should be noted that when the input is the stimulation paradigm, it does not need to pass through the linear layer for linear mapping, but can be directly aligned by the signal alignment module.

[0161] Please refer to Figure 6 In some embodiments, the target generator includes a mapping sub-model and a generation sub-model, the generation sub-model includes a plurality of resolution units with increasing resolutions in sequence, and step S160 can include but is not limited to steps S610 to S650:

[0162] Step S610, randomly generating a noise signal;

[0163] Step S620, mapping the target alignment signal feature and the noise signal by the mapping sub-model to obtain a latent space distribution feature;

[0164] Step S630, performing affine transformation on the latent space distribution feature to obtain a style vector;

[0165] Step S640, determining the output feature of the current resolution unit based on the output feature of the previous resolution unit, the style vector, and the Gaussian noise component generated for the current resolution unit according to the current resolution unit;

[0166] Step S650, taking the output feature of the last resolution unit as the target stimulation paradigm.

[0167] In step S610 of some embodiments, in order to simulate the randomness in a real scene, a pseudo-random number generator or a true random number generator is used to randomly generate a noise signal.

[0168] In step S620 of some embodiments, the target alignment signal feature and the noise signal are spliced to obtain a third spliced feature, the third spliced feature is normalized, the normalized third spliced feature is feature mapped by a mapping sub-model to convert the feature into a hidden space to obtain a hidden space distribution feature. The mapping sub-model learns the hidden space distribution of the intermediate layer of the picture from the training set, and is used to generate the hidden encoding of the control image feature. The hidden space distribution feature is the distribution feature of the third spliced feature in the hidden space. The mapping sub-model is composed of eight fully connected layers (FC), and each FC has the same structure.

[0169] In step S630 of some embodiments, the hidden space distribution feature is affine transformed by a fully connected layer to obtain a style vector. The style vector is a spatially invariant style sampled from the hidden space distribution feature, and the style vector includes a style scale and a style bias. y = (y s ,y b ), y s represents the style scale, and y b represents the style bias.

[0170] In step S640 of some embodiments, the generation sub-model includes nine resolution units with increasing resolutions in turn, and the resolution of the lth resolution unit is 2 l+1 ×2 l+1 , and the resolution change range is 4 2 -1024 2. Except that the first resolution unit has only one 3x3 convolution layer, the other resolution units have two 3x3 convolution layers, and each convolution layer is followed by an instance regularization operation. The target generator introduces an additional Gaussian noise component when calculating the feature map at each scale. A Gaussian noise component is generated for each resolution unit to have some random detail changes in the generated image space. The resolution of the Gaussian noise component is the same as the resolution of the resolution unit. The instance regularization operation after each convolution layer of the generation submodel can be controlled by the style vector and the Gaussian noise component. The generation submodel can generate images for various styles learned under the control of the style vector and the Gaussian noise component. If the current resolution unit is the first resolution unit, the output feature output by the last resolution unit is a constant value with a constant size of 4x4x512. If the current resolution unit is not the first resolution unit, the constant value can be replaced by a 3x3 convolution layer. Specifically, if the current resolution unit is the first resolution unit, the constant value, the style vector, and the Gaussian noise component generated for the current resolution unit are added to obtain the first fused feature. If the current resolution unit is not the first resolution unit, the output feature output by the last resolution unit is convolved by the first 3x3 convolution layer, and the convolved feature, the style vector, and the Gaussian noise component generated for the current resolution unit are added to obtain the first fused feature. The first instance regularization feature is obtained by performing pixel-by-pixel instance regularization on the first fused feature, the style scale, and the style bias. The object of instance regularization is each channel of the first fused feature, so that the feature map can have hidden distribution parameters representing the style. The process of instance regularization is represented as:

[0171]

[0172] wherein x i represents the i-th feature map, i.e., the first fused feature; y represents the style vector; AdaIN represents the instance regularization operation; μ(·) represents the mean, and σ(·) represents the variance; y s,i represents the style scale of the i-th feature map; y b,i represents the style bias of the i-th feature map.

[0173] The first instance regularization feature is convolved by a 3x3 convolution layer to obtain a convolution feature, and the convolution feature, the style vector, and the Gaussian noise component generated for the current resolution unit are added to obtain a second fused feature. The second instance regularization feature is obtained by performing instance regularization on the second fused feature, the style scale, and the style bias. The second instance regularization feature is taken as the output feature output by the current resolution unit.

[0174] In step S650 of some embodiments, the output feature output by the last resolution unit is taken as the target stimulus paradigm.

[0175] Through the steps S610 to S650, the mapping between the ERG signal and the stimulation paradigm can be established to obtain the encoding and processing manner of the retinal cells to the stimulation paradigm.

[0176] Referring to Figure 7 In some embodiments, the training process of the target generator can include, but is not limited to, steps S710 to S770:

[0177] In step S710, a sample stimulation paradigm and a sample electroretinogram signal are obtained.

[0178] In step S720, a preset generator is used to generate a stimulation paradigm for the sample stimulation paradigm to obtain a first predicted stimulation paradigm.

[0179] In step S730, the preset generator is used to generate a stimulation paradigm for the sample electroretinogram signal to obtain a second predicted stimulation paradigm.

[0180] In step S740, the parameters of the preset generator are fixed, and the preset discriminator is updated according to the sample stimulation paradigm, the first predicted stimulation paradigm, and the second predicted stimulation paradigm.

[0181] In step S750, the first predicted stimulation paradigm is discriminated by the updated preset discriminator to obtain a first discrimination feature, and a fourth loss is determined according to the first discrimination feature.

[0182] In step S760, the second predicted stimulation paradigm is discriminated by the preset discriminator to obtain a second discrimination feature, and a fifth loss is determined according to the second discrimination feature.

[0183] In step S770, the preset generator is updated according to the fourth loss and the fifth loss to obtain the target generator.

[0184] In step S710 of some embodiments, the sample stimulation paradigm and the sample electroretinogram signal are obtained from a training set.

[0185] In step S720 of some embodiments, in order to enable the preset generator to reconstruct the stimulation paradigm from the ERG signal and to reconstruct the stimulation paradigm from the stimulation paradigm, the preset generator is used to generate a stimulation paradigm for the sample stimulation paradigm to obtain a first predicted stimulation paradigm. The first predicted stimulation paradigm is a stimulation paradigm generated based on the sample stimulation paradigm, and the preset generator has the same network structure as the target generator. The manner of stimulation paradigm generation can refer to steps S610 to S650, which will not be described here.

[0186] In step S730 of some embodiments, the preset generator is used to generate a stimulation paradigm for the sample retinal electrical imaging signal, to obtain a second predicted stimulation paradigm. The second predicted stimulation paradigm is a stimulation paradigm generated based on the sample retinal electrical imaging signal.

[0187] In step S740 of some embodiments, the preset generator and the preset discriminator are trained in an alternating training manner. Specifically, the parameters of the preset generator are fixed, and the preset discriminator is updated in parameters according to the sample stimulation paradigm, the first predicted stimulation paradigm, and the second predicted stimulation paradigm. The preset discriminator includes nine convolutional layers, and the resolution changes in the range of 1024 2 -4 2 .

[0188] In step S750 of some embodiments, the first predicted stimulation paradigm is discriminated by the updated preset discriminator to obtain a first discrimination feature. The first discrimination feature is used to represent a probability distribution of the first predicted stimulation paradigm being true. A fourth loss is determined according to the first discrimination feature based on a generator loss function. The generator loss function is represented as:

[0189] Loss G =log(exp(D(G(Z)))+1) Formula (14)

[0190] Wherein, loss G is the generator loss function; G represents the generator; D represents the discriminator; and z represents the input of the generator.

[0191] In step S760 of some embodiments, the second predicted stimulation paradigm is discriminated by the updated preset discriminator to obtain a second discrimination feature. The second discrimination feature is used to represent a probability distribution of the second predicted stimulation paradigm being true. A fifth loss is determined according to the second discrimination feature based on the generator loss function.

[0192] In step S770 of some embodiments, the higher the score of the generated sample, the better the goal of the generator, the lower the score of the generated sample, the better the goal of the discriminator, and the higher the score of the real sample, the better. The fourth loss and the fifth loss are summed to obtain a first target loss, and the first target loss is maximized to update the parameters of the preset generator. Alternatively, the mean square error of the first predicted stimulus paradigm and the sample stimulus paradigm is calculated to obtain a first error, the mean square error of the second predicted stimulus paradigm is calculated to obtain a second error, and the first error, the second error, the negative of the fourth loss, and the negative of the fifth loss are summed to obtain the first target loss, and the first target loss is minimized to update the parameters of the preset generator. When the number of times of alternating training reaches a preset number threshold or the discriminator discriminates that the first predicted stimulus paradigm and the second predicted stimulus paradigm are both true, a target generator is obtained to enable the target generator to have the ability to generate a stimulus paradigm infinitely close to the real stimulus paradigm. It can be understood that when the discriminator considers that the image generated by the generator is false, the generator needs to be regenerated.

[0193] Through the above steps S710 to S770, a target generator can be obtained to establish a mapping between an ERG signal and a stimulus paradigm or a mapping between a stimulus paradigm and a stimulus paradigm, thereby accurately simulating the encoding and processing of visual stimulus information in retinal cells.

[0194] Please refer to Figure 8 In some embodiments, step S740 can include but is not limited to steps S810 to S850:

[0195] In step S810, a third discriminative feature is obtained by discriminating the sample stimulus paradigm through a preset discriminator, and a sixth loss is determined according to the third discriminative feature.

[0196] In step S820, a fourth discriminative feature is obtained by discriminating the first predicted stimulus paradigm through the preset discriminator, and a seventh loss is determined according to the fourth discriminative feature.

[0197] In step S830, a fifth discriminative feature is obtained by discriminating the second predicted stimulus paradigm through the preset discriminator, and an eighth loss is determined according to the fifth discriminative feature.

[0198] In step S840, the preset discriminator is updated according to the sixth loss, the seventh loss, and the eighth loss.

[0199] In step S810 of some embodiments, a third discriminative feature is obtained by discriminating the sample stimulus paradigm through a preset discriminator, and the third discriminative feature is used to represent the probability that the sample stimulus paradigm is true, and a sixth loss is determined according to the third discriminative feature. The sixth loss is represented as log(exp(-D(x))+1), and D(x) is the third discriminative feature.

[0200] In step S820 of some embodiments, the first predicted stimulus paradigm is discriminated by the preset discriminator to obtain a fourth discrimination feature, the fourth discrimination feature being used to represent the probability that the first predicted stimulus paradigm is true, and a seventh loss is determined according to the fourth discrimination feature. The seventh loss is represented as log(exp(D(G(z)))+1).

[0201] In step S830 of some embodiments, the second predicted stimulus paradigm is discriminated by the preset discriminator to obtain a fifth discrimination feature, the fifth discrimination feature being used to represent the probability that the second predicted stimulus paradigm is true, and an eighth loss is determined according to the fifth discrimination feature. The eighth loss is calculated in the same way as the seventh loss.

[0202] In step S840 of some embodiments, the sixth loss, the seventh loss and the eighth loss are summed to obtain a second target loss, and the preset discriminator is updated in parameters by minimizing the second target loss.

[0203] Through the above steps S810 to S840, the discrimination ability of the discriminator can be improved, the output result of the discriminator is fed back to the generator, and the generator is prompted to generate a stimulus paradigm with higher quality.

[0204] The training set contains 50000 sample pairs in total, the sample pairs including ERG signals and stimulus paradigms, and the test set contains 500 sample pairs. The training set is used for training of the retinal signal encoder, the signal alignment module and the StyleGAN, and the test set is used for testing of the reconstruction effect of the retinal signal. The retinal signal encoder and the StyleGAN are both trained on an NVIDIA RTX A6000, with a batch size of 16. When trained for 3 cycles of epochs, the loss of the StyleGAN converges. After the model training is completed, the test set is used to test the results of the ERG signal to stimulus paradigm and the image to image generation. Figure 9A and Figure 9B The results of stimulus paradigm reconstruction for two different ERG signals are shown. Figure 9A The real stimulus paradigms corresponding to the ERG signals in include local stimuli, global stimuli, square shape stimuli and color stimuli. It can be seen from the reconstructed stimulus paradigms that the present application can accurately reconstruct these stimuli.

[0205] Figure 9B A more complex stimulus paradigm and the real ERG signal corresponding to the stimulus paradigm are shown. Compared with the real stimulus paradigm of Figure 9A , Figure 9BThe real stimulus paradigm in the box contains more complex shape stimuli, such as a circular area in the box, and the color of the box and the color of the circular area are different. For this more complex stimulus paradigm, the present application can also accurately reconstruct the stimulus paradigm. The generator of the present application can not only realize the reconstruction from the ERG signal to the stimulus paradigm, but also realize the reconstruction from the stimulus paradigm to the stimulus paradigm, and the reconstruction result is as shown in Figure 9C Figure 9C The real stimulus paradigm in the box is used for stimulus paradigm reconstruction to obtain the reconstructed stimulus paradigm. Compared with the real stimulus paradigm, the reconstructed stimulus paradigm can accurately reconstruct the color, shape, local, and global stimulus in the real stimulus paradigm.

[0206] Please refer to Figure 10 The embodiment of the present application also provides a multi-modal retinal signal reconstruction device based on a long-distance attention mechanism, which can realize the multi-modal retinal signal reconstruction method based on the long-distance attention mechanism. The multi-modal retinal signal reconstruction device based on the long-distance attention mechanism comprises:

[0207] The acquisition module 1010 is configured to acquire a target retinal electrical imaging signal.

[0208] The input module 1020 is configured to input the target retinal electrical imaging signal to a retinal signal encoder, and the retinal signal encoder comprises a first long short-term memory sub-model and a second long short-term memory sub-model.

[0209] The first encoding module 1030 is configured to, for each time step in a plurality of time steps, perform first feature encoding on a first output state output by the first long short-term memory sub-model at a previous time step, a second output state output by the second long short-term memory sub-model at the previous time step, and the target retinal electrical imaging signal by the first long short-term memory sub-model, to obtain the first output state output by the first long short-term memory sub-model, and determine a long-distance attention according to the first output state.

[0210] The second encoding module 1040 is configured to perform second feature encoding on the second output state, the first output state, and the long-distance attention at the previous time step by the second long short-term memory sub-model to obtain the second output state, and take the second output state of all time steps as an initial retinal signal feature.

[0211] The alignment module 1050 is configured to perform signal alignment on the initial retinal signal feature by a signal alignment module to obtain a target aligned signal feature; and the signal alignment module is trained based on a sample stimulus paradigm feature and a sample retinal electrical imaging signal feature.

[0212] The generation module 1060 is configured to perform stimulus paradigm generation on the target aligned signal feature by a target generator to obtain a target stimulus paradigm. ​

[0213] The specific implementation of the multi-modal retinal signal reconstruction device based on the long-distance attention mechanism is basically the same as the specific embodiment of the multi-modal retinal signal reconstruction method based on the long-distance attention mechanism, and will not be repeated here.

[0214] The embodiments of the present application also provide an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor implements the multi-modal retinal signal reconstruction method based on the long-distance attention mechanism when executing the computer program. The electronic device can be any intelligent terminal including a tablet computer, a vehicle-mounted computer, etc.

[0215] Please refer to Figure 11 , Figure 11 The hardware structure of the electronic device of another embodiment is illustrated, which includes:

[0216] The processor 1110 can be implemented in the form of a general-purpose CPU (Central Processing Unit), a microprocessor, an ASIC (Application Specific Integrated Circuit), or one or more integrated circuits, etc., for executing related programs to implement the technical solutions provided by the embodiments of the present application.

[0217] The memory 1120 can be implemented in the form of a ROM (ReadOnly Memory), a static storage device, a dynamic storage device, or a RAM (Random Access Memory), etc. The memory 1120 can store an operating system and other application programs. When the technical solutions provided by the embodiments of the present application are implemented by software or firmware, the related program codes are saved in the memory 1120 and called and executed by the processor 1110 to implement the multi-modal retinal signal reconstruction method based on the long-distance attention mechanism.

[0218] The input / output interface 1130 is used to realize information input and output.

[0219] The communication interface 1140 is used to realize the communication interaction between the device and other devices. The communication can be realized by a wired manner (such as a USB, a network cable, etc.) or a wireless manner (such as a mobile network, WI FI, Bluetooth, etc.).

[0220] The bus 1150 transmits information between various components (such as the processor 1110, the memory 1120, the input / output interface 1130, and the communication interface 1140) of the device.

[0221] The processor 1110, the memory 1120, the input / output interface 1130, and the communication interface 1140 are communicatively connected with each other through a bus 1150.

[0222] The computer readable storage medium stores a computer program, and the computer program is executed by the processor to implement the multi-modal retinal signal reconstruction method based on the long-distance attention mechanism.

[0223] The memory, as a non-transitory computer readable storage medium, can be used to store non-transitory software programs and non-transitory computer executable programs. In addition, the memory can include a high-speed random access memory, and can also include a non-transitory memory, such as at least one magnetic disk storage device, a flash memory device, or other non-transitory solid-state memory device. In some embodiments, the memory can optionally include a memory remotely arranged relative to the processor, and these remote memories can be connected to the processor through a network. Examples of the above network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0224] The embodiments described in the embodiments of the present application are used to more clearly illustrate the technical solutions of the embodiments of the present application, and do not constitute a limitation on the technical solutions provided by the embodiments of the present application. Those skilled in the art can know that, with the evolution of technology and the appearance of new application scenarios, the technical solutions provided by the embodiments of the present application are also applicable to similar technical problems.

[0225] Those skilled in the art can understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and can include more or fewer steps than the figures shown, or combine certain steps, or different steps.

[0226] The device embodiments described above are only schematic, and the units described as separate components can or can not be physically separate, that is, can be located in one place, or can be distributed on multiple network units. Part or all of the modules can be selected according to actual needs to achieve the purpose of the embodiments of the present application.

[0227] In addition, each functional unit in each embodiment of the present application can be integrated in one processing unit, or each unit can be physically present alone, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of a software functional unit.

[0228] The integrated unit, if implemented in the form of a software function unit and sold or used as an independent product, can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application, essentially or in other words, the part that contributes to the prior art or the whole or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium, and includes multiple instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods of the various embodiments of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various program storage media.

[0229] The preferred embodiments of the embodiments of the present application are described above with reference to the accompanying drawings, and are not limited to the scope of the embodiments of the present application. Any modifications, equivalent replacements and improvements made by those skilled in the art without departing from the scope and essence of the embodiments of the present application shall be within the scope of the embodiments of the present application.

Claims

1. A method for multi-modal retinal signal reconstruction based on long-range attention mechanism, characterized in that, The method comprises: acquiring a target retinal electrical imaging signal; inputting the target retinal electrical imaging signal into a retinal signal encoder, the retinal signal encoder comprising a first long short-term memory submodel and a second long short-term memory submodel; for each time step in a plurality of time steps, performing first feature encoding on a first output state output by the first long short-term memory submodel at a previous time step, a second output state output by the second long short-term memory submodel at the previous time step, and the target retinal electrical imaging signal by the first long short-term memory submodel to obtain a first output state output by the first long short-term memory submodel, and determining long-distance attention according to the first output state; performing second feature encoding on the second output state at the previous time step, the first output state, and the long-distance attention by the second long short-term memory submodel to obtain a second output state, and taking the second output states of all the time steps as initial retinal signal features; performing signal alignment on the initial retinal signal features by a signal alignment module to obtain target aligned signal features, the signal alignment module being trained based on sample stimulus paradigm features and sample retinal electrical imaging signal features; generating a target stimulus paradigm by a target generator based on the target aligned signal features.

2. The long-range attention mechanism based multi-modal retinal signal reconstruction method according to claim 1, characterized in that, The determination of the long-distance attention according to the first output state comprises: determining a retrieval feature vector, a key feature vector, and a value feature vector according to the first output state, and determining a signal relationship feature according to the retrieval feature vector, the key feature vector, and the value feature vector; determining a target threshold probability according to the first output state of the current time step and the signal relationship feature based on a threshold unit; determining the long-distance attention according to the first output state of the current time step, the signal relationship feature, and the target threshold probability.

3. The long-range attention mechanism based multi-modal retinal signal reconstruction method according to claim 1, characterized in that, The retinal signal encoder is trained according to the following steps: acquiring a reference retinal electrical imaging signal sample, a positive retinal electrical imaging signal sample, and a negative retinal electrical imaging signal sample; performing retinal signal encoding on the reference retinal electrical imaging signal sample by a preset encoder to obtain reference sample retinal signal features; performing retinal signal encoding on the positive retinal electrical imaging signal sample by the preset encoder to obtain positive sample retinal signal features; performing retinal signal encoding on the negative retinal electrical imaging signal sample by the preset encoder to obtain negative sample retinal signal features; determining a first loss according to the reference sample retinal signal features and the positive sample retinal signal features, and determining a second loss according to the reference sample retinal signal features and the negative sample retinal signal features; calculating a margin distance between the positive sample retinal signal features and the negative sample retinal signal features; updating parameters of a preset encoder according to the first loss, the second loss and the margin distance until the first loss is less than the second loss, and the second loss is less than a sum of the first loss and the margin distance, and taking the preset encoder as the retinal signal encoder.

4. The long-range attention mechanism based multi-modal retinal signal reconstruction method according to claim 1, characterized in that, The signal alignment module is trained according to the following steps: obtaining a sample stimulus paradigm and a sample retinal electrical imaging signal; encoding the sample stimulus paradigm to obtain a sample stimulus paradigm feature; encoding the sample retinal electrical imaging signal by the retinal signal encoder to obtain a sample retinal electrical imaging signal feature; encoding the sample stimulus paradigm feature by a preset alignment module to obtain an encoded stimulus paradigm feature; encoding the sample retinal electrical imaging signal feature by the preset alignment module to obtain an encoded retinal signal feature; calculating a similarity score between the encoded stimulus paradigm feature and the encoded retinal signal feature; determining a third loss based on a preset loss function according to the similarity score and the encoded stimulus paradigm feature; updating parameters of the preset alignment module according to the third loss to obtain the signal alignment module.

5. The long-range attention mechanism based multi-modal retinal signal reconstruction method according to any one of claims 1 to 4, characterized in that, The target generator includes a mapping sub-model and a generation sub-model, the generation sub-model includes a plurality of resolution units with increasing resolutions, and the target generator encodes the target alignment signal feature to obtain a target stimulus paradigm, including: randomly generating a noise signal; mapping the target alignment signal feature and the noise signal by the mapping sub-model to obtain a latent space distribution feature; performing affine transformation on the latent space distribution feature to obtain a style vector; determining an output feature output by a current resolution unit based on an output feature output by a previous resolution unit, the style vector and a Gaussian noise component generated for the current resolution unit; taking the output feature output by the last resolution unit as the target stimulus paradigm.

6. The long-range attention mechanism based multi-modal retinal signal reconstruction method according to any one of claims 1 to 4, characterized in that, The target generator is trained according to the following steps: obtaining a sample stimulus paradigm and a sample retinal electrical imaging signal; generating a first predicted stimulus paradigm by a preset generator from the sample stimulus paradigm; generating a second predicted stimulus paradigm by the preset generator from the sample retinal electrical imaging signal; updating parameters of a preset discriminator based on the sample stimulus paradigm, the first predicted stimulus paradigm and the second predicted stimulus paradigm while fixing parameters of the preset generator; determining a fourth loss based on a first discrimination feature obtained by discriminating the first predicted stimulus paradigm by the updated preset discriminator and the first discrimination feature; determining a fifth loss based on a second discrimination feature obtained by discriminating the second predicted stimulus paradigm by the preset discriminator and the second discrimination feature; updating parameters of the preset generator according to the fourth loss and the fifth loss to obtain the target generator.

7. The long-range attention mechanism based multi-modal retinal signal reconstruction method according to claim 6, characterized in that, The parameter updating of the preset discriminator according to the sample stimulus paradigm, the first predicted stimulus paradigm and the second predicted stimulus paradigm comprises: obtaining third discrimination features by discriminating the sample stimulus paradigm through the preset discriminator, and determining a sixth loss according to the third discrimination features; obtaining fourth discrimination features by discriminating the first predicted stimulus paradigm through the preset discriminator, and determining a seventh loss according to the fourth discrimination features; obtaining fifth discrimination features by discriminating the second predicted stimulus paradigm through the preset discriminator, and determining an eighth loss according to the fifth discrimination features; updating parameters of the preset discriminator according to the sixth loss, the seventh loss and the eighth loss.

8. A multi-modal retinal signal reconstruction apparatus based on long-range attention mechanism, characterized in that, The device comprises: an acquisition module configured to acquire a target retinal electrical imaging signal; an input module configured to input the target retinal electrical imaging signal to a retinal signal encoder, the retinal signal encoder comprising a first long short-term memory sub-model and a second long short-term memory sub-model; a first encoding module configured to, for each time step in a plurality of time steps, perform first feature encoding on a first output state output by the first long short-term memory sub-model at a previous time step, a second output state output by the second long short-term memory sub-model at the previous time step and the target retinal electrical imaging signal through the first long short-term memory sub-model, to obtain a first output state output by the first long short-term memory sub-model, and determine long-distance attention according to the first output state; a second encoding module configured to perform second feature encoding on the second output state at the previous time step, the first output state and the long-distance attention through the second long short-term memory sub-model to obtain a second output state, and take the second output state of all the time steps as initial retinal signal features; an alignment module configured to perform signal alignment on the initial retinal signal features through a signal alignment module to obtain target aligned signal features, the signal alignment module being trained based on sample stimulus paradigm features and sample retinal electrical imaging signal features; a generation module configured to perform stimulus paradigm generation on the target aligned signal features through a target generator to obtain a target stimulus paradigm.

9. An electronic device, characterized by The electronic device comprises a memory and a processor, the memory stores a computer program, and the processor implements the long-distance attention mechanism based multi-modal retinal signal reconstruction method of any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium storing a computer program, the computer program comprising instructions that, when executed by a computer, cause the computer to perform the method of any one of claims 1 to 9. The computer program is executed by the processor to implement the long-distance attention mechanism based multi-modal retinal signal reconstruction method of any one of claims 1 to 7.

Images