Electrical stimulation method, device and equipment and storage medium

Abstract

The invention relates to the field of artificial intelligence (AI), in particular to an electrical stimulation method, an electrical stimulation device, electrical stimulation equipment and a storage medium. The method comprises the steps of obtaining a to-be-perceived image of a target object, inputting the to-be-perceived image into a trained electrical stimulation model to obtain electrical stimulation parameters output by the electrical stimulation model, and performing corresponding electrical stimulation operation on the target object to recover visual perception of the target object on the to-be-perceived image. An electrical stimulation generator in electrical stimulation is obtained by performing first link training and second link training based on a plurality of sample images and corresponding sample neural signals, the first link represents a verification link among electrical stimulation parameters, analog neural signals and reconstructed images, and the second link represents a verification link among the reconstructed images. The second link represents a verification link among the electrical stimulation parameters, the light magical view image and the sample image, so that the electrical stimulation model can greatly optimize the generation process of the electrical stimulation parameters, and the accuracy and generalization of restoring visual perception through electrical stimulation are improved.

Description

Technical Field

[0001] This application relates to the technical field of Artificial Intelligence (AI), and particularly to an electrical stimulation method, device, equipment, and storage medium. Background Art

[0002] Visual impairment is a common problem globally, especially among the blind population. The defect in visual perception severely restricts daily life and information interaction. Currently, related technologies can use cortical electrical stimulation technology to stimulate the visual cortex of the brain to generate phosphenes for visual restoration.

[0003] The effectiveness of electrical stimulation is closely related to the electrical stimulation parameters and the accuracy of neural activity decoding. However, in related technologies, the setting of electrical stimulation parameters highly depends on manual experience adjustment, resulting in difficulty in accurately matching the visual perception characteristics of different objects for electrical stimulation parameters, and relatively low neural decoding accuracy, making it difficult to accurately map the neuron activities induced by electrical stimulation into expected visual perception, leading to relatively low accuracy in visual perception simulation, thus greatly reducing the accuracy and generalization of electrical stimulation. Summary of the Invention

[0004] The present invention provides an electrical stimulation method, device, equipment, and storage medium to improve the accuracy and generalization of visual restoration through electrical stimulation.

[0005] In a first aspect, this application provides an electrical stimulation method, and the method includes:

[0006] Obtain a to-be-perceived image of a target object, and input the to-be-perceived image into a trained electrical stimulation model;

[0007] Obtain the electrical stimulation parameters output by the electrical stimulation model; the electrical stimulation parameters are used to perform corresponding electrical stimulation operations on the target object to restore the visual perception of the target object for the to-be-perceived image;

[0008] Wherein, the electrical stimulation model includes an electrical stimulation generator, and the electrical stimulation generator is obtained through first-link training and second-link training based on a sample data set; the sample data set includes multiple sample images and corresponding sample neural signals, and the sample neural signals are generated by a sample object in response to the multiple sample images; the first link represents the verification link among electrical stimulation parameters, simulated neural signals, and reconstructed images, and the second link represents the verification link among electrical stimulation parameters, phosphene images, and sample images.

[0009] In a second aspect, the present application provides a computer device, including a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the computer program, any one of the electrical stimulation methods in the above first aspect is implemented.

[0010] In a third aspect, the present application provides a computer storage medium. The computer-readable storage medium stores computer program instructions, and the computer program instructions are executed by the processor to implement any one of the electrical stimulation methods in the above first aspect.

[0011] In a fourth aspect, a computer program product provided by an embodiment of the present application includes computer program instructions. When the computer program instructions are executed by the processor, any one of the electrical stimulation methods in the above first aspect is implemented.

[0012] The beneficial effects of the present invention are as follows:

[0013] An embodiment of the present application provides an electrical stimulation method. The method acquires a to-be-perceived image of a target object, and inputs the to-be-perceived image into a trained electrical stimulation model to obtain electrical stimulation parameters output by the electrical stimulation model, which are used to perform corresponding electrical stimulation operations on the target object to restore the visual perception of the target object for the to-be-perceived image. The electrical stimulation generator in this electrical stimulation is obtained through first-link training and second-link training based on multiple sample images and corresponding sample nerve signals. The first link represents the verification link among the electrical stimulation parameters, the simulated nerve signals, and the reconstructed images, and the second link represents the verification link among the electrical stimulation parameters, the phosphene images, and the sample images, enabling the electrical stimulation model to greatly optimize the generation process of the electrical stimulation parameters and improve the accuracy and generalization of the electrical stimulation for restoring visual perception. Description of the Drawings

[0014] To more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the following will briefly introduce the drawings required for use in the description of the embodiments or related technologies. Obviously, the drawings in the following description are only the embodiments of the present application. For those of ordinary skill in the art, other drawings can be obtained based on the provided drawings without creative efforts.

[0015] Figure 1 It is a schematic flowchart of a training method for an electrical stimulation model provided by an embodiment of the present application;

[0016] Figure 2 It is a schematic diagram of a training process of an electrical stimulation model provided by an embodiment of the present application;

[0017] Figure 3 It is a flowchart of an electrical stimulation method provided by an embodiment of the present application;

[0018] Figure 4 A schematic flowchart of model training and model application provided by an embodiment of the present application;

[0019] Figure 5 A schematic structural diagram of a computer device provided by an embodiment of the present application. Detailed implementation manners

[0020] To make the objectives, technical solutions, and advantages of the present application clearer and more understandable, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present application. Apparently, the described embodiments are only a part of the embodiments of the present application, rather than all of them. Based on the embodiments of the present application, all other embodiments obtained by those of ordinary skill in the art without making creative efforts shall fall within the protection scope of the present application. Without conflict, the embodiments in the present application and the features in the embodiments may be combined with each other arbitrarily. And although the logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than here.

[0021] The terms "first" and "second" in the description and claims of the present application and the above accompanying drawings are used to distinguish different objects, rather than to describe a specific order. In addition, the term "including" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or units is not limited to the listed steps or units, but optionally further includes steps or units not listed, or optionally further includes other steps or units inherent to these processes, methods, products, or devices. "Multiple" in the present application may mean at least two, for example, it may be two, three, or more, and the embodiments of the present application do not make limitations.

[0022] The term "and / or" in the embodiments of the present application is merely a description of the association relationship between associated objects, indicating that three relationships may exist. For example, A and / or B may represent: A exists alone, A and B exist simultaneously, and B exists alone. In addition, the character " / " in this article generally represents an "or" relationship between the associated objects before and after.

[0023] It should be understood that in the following specific embodiments of the present application, data related to neural signals and the like are involved. When the embodiments of the present application are applied to specific products or technologies, relevant permissions or consents need to be obtained, and the collection, use, and processing of relevant data need to comply with the relevant laws, regulations, and standards of relevant countries and regions. For example, relevant volunteers can be recruited and relevant agreements for authorizing data of the volunteers can be signed, and then the data of these volunteers can be used for implementation; or, implementation can be carried out within the scope of an authorized organization, and the following embodiments can be implemented by using the data of the internal members of the organization for data management; or, the relevant data used in specific implementation are all simulated data, such as simulated data generated in a virtual scenario.

[0024] The embodiments of the present application relate to artificial intelligence and machine learning (ML) technologies, and are mainly designed based on machine learning in artificial intelligence.

[0025] Artificial intelligence is to use a digital computer or a machine controlled by a digital computer to simulate, extend, and expand human intelligence, and is a theory, method, technology, and application system that can perceive the environment, acquire knowledge, and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science, which attempts to understand the essence of intelligence and produce a new intelligent machine that can react in a way similar to human intelligence. Artificial intelligence also studies the design principles and implementation methods of various intelligent machines, so that the machines have the functions of perception, reasoning, and decision-making.

[0026] Artificial intelligence technology is an interdisciplinary subject involving a wide range of fields, including both hardware-level technologies and software-level technologies. The basic technologies of artificial intelligence generally include technologies such as sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, and mechatronics. The software technologies of artificial intelligence mainly include several major directions such as computer vision technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0027] Machine learning is an interdisciplinary subject involving multiple fields such as probability theory, statistics, approximation theory, convex analysis, and algorithm complexity theory. It specifically studies how a computer simulates or realizes human learning behaviors to acquire new knowledge or skills and reorganize the existing knowledge structure to continuously improve its own performance. Machine learning is the core of artificial intelligence and the fundamental way to make a computer intelligent, and its applications cover all fields of artificial intelligence. Machine learning and deep learning usually include technologies such as artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and rote learning.

[0028] Machine learning is the core of artificial intelligence and the fundamental way to endow computers with intelligence. Its applications cover all fields of artificial intelligence. Machine learning and deep learning generally include technologies such as artificial neural networks, belief networks, reinforcement learning, transfer learning, and inductive learning. An artificial neural network (ANN) abstracts the human brain neuron network from the perspective of information processing, establishes a simple model, and forms different networks according to different connection methods. A neural network is an operation model composed of a large number of nodes (or neurons) connected to each other. Each node represents a specific output function, called an activation function. The connection between every two nodes represents a weighted value for the signal passing through this connection, which is called a weight. This is equivalent to the memory of the artificial neural network. The output of the network varies depending on the connection method of the network, the weight value, and the activation function. Usually, the network itself is an approximation of a certain algorithm or function in nature, or it may be an expression of a logical strategy.

[0029] To facilitate the understanding of the technical solutions provided by the embodiments of the present disclosure, some key terms used in the embodiments of the present disclosure are explained here first:

[0030] Phosphene: An illusory light spot or pattern generated when the visual cortex is electrically stimulated, which is the basis of artificial vision.

[0031] Electrical stimulation simulator: A mathematical model that simulates the diffusion of electrical stimulation parameters in the cortex and outputs a predicted phosphene image.

[0032] Flexible electrode array: A micro device implanted in the visual cortex for recording neuron activities and applying electrical stimulation.

[0033] Receptive field: The smallest area on the retina where visual stimuli in a specific area can activate the cortical neurons.

[0034] Retinal mapping: The process in which visual information from different regions on the retina is spatially corresponding in the brain visual cortex, maintaining the spatial relationship between each point on the retina and the cortical neurons.

[0035] Local Field Potential (LFP): The signal of the electrical activity recorded by an electrode in a brain region, representing the activity of a neuron population.

[0036] The design concept of the embodiments of the present application is briefly introduced below.

[0037] Visual impairment is a common problem globally, especially among the blind population. The defects in visual perception severely limit daily life and information interaction. Currently, relevant technologies can use cortical electrical stimulation technology to stimulate the visual cortex of the brain to generate phosphenes for visual restoration.

[0038] However, the effectiveness of electrical stimulation is closely related to the electrical stimulation parameters and the accuracy of neural activity decoding. In practical applications, relevant technologies face the problem of insufficient precision in generating electrical stimulation parameters, resulting in poor visual perception restoration and reconstruction effects. In related technologies, the setting of electrical stimulation parameters usually highly depends on manual experience adjustment. For example, doctors or researchers repeatedly adjust electrical stimulation parameters such as voltage amplitude to match the fuzzy light spot positions reported by patients. This not only has cumbersome operations, a long parameter adjustment cycle, and high costs, but also has low accuracy of electrical stimulation parameters and is difficult to systematically adapt to the visual perception characteristics of different objects. Moreover, in related technologies, the neural decoding accuracy is low, and it is difficult to accurately map the neuron activities induced by electrical stimulation into the expected visual perception. For example, neural decoding cannot adapt to individual neural plasticity differences or environmental interference, resulting in the clarity, accuracy, and stability of visual reconstruction being difficult to meet actual requirements. In summary, due to the limitations of manual experience and the complexity of neural coding, the accuracy and generalization of electrical stimulation in related technologies are greatly reduced.

[0039] In view of the above problems, the embodiments of the present application provide an electrical stimulation method. This method obtains the image to be perceived of a target object and inputs the image to be perceived into a trained electrical stimulation model to obtain the electrical stimulation parameters output by the electrical stimulation model, which are used to perform corresponding electrical stimulation operations on the target object to restore the visual perception of the target object for the image to be perceived. The electrical stimulation generator in this electrical stimulation is obtained through the first-link training and the second-link training based on multiple sample images and corresponding sample neural signals. The first link represents the verification link among the electrical stimulation parameters, the simulated neural signals, and the reconstructed images, and the second link represents the verification link among the electrical stimulation parameters, the phosphene images, and the sample images, enabling the electrical stimulation model to greatly optimize the generation process of electrical stimulation parameters and improve the accuracy and generalization of visual perception restoration by electrical stimulation.

[0040] To further improve the performance of the electrical stimulation model, the embodiments of the present application train the electrical stimulation generator in the electrical stimulation model through the first link, including using an architecture similar to encoding-decoding to train the electrical stimulation generator and forcing the training and performance improvement of the electrical stimulation generator according to the decoder.

[0041] To further improve the performance of the electrical stimulation model, in the embodiments of the present application, the electrical stimulation generator in the electrical stimulation model is trained through a second link, and a brain state transition process is proposed. The spatio-temporal Gaussian distribution and electric field superposition are used to simulate the change of nerve signals after electrical stimulation, making the state transition process differentiable, so that backpropagation can be performed to train the electrical stimulation generator.

[0042] To further improve the efficiency and safety of visual restoration achieved by electrical stimulation, the electrical stimulation model in the embodiments of the present application further includes an information filtering network to filter out invalid visual information inputs and reduce the number of invalid electrical stimulations.

[0043] To further improve the efficiency of electrical stimulation, the embodiments of the present application also introduce a constraint penalty term into the loss function of the electrical stimulation model, so as to activate as many electrode channels to be activated as possible and improve the efficiency of the electrode activation corresponding to the receptive field area.

[0044] Some simple introductions will be made below to the application scenarios applicable to the technical solutions of the embodiments of the present application. It should be noted that the application scenarios introduced below are only used to illustrate the embodiments of the present application rather than to limit them. In the specific implementation process, the technical solutions provided by the embodiments of the present application can be flexibly applied according to actual needs.

[0045] The solution provided by the embodiments of the present application can be applicable to most application scenarios for visual reconstruction and restoration through cortical electrical stimulation, and is used to improve the accuracy, clarity and dynamic adaptability of visual perception restoration. For example, in the traffic intersection scenario, the blind need to identify the color of traffic lights or direction indicators. In the embodiments of the present application, through the electrical stimulation model, deep learning algorithms are used to analyze the environmental images in real time and generate electrical stimulation parameters adapted to the individual neural coding characteristics, so that the target object can restore the visual perception of the color of traffic lights or direction indicators and help the target object safely pass through the traffic intersection.

[0046] Of course, the method provided by the embodiments of the present application is not limited to the above application scenarios and can also be used in other possible application scenarios, which are not restricted by the embodiments of the present application. The functions that can be realized by each device in the above application scenarios will be described together in the subsequent method embodiments and will not be elaborated here too much.

[0047] Next, in combination with the above-described application scenarios, the method provided by the exemplary embodiment of the present application will be described with reference to the accompanying drawings. It should be noted that the above application scenarios are only shown for the convenience of understanding the spirit and principle of the present application, and the embodiments of the present application are not restricted in this regard.

[0048] In the embodiments of the present application, the electrical stimulation model mainly optimizes the generation process of cortical electrical stimulation parameters through artificial neural networks and deep learning algorithms, improves the accuracy and clarity of visual perception, and thus provides a more effective means of visual restoration for relevant groups such as the blind.

[0049] Therefore, before the electrical stimulation model is put into use, it is necessary to pre-collect a sample data set and train the model with the sample data set to make the electrical stimulation model converge. Next, for the convenience of describing the model application process, the training process of the electrical stimulation model will be introduced first:

[0050] Please refer to Figure 1 , which is a schematic diagram of the training process of an electrical stimulation model provided by the embodiments of the present application. The specific implementation process of this training process is as follows:

[0051] Step 101: Obtain the sample data set of the sample object.

[0052] In the embodiments of the present application, the sample data set of the sample object can be obtained through multiple sample images and the sample neural signals generated by the sample object in response to the multiple sample images.

[0053] Specifically, in the collection stage of the sample data set, the present application can show multiple sample pictures to the sample object (for example, animals such as monkeys), implant flexible electrodes in the visual cortex of the sample object (for example, mainly implanted in brain regions V1 and V2), record the neuronal activities generated by the sample object each time it looks at the sample pictures, and record its corresponding eye movement coordinates through an eye tracker. The present application can present each picture to the sample object multiple times to ensure that a sufficient amount of neural activity data is collected.

[0054] In a possible implementation manner, the present application can perform corresponding preprocessing operations on the collected sample data set (sample pictures and corresponding sample neural signals) respectively, so as to ensure that the input image data and neural signals are more consistent, reduce noise interference, and help the electrical stimulation model better extract useful visual features and signal features.

[0055] Specifically, for neural signals, the present application can perform preprocessing to remove noise and artifacts, thereby eliminating irrelevant noise and improving the learning efficiency of the model for neural features. Taking 1280-channel neural signals as an example, the average signal of all channels is subtracted from the neural signals of each channel to remove common noise and artifacts in multi-channel data. Further, for neural signals, the present application can also perform preprocessing such as band-pass filtering, downsampling, and normalization. For example, a band-pass filter is used to extract a specific frequency band (e.g., 1 - 300 Hz) of the original neural signal to obtain the Local Field Potential (LFP). The LFP signal after band-pass filtering is downsampled (e.g., from 20 kHz to 1 kHz, with a downsampling factor of 20), and the downsampled LFP signal is subjected to Min-Max Normalization.

[0056] Specifically, for image data, the present application can perform preprocessing such as cropping, grayscaling, and binarization to make images under different conditions have the same size and brightness range, avoiding the model being affected by irrelevant factors. For example, taking the eye movement coordinates obtained by the eye tracker as the central visual field point, a square area (e.g., a 10-degree * 10-degree area) below the coordinates of the central visual field point of the natural image is taken. In this way, the corresponding RGB image size will be cropped to 64 * 64 * 3. Further, the RGB image is converted into a grayscale image. For example, the pixel values of each RGB image channel (R, G, B) are multiplied by the corresponding weights and summed to obtain a single-channel grayscale image, and the image is segmented into foreground and background, with the foreground (value 255) and the rest as the background (value 0) to form a binarized image.

[0057] Step 102: Based on the sample data set, iteratively train the decoder to obtain a trained decoder.

[0058] In the embodiment of the present application, the decoder is used to decode the neural signals generated when the sample object views the sample image into corresponding image data. In this way, by training the decoder with a large number of sample images and sample neural signals, the decoder can learn the bidirectional mapping relationship between the neural signal pattern and the visual image features, thereby providing the encoding rule from image to neural signal for the subsequent electrical stimulation parameter generator.

[0059] Specifically, referring to Figure 2 shown is a schematic diagram of the training process of an electrical stimulation model provided by the embodiment of the present application. Figure 2The LFP-Image decoder in the upper-middle part consists of an LFP-Image converter and an Image-Image autoencoder. The LFP-Image converter is mainly composed of a fully connected neural network, which is used to convert the LFP signals in the sample dataset into intermediate images; the Image-Image autoencoder is composed of modules such as convolution, pooling, and upsampling, which is used to generate a reconstructed image for the LFP signal according to the intermediate image output by the LFP-Image converter.

[0060] In a possible implementation, referring to the above Figure 2 As shown, in the embodiment of the present application, the sample neural signals in the sample dataset are input into the LFP-Image decoder to obtain the reconstructed image data output by the decoder, and a loss function is constructed based on the difference between the reconstructed data and the sample image. When the loss function meets the preset convergence condition, a trained decoder is obtained.

[0061] Specifically, taking the LFP-Image decoder as an example, the present application inputs the time series of the LFP signal, obtains the corresponding RGB image output by the decoder, and constructs the loss function as the pixel-level mean squared error (MSE). The forward process of the decoder is as follows:

[0062] y i ' = D 2 (D 1 (x i ))

[0063] Among them, D 1 and D 2 are the parameters of the LFP-Image converter and the Image-Image autoencoder respectively, x i is the LFP of the i-th sample, and y i ' is the corresponding decoded picture.

[0064] In a possible implementation, in the embodiment of the present application, the adaptive moment estimation (Adam) optimizer can be used in each iteration training to minimize the loss function until the loss function is minimized, or when after multiple iterations, the loss function no longer significantly decreases, or reaches the preset maximum number of iterations, the training process stops, and a trained decoder is obtained. In this way, the decoder is optimized by minimizing the loss function, so that the data representation ability of the decoder reaches the best.

[0065] Specifically, the loss function in the present application is as follows:

[0066]

[0067] Among them, y i represents the real image of the i-th sample, and y i ' is the corresponding decoded image, and N is the batch size.

[0068] It is worth mentioning that the loss function in the embodiments of the present application can also adopt the Structural Similarity Index Measure (SSIM), perceptual loss function, adversarial loss function, etc. The optimizer can also adopt optimizers such as Root Mean Square Propagation (RMSProp) and Nesterov-accelerated Adaptive Moment Estimation (Nadam). The embodiments of the present application do not make specific limitations in this regard.

[0069] Step 103: Based on the sample data set, combine the trained decoder to perform the first-link training and the second-link training on the electrical stimulation generator to obtain a trained electrical stimulation model.

[0070] In the embodiments of the present application, the training process of the electrical stimulation generator consists of two forward training processes of the first link and the second link. Among them, the first link is the verification link between the electrical stimulation parameters, the simulated nerve signal, and the reconstructed image, representing the reverse mapping process of "electrical stimulation parameters to nerve signal to reconstructed image"; the second link is the verification link between the electrical stimulation parameters, the phosphene image, and the sample image, representing the direct mapping process of "electrical stimulation parameters to phosphene image to image matching". Through the first link and the second link, the present application can enable the electrical stimulation generator to reverse-infer the generation strategy of "target image to nerve signal to adapted electrical stimulation parameters", ensuring that the electrical stimulation parameters can accurately generate nerve activities that match the target visual perception, thereby improving the accuracy and cross-scene generalization of the electrical stimulation.

[0071] In a possible implementation manner, the embodiments of the present application will obtain the total loss function of the electrical stimulation generator through the first loss function corresponding to the first link and the second loss function corresponding to the second link, and thus adjust the parameters of the electrical stimulation generator through the loss value of the total loss function until the loss value meets the preset convergence condition to obtain a trained electrical stimulation generator.

[0072] Specifically, the present application can sum the losses of the first link and the second link respectively to obtain the total loss function, and minimize the total loss function to optimize the model and obtain a trained electrical stimulation model.

[0073] Specifically, referring to Figure 2 as shown, Figure 2The lower part in [description] is the electrical stimulation generator of this application. The electrical stimulation generator is used to receive image data as input and output corresponding electrical stimulation parameters, such as the voltage amplitude of the applied electrical stimulation corresponding to each electrode channel, and the number of pulses during the application of the electrical stimulation, etc.

[0074] In a possible implementation manner, this application processes the sample image through the electrical stimulation generator trained this time, generates corresponding sample electrical stimulation parameters, and uses the electrical stimulation simulator to simulate the electrical stimulation with the sample electrical stimulation parameters to obtain the simulated neural signals corresponding to the brain state transition caused by the electrical stimulation. Then, the trained decoder is used for image reconstruction to obtain the reconstructed image corresponding to the simulated neural signals, and it is compared with the original input sample image to construct the first loss function for optimizing the electrical stimulation generator.

[0075] Specifically, referring to the above Figure 2 As shown, the first link training in the embodiment of this application is as follows: Input the natural image used in this training, and generate corresponding electrical stimulation parameters (for example, the voltage amplitude values of 1280 channels) through the neural network with a convolutional architecture in the electrical stimulation parameter generator. Use an electrical stimulation simulator similar to the current diffusion process to simulate the effect of the electrical stimulation on the electrodes and the brain state transition caused by the electrical stimulation. Input the simulated LFP signal corresponding to the brain state transition into the LFP-Image decoder to decode the corresponding reconstructed image. The LFP-Image decoder uses the model parameters of the trained and fixed LFP-Image decoder.

[0076] Specifically, the forward process of the first link is as follows:

[0077] y i ' = D 2 (D 1 (T(G(y i ))))

[0078] Among them, D 1 and D 2 are the parameters of the trained LFP-Image converter and the Image-Image autoencoder respectively, and are kept fixed;

[0079] G is the parameter of the electrical stimulation parameter generator, T is the brain state transition process, y i represents the real picture of the i-th sample, and y i ' represents the decoded picture corresponding to it.

[0080] Among them, the input of the brain state transition process T is the voltage amplitude of the applied electrical stimulation at the corresponding site of each electrode channel. This voltage amplitude will first be divided by the electrode impedance to convert it into the current value A. Assuming that the impedances of different channels are equal, that is, A = μG(yi ), where the value of μ depends on the electrode impedance. The output of the brain state transition process T is the brain state h within a period of time after the application of the electrical stimulation, that is, the current values at all times within that period of time for all recording channels, as shown specifically below:

[0081]

[0082] where, I k (x, y, z, t) represents a spatio-temporal Gaussian distribution, which is used to describe the spatio-temporal distribution of the electric field strength caused after the application of the electrical stimulation to the k-th channel. Among them, the parameters x, y, z represent the three-dimensional coordinates of space, and the parameter t represents time.

[0083] A k and (x k , y k , z k ) are respectively the current value and the spatial coordinates generated by the electrical stimulation at the corresponding site of the k-th channel;

[0084] represents the diffusion process of the spatio-temporal distribution (Gaussian distribution) of the electric field strength over time. Among them, σ(0) is the initial standard deviation, D is the diffusion rate, and t is time.

[0085] In summary, I k (x, y, z, t) represents the spatio-temporal distribution of the electric field strength caused after the application of the electrical stimulation at a single channel site. Therefore, the brain state h within a period of time after the overall electrical stimulation can be described as the superposition of all I k (x, y, z, t), as shown below:

[0086]

[0087] where, t represents time, and j represents the channel identifier. In this way, the current value at any time within a specific time for any recording channel can be calculated through this expression.

[0088] Input the calculated h into the LFP-Image decoder with fixed parameters to obtain the corresponding reconstructed picture y i '.

[0089] Based on the difference between the reconstructed picture output by the decoder and the original input natural image, the loss function corresponding to the training process of this first link can be constructed, as shown below:

[0090]

[0091] where, y i represents the real picture of the i-th sample, and y i ' is the corresponding decoded reconstructed picture, and N is the batch size.

[0092] In a possible implementation, the embodiment of the present application generates sample electrical stimulation parameters corresponding to a sample image through an electrical stimulation generator, and performs phosphene simulation on the sample electrical stimulation parameters and corresponding electrode site information through a phosphene simulator to obtain a corresponding phosphene image. The second loss function corresponding to the second link is determined based on the difference between the phosphene image and the sample image.

[0093] Specifically, referring to the above Figure 2 As shown, the training of the second link in the embodiment of the present application is as follows: Input a natural picture in the sample dataset, and generate electrical stimulation parameters through an electrical stimulation parameter generator (shared with the first link). Input the electrical stimulation parameters of all channels and the three-dimensional coordinates of the corresponding electrode sites into the phosphene simulator to generate a corresponding simulated phosphene picture.

[0094] Specifically, the forward process of the second link is as follows:

[0095] p i = S(μG(y i ))

[0096] Among them, G represents the parameters of the electrical stimulation parameter generator;

[0097] S represents the parameters of the phosphene simulator. In a possible implementation, the present application can be set according to biological prior knowledge and kept fixed during the training process;

[0098] μ represents the electrode impedance coefficient. The higher the electrode impedance coefficient, the smaller the actual stimulation current (the voltage amplitude needs to be compensated), and the voltage can be adjusted by dynamically measuring the impedance.

[0099] y i represents the real picture of the i-th sample, and p i represents the generated phosphene picture.

[0100] In this way, based on the difference between the phosphene picture simulated and generated by the phosphene simulator and the original input sample picture, the loss function corresponding to the training of the second link can be constructed, which is specifically as follows:

[0101]

[0102] Among them, represents the picture data after grayscale and binarization processing of the i-th sample picture, and p i is the corresponding generated phosphene picture, and N is the batch size.

[0103] In summary, in this application, the iterative training of the electrical stimulation generator is jointly performed through the first link and the second link, and the loss functions of the two links can be summed to obtain the total loss function. Then, the total loss function is minimized by means of an optimizer until the total loss function is minimized, or when the total loss function no longer significantly decreases after multiple iterations, or reaches the preset maximum number of iterations, the training process stops, and a trained electrical stimulation generator is obtained.

[0104] In a possible implementation manner, in order to improve the activation rate of electrode channels and enhance the visual perception efficiency, the embodiments of this application consider that the higher the number of electrode channels being stimulated, the higher the probability that this piece of electrode is activated to generate corresponding visual perception, and when the number of stimulated channels of a piece of electrode is less than a certain threshold, the electrical stimulation will be ineffective. Therefore, when training the electrical stimulation generator in this application, in addition to jointly performing iterative training through the first and second links, a constraint penalty term (i.e., a constraint penalty function) for restricting the number of stimulated electrode channels can be added to try to increase the number of electrode channels to be activated as much as possible, so as to ensure that as many sites on the same piece of electrode are stimulated simultaneously. For example, if a piece of electrode has 128 channels, it is necessary to increase the number of stimulated channels as much as possible, and ensure that the points with similar spatial positions on the channels are stimulated simultaneously as much as possible.

[0105] Specifically, the constraint penalty term in the embodiments of this application is as follows:

[0106]

[0107] Among them, G are the model structure parameters of the electrical stimulation generator, μ is the electrode impedance coefficient, y i represents the real picture of the i-th sample, and k represents the electrode channel serial number;

[0108] ε is the attenuation coefficient, γ is the penalty coefficient. The smaller the attenuation coefficient, the denser the electrode channels, and the larger the penalty coefficient, the denser the electrode channels. Thus, the sparsity of the electrode channels is jointly controlled by the attenuation coefficient and the penalty coefficient to ensure a sufficient number of electrical stimulation channels and ensure the simulation effect of visual perception.

[0109] N is the batch size.

[0110] In this way, after introducing the constraint penalty term, the total loss function of the embodiments of this application is the sum of the loss functions of the first link, the second link, and the constraint penalty term. Therefore, in each iteration of training the electrical stimulation generator in this application, an optimizer can be used to minimize the following loss function:

[0111] L = αL gen + βL pho + L con

[0112] Among them, α and β are the weight coefficients corresponding to the loss function terms, which can be set according to specific application scenarios, and the embodiments of the present application do not make specific limitations on this.

[0113] In a possible implementation manner, the electrical stimulation model in the present application further includes an information filtering network, which is used to filter out external invalid visual inputs and determine to perform corresponding electrical stimulation only on pictures containing key information. Therefore, the training process of the electrical stimulation model in the embodiments of the present application further includes: iteratively training the information filtering network through multiple sample images and key information labels representing whether each sample image contains key information, and obtaining a trained information filtering network when the loss function of the information filtering network meets the preset convergence condition.

[0114] Specifically, referring to the Figure 2 as shown, the Figure 2 lateral geniculate nucleus (LGN) in the lower half shown herein is the information filtering network in the embodiments of the present application, which is used to filter external visual inputs of the model, only select natural pictures containing key information for electrical stimulation, and do not perform electrical stimulation on background pictures of irrelevant information, thereby performing certain filtering and screening on the visual signal input and improving the effectiveness and efficiency of electrical stimulation.

[0115] In a possible implementation manner, the network structure of LGN in the embodiments of the present application is composed of ResNet integrated with an attention mechanism (Convolutional Block Attention Module, CBAM). LGN takes real pictures as inputs and outputs a judgment result on whether the visual signal contains key information.

[0116] Specifically, the forward process of LGN is as follows:

[0117]

[0118] Among them, y i represents the i-th real picture sample, is the probability of judging whether it contains key information.

[0119] The loss function of LGN is as follows:

[0120]

[0121] Among them, z i is the label of whether the i-th sample contains key information, is the probability of judging whether it contains key information, and N is the batch size.

[0122] Thus, in each iteration of training LGN, the present application may use optimization means such as the Adam optimizer to minimize the loss function so that LGN reaches a preset convergence condition.

[0123] After obtaining the trained electrical stimulation model, the embodiment of the present application uses macaques and other animals for experiments, and verifies that the electrical stimulation model has achieved international leading level in the DMS task by observing and counting the visual perception of macaques. The embodiment of the present application can process the image to be perceived by the target object through the electrical stimulation model to generate corresponding electrical stimulation parameters for performing electrical stimulation operations on the target object to restore the target object's visual perception of the image to be perceived.

[0124] refer to Figure 3 FIG. 1 is a flow chart of an electrical stimulation method provided in an embodiment of the present application. The specific implementation process of the method is as follows:

[0125] Step 301: Acquire the image to be sensed of the target object.

[0126] In the embodiment of the present application, after the electrical stimulation model is trained, the real-time image to be perceived of the target object can be processed by the electrical stimulation model to obtain corresponding electrical stimulation parameters.

[0127] In one possible implementation, the present application can obtain the eye movement coordinates of the target object in real time through an eye tracker and use it as the central field of view point, thereby taking the square area below the coordinates of the central field of view point corresponding to the natural image acquired by the camera as the image to be perceived, that is, the input data of the electrical stimulation model.

[0128] Step 302: input the image to be sensed into the trained electrical stimulation model to obtain output electrical stimulation parameters.

[0129] In the embodiment of the present application, the trained electrical stimulation model can be processed by the electrical stimulation generator for the input image to be sensed to generate corresponding electrical stimulation parameters. The processing process of the image to be sensed during the training process is the same as the processing process of the sample image during the actual application process, so the process can refer to the detailed introduction of the aforementioned training process, and no further elaboration is given here.

[0130] In a possible implementation, the trained electrical stimulation model also includes an information filtering network LGN. After the image to be perceived is input into the electrical stimulation model, the present application can also determine through LGN whether the target image contains key information, thereby determining whether electrical stimulation is needed. When it is determined that it contains key information, the target object is subjected to corresponding electrical stimulation operations through electrical stimulation parameters so that the target object can restore visual perception of the image to be perceived.

[0131] In one possible implementation, reference Figure 4The following is a schematic flow chart of model training and model application provided by an embodiment of the present application. The Figure 4 includes an offline process (model training process) and an online process (model application process). The offline process includes a data acquisition process (by implanting cortical flexible electrodes in the cerebral cortex of the target object, collecting the corresponding neural signals generated in the cerebral cortex when natural images are presented to the target object), data preprocessing (denoising and artifact removal, band-pass filtering, downsampling, and normalization of neural signals, and at the same time cropping, grayscale conversion, and binarization of natural images), and model training (using the preprocessed neural signals and natural image data, converting the neural signals into reconstructed images through a decoder, and combining LGN as an information filtering network to train an electrical stimulation generation encoder, and finally obtaining an electrical stimulation model). After the electrical stimulation model is trained, the model is applied to the online process. The online process includes obtaining visual input (capturing target image data through a camera and an eye tracker), generating electrical stimulation strategies and parameters (outputting electrical stimulation parameters based on an electrical stimulation generator and screening effective images and electrical stimulation parameters through LGN), performing electrical stimulation operations (applying optimized electrical stimulation parameters), and finally enabling the target object to generate visual perception and optimizing the visual restoration system through the behavioral feedback loop of the target object. In this way, through the cooperation of the offline and online stages, an accurate mapping from neural signals to electrical stimulation is achieved, improving the efficiency and accuracy of visual restoration.

[0132] Please refer to Figure 5 As shown, based on the same technical concept, an embodiment of the present application also provides a computer device 50. In one embodiment, the computer device may be a visual restoration system. The computer device is as Figure 5 shown, including a memory 501, a communication module 503, and one or more processors 502.

[0133] The memory 501 is used to store computer programs executed by the processor 502. The memory 501 may mainly include a program storage area and a data storage area. Among them, the program storage area may store an operating system and programs required to run an instant messaging function, etc.; the data storage area may store various instant messaging information and operation instruction sets, etc.

[0134] The memory 501 can be a volatile memory, such as a random-access memory (RAM); the memory 501 can also be a non-volatile memory, such as a read-only memory, a flash memory, a hard disk drive (HDD), or a solid-state drive (SSD); or the memory 501 is any other medium that can be used to carry or store the desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto. The memory 501 can be a combination of the above memories.

[0135] The processor 502 can include one or more central processing units (CPUs) or be a digital processing unit, etc. The processor 502 is used to implement the above-mentioned electrical stimulation method when calling the computer program stored in the memory 501.

[0136] The communication module 503 is used to communicate with data acquisition devices such as an eye tracker and a camera, terminal devices, or other servers.

[0137] In the embodiments of the present application, the specific connection medium between the memory 501, the communication module 503, and the processor 502 is not limited. In the embodiments of the present application Figure 5 it is described that the memory 501 and the processor 502 are connected through a bus 504, and the bus 504 is described by a thick line in Figure 5 The connection manners between other components are only for illustrative purposes and are not limited thereto. The bus 504 can be divided into an address bus, a data bus, a control bus, etc. For the sake of convenience of description, Figure 5 only a thick line is used to describe it in

[0138] The memory 501 stores a computer storage medium, and the computer storage medium stores computer-executable instructions for implementing the electrical stimulation method of the embodiments of the present application. The processor 502 is used to execute the electrical stimulation methods of the above embodiments.

[0139] Based on the same inventive concept, the embodiments of the present application also provide a storage medium that stores a computer program. When the computer program runs on a computer, it causes the computer to execute the steps in the electrical stimulation method according to various exemplary embodiments of the present application described above in this specification.

[0140] In some possible embodiments, aspects of the electrical stimulation method provided by the present application can also be implemented in the form of a computer program product, which includes a computer program. When the program product runs on a computer device, the computer program is used to cause the computer device to execute the steps in the electrical stimulation method according to various exemplary embodiments of the present application described above in this specification. For example, the computer device can execute the steps of each embodiment.

[0141] The program product can adopt any combination of one or more readable media. The readable media can be a readable signal medium or a readable storage medium. The readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the above. More specific examples (non-exhaustive list) of the readable storage medium include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0142] The program product of the embodiments of the present application can adopt a portable compact disk read-only memory (CD-ROM) and include a computer program, and can run on a computer device. However, the program product of the present application is not limited to this. In the present application, the readable storage medium can be any tangible medium that contains or stores a program, and the computer program included therein can be used by or in combination with a command execution system, apparatus, or device.

[0143] The readable signal medium can include a data signal propagated in a baseband or as part of a carrier wave, which carries a readable computer program. Such a propagated data signal can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The readable signal medium can also be any readable medium other than the readable storage medium, and this readable medium can send, propagate, or transmit a program for use by or in combination with a command execution system, apparatus, or device.

[0144] The computer program included on the readable medium can be transmitted by any suitable medium, including but not limited to wireless, wired, optical fiber, RF, etc., or any suitable combination of the above.

[0145] The computer program for performing the operations of the present application can be written in any combination of one or more programming languages. The programming languages include object-oriented programming languages, such as Java, C++, etc., and also include conventional procedural programming languages, such as the "C" language or similar programming languages.

[0146] It should be noted that although several units or subunits of the device are mentioned in the above detailed description, this division is merely exemplary and not mandatory. In fact, according to the embodiments of the present application, the features and functions of two or more units described above can be embodied in one unit. Conversely, the features and functions of one unit described above can be further divided and embodied by multiple units.

[0147] In addition, although the operations of the method of the present application are described in a specific order in the drawings, this does not require or imply that these operations must be performed in that specific order, or that all the operations shown must be performed to achieve the desired result. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step for execution, and / or one step may be decomposed into multiple steps for execution.

[0148] Those skilled in the art should understand that the embodiments of the present application can be provided as a method, a system, or a computer program product. Therefore, the present application can take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk memories, CD-ROMs, optical memories, etc.) that contain computer-usable program code.

[0149] Although the preferred embodiments of the present application have been described, those skilled in the art can make additional changes and modifications once they learn the basic creative concept. Therefore, the appended claims are intended to be construed to include the preferred embodiments as well as all changes and modifications falling within the scope of the present application.

[0150] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalent technologies, the present application is also intended to include these changes and modifications.

Claims

1. An electrical stimulation method, characterized in that: The method comprises: Acquire a to-be-perceived image of a target object, and input the to-be-perceived image into a trained electrical stimulation model; Obtaining electrical stimulation parameters output by the electrical stimulation model; the electrical stimulation parameters are used to perform corresponding electrical stimulation operations on the target object to restore the target object's visual perception of the image to be perceived; Wherein, the electrical stimulation model includes an electrical stimulation generator, which is obtained by first link training and second link training based on a sample data set; the sample data set includes multiple sample images and corresponding sample neural signals, and the sample neural signals are generated by the sample object in response to the multiple sample images; the first link represents the verification link between electrical stimulation parameters, simulated neural signals and reconstructed images, and the second link represents the verification link between electrical stimulation parameters, light hallucination images and sample images.

2. The method according to claim 1, characterized in that The electrical stimulation model further includes an information filtering network. After the image to be sensed is input into the trained electrical stimulation model, the method further includes: Based on the information filtering network, determining whether the image to be sensed contains key information; When it is determined that the image to be perceived contains the key information, a corresponding electrical stimulation operation is performed on the target object based on the electrical stimulation parameters.

3. The method according to claim 1, characterized in that The electrical stimulation model is trained based on the following method: Get the sample data set of the sample object; Based on the sample data set, iteratively train the decoder to obtain a trained decoder; the decoder is used to generate a corresponding reconstructed image based on the sample neural signal; Based on the sample data set, the first link training and the second link training are performed on the electrical stimulation generator in combination with the trained decoder to obtain a trained electrical stimulation model.

4. The method according to claim 3, characterized in that The performing first link training on the electrical stimulation generator comprises: Based on the electrical stimulation generator, generating sample electrical stimulation parameters corresponding to the sample image; Based on the trained decoder, the simulated neural signal of the sample electrical stimulation parameters is image reconstructed to obtain a reconstructed image; the simulated neural signal represents the brain state transition generated by the electrical stimulation simulator performing electrical stimulation simulation based on the sample electrical stimulation parameters; Based on the difference between the reconstructed image and the sample image, a first loss function corresponding to the first link is determined.

5. The method according to claim 3, characterized in that The performing second link training on the electrical stimulation generator comprises: Based on the electrical stimulation generator, generating sample electrical stimulation parameters corresponding to the sample image; Based on a phosphene simulator, phosphene simulation is performed on the sample electrical stimulation parameters and electrode site information to obtain a corresponding phosphene image; the electrode site information is the position information of the electrode site on the electrode channel corresponding to the corresponding sample electrical stimulation parameters; Based on the difference between the optical illusion image and the sample image, a second loss function corresponding to the second link is determined.

6. The method according to claim 3, characterized in that The electrical stimulation model further includes an information filtering network, and the training process of the electrical stimulation model further includes: Iteratively training the information filtering network based on the multiple sample images and the key information label corresponding to each sample image; the key information label indicates whether the corresponding sample image contains key information; When the loss function of the information filtering network meets a preset convergence condition, a trained information filtering network is obtained.

7. The method according to claim 3, characterized in that: The performing first link training and second link training on the electrical stimulation generator further includes: obtaining a total loss function of the electrical stimulation generator based on a first loss function of the first link and a second loss function of the second link; Based on the loss value of the total loss function, the parameters of the electrical stimulation generator are adjusted until the loss value meets a preset convergence condition, thereby obtaining a trained electrical stimulation generator.

8. The method according to claim 7, characterized in that: The total loss function also includes a constraint penalty function, which is used to constrain the activation number of electrode channels.

9. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 8 are implemented.

10. A computer storage medium having computer program instructions stored thereon, characterized in that: When the computer program instructions are executed by a processor, the steps of the method according to any one of claims 1 to 8 are implemented.

11. A computer program product comprising computer program instructions, characterized in that When the computer program instructions are executed by a processor, the steps of the method according to any one of claims 1 to 8 are implemented.

Images