Pulse stimulation method, apparatus, electronic device, and storage medium

Abstract

The application relates to the technical field of medical devices, and discloses a pulse stimulation method and device, an electronic device and a storage medium. The method comprises the following steps: acquiring a pulse sequence corresponding to an external scene; inputting the pulse sequence into a pre-trained pulse neural network to acquire a response prediction result; wherein the response prediction result is a predicted action potential response of neurons in a primary visual cortex to the external scene; generating a corresponding stimulation pulse according to the response prediction result, and transmitting the stimulation pulse to the neurons in the primary visual cortex or the ganglion cells of the retina. By acquiring the pulse sequence corresponding to the external scene and based on the pulse sequence, the action potential response of the neurons is predicted by using the pulse neural network, the humanization of the prediction process is improved, the effect of the visual prosthesis on the function replacement of the retina is greatly improved, the response prediction is performed based on the pulse sequence, the function replacement of the visual prosthesis on the retina is realized by using a full-pulse processing algorithm, and the power consumption of the visual prosthesis is greatly reduced.

Description

Technical Field

[0001] This application relates to the field of medical devices, and in particular to a pulse stimulation method, apparatus, electronic device, and storage medium. Background Technology

[0002] More than 100 million people worldwide suffer from age-related macular degeneration (AMD) and retinitis pigmentosa (RP), two degenerative vision diseases. AMD and RP patients experience blurred vision due to the loss of numerous cone and rod cells in the retina. The only difference between the two is the location of the lost cone and rod cells and the order of cell apoptosis. Currently, existing drug and gene therapy treatments have limited effectiveness for severe AMD and RP patients. With the development of optogenetics, optogenetic-based visual prostheses have become an important means of treating these two diseases. In a normally functioning retina, cone and rod cells effectively perceive external images and then generate neural electrical signals to encode those images. Bipolar cells and other cell layers process these neural electrical signals and then transmit the processed neural codes to the cerebral cortex. In contrast, the retina using a visual prosthesis utilizes algorithms to replace the retina's signal processing capabilities to perceive and process images. The processed signals are then transmitted to the primary visual cortex via light stimulation, where visual perception is formed.

[0003] The commonly used processing algorithm for visual prostheses is to use a camera to acquire external images, then perform grayscale conversion, contrast enhancement, edge extraction, and pixel merging on the images to obtain a simplified image after pixel merging. Then, light stimulation signals are generated based on the simplified image and transmitted to the primary visual cortex.

[0004] However, the method of replacing the functions of cell layers such as cones and rods by generating simplified images after pixel synthesis has poor imaging results in practical applications and consumes a lot of power. Summary of the Invention

[0005] The main objective of this application is to provide a pulse stimulation method, apparatus, electronic device, and storage medium, which aims to improve the ability and effect of visual prostheses to replace retinal cell layers in practical applications while reducing the power consumption of visual prostheses.

[0006] To achieve the above objectives, embodiments of this application provide a pulse stimulation method, comprising: acquiring a pulse sequence corresponding to an external scene; inputting the pulse sequence into a pre-trained spiking neural network to obtain a response prediction result; wherein the response prediction result is a predicted action potential response of neurons in the primary visual cortex to the external scene; generating a corresponding stimulation pulse based on the response prediction result, and transmitting the stimulation pulse to ganglion cells in the primary visual cortex or retina.

[0007] To achieve the above objectives, this application also proposes a pulse stimulation device, comprising: an acquisition module for acquiring a pulse sequence corresponding to an external scene; a prediction module for inputting the pulse sequence into a pre-trained spiking neural network to acquire a response prediction result; wherein the response prediction result is the predicted action potential response of neurons in the primary visual cortex to the external scene; and a stimulation module for generating a corresponding stimulation pulse based on the response prediction result and transmitting the stimulation pulse to ganglion cells in the primary visual cortex or retina.

[0008] To achieve the above objectives, embodiments of this application also propose an electronic device, the device comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the pulse stimulation method as described above.

[0009] To achieve the above objectives, embodiments of this application also propose a computer-readable storage medium storing a computer program that, when executed by a processor, implements the pulse stimulation method described above.

[0010] The pulse stimulation method provided in this application, when acquiring an external scene, directly obtains the pulse sequence corresponding to the external scene as input for predicting the action potential response of neurons in the primary visual cortex. Then, the acquired pulse sequence is input into a pre-trained spiking neural network. The spiking neural network predicts the action potential response of neurons in the primary visual cortex to the external scene based on the pulse sequence and obtains the response prediction result. Then, based on the response prediction result, a corresponding stimulation pulse is generated and transmitted to the ganglion cells of the primary visual cortex or retina, thereby replacing the function of the retina. By acquiring the pulse sequence corresponding to the external scene and using a spiking neural network to predict the action potential response of neurons based on the acquired pulse sequence, the anthropomorphism of the action potential response prediction process is improved, greatly enhancing the effect of the visual prosthesis in replacing retinal function. Based on the pulse sequence for response prediction, the full-pulse processing algorithm achieves the functional replacement of the retina by the visual prosthesis, significantly reducing the power consumption of the visual prosthesis. Attached Figure Description

[0011] One or more embodiments are illustrated by way of example with reference to the accompanying drawings, and these illustrative descriptions do not constitute a limitation on the embodiments.

[0012] Figure 1 This is a flowchart of the pulse stimulation method in the embodiments of this application;

[0013] Figure 2 This is a schematic diagram of the preset path in the embodiments of this application;

[0014] Figure 3 This is a schematic diagram of membrane potential changes in the embodiments of this application;

[0015] Figure 4 This is a schematic diagram of the prediction effect of the spiking neural network in the embodiments of this application;

[0016] Figure 5 This is a schematic diagram of the pulse stimulation device in another embodiment of this application;

[0017] Figure 6 This is a schematic diagram of the structure of an electronic device in another embodiment of this application. Detailed Implementation

[0018] As the background technology shows, the method of generating simplified images through pixel synthesis to replace the functions of cellular layers such as cones and rods has poor imaging effects in practical applications and consumes a lot of power. Therefore, how to further improve the effectiveness of visual prostheses in replacing retinal functions and reduce the power consumption during the use of visual prostheses is an urgent problem that needs to be solved.

[0019] To address the aforementioned issues, embodiments of this application provide a pulse stimulation method, comprising: acquiring a pulse sequence corresponding to an external scene; inputting the pulse sequence into a pre-trained spiking neural network to obtain a response prediction result; wherein the response prediction result is the predicted action potential response of neurons in the primary visual cortex to the external scene; generating a corresponding stimulation pulse based on the response prediction result, and transmitting the stimulation pulse to ganglion cells in the primary visual cortex or retina.

[0020] The pulse stimulation method provided in this application, when acquiring an external scene, directly obtains the pulse sequence corresponding to the external scene as input for predicting the action potential response of neurons in the primary visual cortex. Then, the acquired pulse sequence is input into a pre-trained spiking neural network. The spiking neural network predicts the action potential response of neurons in the primary visual cortex to the external scene based on the pulse sequence and obtains the response prediction result. Then, based on the response prediction result, a corresponding stimulation pulse is generated and transmitted to the ganglion cells of the primary visual cortex or retina, thereby replacing the function of the retina. By acquiring the pulse sequence corresponding to the external scene and using a spiking neural network to predict the action potential response of neurons based on the acquired pulse sequence, the anthropomorphism of the action potential response prediction process is improved, greatly enhancing the effect of the visual prosthesis in replacing retinal function. Based on the pulse sequence for response prediction, the full-pulse processing algorithm achieves the functional replacement of the retina by the visual prosthesis, significantly reducing the power consumption of the visual prosthesis.

[0021] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the various embodiments of this application will be described in detail below with reference to the accompanying drawings. However, those skilled in the art will understand that many technical details have been provided in the various embodiments of this application to help readers better understand this application. However, the technical solutions claimed in this application can be implemented even without these technical details and various changes and modifications based on the following embodiments. The division of the various embodiments below is for the convenience of description and should not constitute any limitation on the specific implementation of this application. The various embodiments can be combined with and referenced by each other without contradiction.

[0022] The implementation details of the pulse stimulation method described in this application will be specifically described below with reference to specific embodiments. The following implementation details are provided for ease of understanding only and are not necessary for implementing this solution.

[0023] The first aspect of this application provides a pulse stimulation method, the flow of which is referenced below. Figure 1 In some embodiments, the pulse stimulation method is applied to the control chip of a visual prosthesis or to an external terminal capable of communicating with the visual prosthesis control chip, such as an electronic device like a computer or mobile phone. This embodiment uses its application in the control chip of a visual prosthesis as an example. The pulse stimulation method includes the following steps:

[0024] Step 101: Obtain the pulse sequence corresponding to the external scene.

[0025] Specifically, after the visual prosthesis is put into use, the control chip instructs the image acquisition device, such as a Dynamic Vision Sensor (DVS) camera, to acquire external scene information and generate a pulse sequence corresponding to the external scene based on the acquired external scene information.

[0026] It should be noted that when collecting external scene information, an image of the external scene at a single moment can be collected, or a video of the external scene composed of multiple images of the external scene over a certain period of time can be collected. This embodiment does not limit the specific method of collecting external scene information.

[0027] In one example, the control chip in the visual prosthesis acquires a pulse sequence corresponding to the external scene, including: pixel detection of the external scene; generating corresponding pulse events for pixels whose pixel values ​​change in the external scene; wherein, the pulse event includes the pixel value change time, pixel coordinates, and pixel value change polarity; and generating a pulse sequence based on all pulse events of the external scene. Specifically, the control chip instructs the DVS camera to receive information about the external scene, acquire the image or video corresponding to the currently acquired external scene, and then the DVS camera performs pixel detection on the external scene, acquires the pixel value changes of each pixel in the external scene, and then generates a corresponding pulse event for each pixel whose pixel value changed based on the pixel change polarity, pixel coordinates, and pixel value change time. The pixel change polarity refers to the direction of the pixel value change; for example, if the pixel value increases by 5, the pixel change polarity is 1, and if the pixel value decreases by 5, the pixel change polarity is 0. Then, based on the pulse events corresponding to all pixels in the current external scene whose pixel values ​​have changed, a pulse sequence corresponding to the external scene is generated. By using a DVS camera to record external scene signals in a pulse coding manner, and focusing only on pixels whose pixel values ​​change, generating corresponding pulse events for them, and thus obtaining the pulse sequence corresponding to the external scene, the power consumption and bandwidth in the process of obtaining the pulse sequence corresponding to the external scene are greatly reduced.

[0028] In another example, before the visual prosthesis acquires the pulse sequence corresponding to the external scene, the process includes: training an initial spiking neural network based on data from the response database of the primary visual cortex to acquire a pre-trained spiking neural network; wherein the response database includes multiple images, the pulse sequence corresponding to each image, and the action potential response corresponding to each image.

[0029] Specifically, one of the core components of a visual prosthesis is the processing algorithm for external scene signals. Therefore, before the visual prosthesis is put into use, an initial spiking neural network needs to be trained based on data from a pre-created response database of the primary visual cortex to obtain a dynamic vision sensor (SNN) capable of accurately replacing retinal function. The response database of the primary visual cortex can include a public database recorded from the primary visual cortex of monkeys using a 32-channel microelectrode array. The public database contains 7250 images and the number of action potential responses from 166 primary visual cortex neurons for each image. To further improve the accuracy of the action potential response recording, each image was shown to the monkeys four times, and the number of action potential responses obtained for each image during the four viewings was recorded. Then, a corresponding pulse sequence was generated for each image, and the pulse sequence and the corresponding image were associated and stored.

[0030] When training the initial spiking neural network, the number of action potential responses and the pulse sequence corresponding to each image are used sequentially, or the number of action potential responses and the pulse sequence corresponding to a subset of images are selected for training. During training, the number of action potential responses corresponding to the selected images is used as the supervision signal, and the corresponding pulse sequence is used as the input for training, until the initial spiking neural network's prediction of the action potential response to the input pulse sequence is consistent with the supervision signal, or the error with the supervision signal is less than a certain threshold. At this point, the initial spiking neural network is considered to have completed training, and a trained spiking neural network is obtained.

[0031] It should be noted that before training the initial spiking neural network based on the data in the primary visual cortex response database, the process includes: smoothly moving pre-acquired images along a preset path on a preset display, acquiring pulse sequences generated by each image during the movement; and for each pulse sequence, associating and storing the pulse sequence with the corresponding image. Specifically, when creating the primary visual cortex response database, after obtaining the common database recorded from the monkey's primary visual cortex, it is necessary to acquire the pulse sequence corresponding to each image in the common database one by one. The pulse sequence acquisition is performed using a synchronous recording system, which is a system that records image signals using pulse coding, consisting of a display device and a recording device, for example, an LCD screen and a DVS camera. The display device is initialized, and after ensuring normal operation, the image is displayed on the LCD screen and smoothly moved along a specified preset path at a certain speed. During the movement of the image along the preset path, the recording device records the generated pulse sequences based on the pixel changes between two adjacent images during the image movement, and after the image movement is complete, it pauses for a certain period of time to associate and store the pulse sequences generated by the scene with the images. By moving along a preset path, the pulse sequence corresponding to the image can be accurately obtained, thereby improving the effectiveness of acquiring training data used in the training process.

[0032] Furthermore, as the image moves along the preset path, it shifts simultaneously in both the horizontal and vertical directions. For example, the recording device is a DVS camera with the product model Inivation-DAVIS346, whose temporal and spatial resolutions are 1μs and 346×240, respectively. The preset path is as follows: Figure 2As shown, the motion path is A→B→C→D→A with the center of the LCD screen as the center. During the motion, the angle between each segment of the path and the horizontal axis and the vertical axis is 45 degrees. The image position is represented by the center coordinate divided by the image width. Based on the time resolution of the recorded radio frequency, the movement time on each path is 250ms. Therefore, during the movement from A to B, the path starts at (0, 0) and ends at (0.05, 0.05), with a horizontal speed of 0.2 and a vertical speed of 0.2. During the movement from B to C, the path starts at (0.05, 0.05) and ends at (0, 0.10), with a horizontal speed of -0.2 and a vertical speed of 0.2. During the movement from C to D, the path starts at (0, 0.10) and ends at (-0.05, 0.05), with a horizontal speed of -0.2 and a vertical speed of -0.2. During the movement from D to A, the path starts at (-0.05, 0.05) and ends at (0, 0), with a horizontal speed of 0.2 and a vertical speed of -0.2. The DVS camera acquires data about the external scene during image movement, generates a 1000ms pulse sequence corresponding to each image, and stores it in a response database along with the corresponding images. This process continues until pulse sequences are associated with all images, completing the creation of the response database. By setting a preset path that causes the image to shift simultaneously in both the horizontal and vertical directions, it ensures that pixel intensity changes on both axes are recorded synchronously.

[0033] It is worth mentioning that the specific model of the recording device used in the synchronous recording system can be adjusted according to actual needs, and the motion speed of the image can also be adaptively adjusted according to the parameters of the recording device or actual needs. This embodiment does not limit the specific model of the recording device, the image motion path, and the motion speed used in the synchronous recording system.

[0034] Furthermore, when creating the response database, before associating and storing the pulse sequences and their corresponding images, the process includes: filtering each pulse sequence according to the preset refresh rate of the display; and associating and storing the pulse sequences and their corresponding images, including associating and storing the filtered pulse sequences and their corresponding images. Specifically, the parameters of the recording device in a synchronous recording system generally do not match the parameters of the display device. Therefore, the image motion speed set according to the recording device's parameters is inconsistent with the screen refresh rate of the display device, inevitably resulting in noise generated by the display device's screen refresh in the pulse sequences acquired by the recording device. Therefore, after acquiring the pulse sequences generated during image motion through the synchronous recording system, it is necessary to perform noise removal on each acquired pulse sequence according to the acquired display screen refresh rate. For example, if the refresh rate of a commonly used display screen is 60Hz, a filter is added to remove the 60Hz signal from the acquired pulse sequences. Then, the noise-removed pulse sequences are associated and stored with their corresponding images, ultimately completing the creation of the response database. By filtering according to the display refresh rate, noise interference in the pulse sequences is effectively removed, improving the accuracy of the obtained pulse sequences.

[0035] It should be noted that noise removal of the pulse sequence is only one way of signal preprocessing. In practical applications, other preprocessing can be performed on the obtained pulse sequence according to specific needs to improve the efficiency and effect of neural network training. This embodiment does not limit the specific signal preprocessing method used.

[0036] It is worth mentioning that the noise cancellation process is mainly based on the screen refresh rate of the monitor. Therefore, in practical applications, the filtered signal can be adjusted according to the screen refresh rate of the selected monitor. This embodiment does not impose any restrictions on this.

[0037] In another example, the visual prosthesis trains the initial spiking neural network (SNN) based on data from the response database of the primary visual cortex. This includes: for each pulse sequence, segmenting the pulse sequence according to a preset time window to generate several pulse sub-sequences; and inputting each pulse sub-sequence into the SNN one by one in chronological order to train the SNN. Specifically, during the training and learning process of the SNN based on the input pulse sequence, the pulse sequence is first segmented according to the preset time window, dividing one pulse sequence into multiple pulse sub-sequences. The number of action potential responses corresponding to the pulse sequence is used as a supervision signal. The pulse sub-sequences are then input into the SNN one by one in chronological order to train the SNN. For example, when a 1000ms pulse sequence during image motion is acquired using an Inivation-DAVIS346 DVS camera, the preset time window can be set to 33ms, the pulse sequence can be divided into 30 pulse sub-sequences, and each can be input into the SNN one by one for training. By segmenting the pulse sequence, the impact of excessive single input information on training efficiency is avoided, thus improving training efficiency. By segmenting according to a specified preset time window, it is ensured that each input pulse subsequence contains sufficient information, thereby guaranteeing training effectiveness.

[0038] It is worth mentioning that the preset time window is determined based on factors such as the parameters of the recording device, the speed of image motion, and the total duration of the pulse sequence. In practical applications, it can be adjusted according to the specific application scenario, as long as the pulse subsequence within a single time window can contain enough information. This embodiment does not impose any restrictions on the specific setting of the preset time window.

[0039] Additionally, it should be noted that pre-trained spiking neural networks can also be obtained through model transformation. For example, a pre-convolutional neural network to be trained can be constructed. This network can contain three convolutional layers, one depthwise separable convolutional layer, and one nonlinear layer. The three convolutional layers replace photoreceptors, bipolar cells, and ganglion cells, respectively. The depthwise separable convolutional layer replaces the function of the primary visual cortex, and the nonlinear layer reduces the correlation between the predictive responses of different neurons, thereby improving prediction accuracy.

[0040] Predicting the action potential response of neurons in the visual cortex requires ensuring similarity to the label value in both temporal and spatial dimensions. Spatial similarity refers to the similarity between the responses of multiple neurons to the same image and the label value. Temporal similarity refers to the similarity between the responses of the same neuron to different images and the label value. Therefore, nonlinear layers are needed to achieve similarity in these two dimensions. First, using dot products instead of fully connected layers can effectively reduce the correlation between different neurons and improve spatial similarity. Second, adjusting different parameters based on the output value of the previous layer can improve temporal similarity. For example, the parameter index can be calculated using the output value of the previous layer according to the following formula:

[0041] Index=min

max(X / / 0.2,0),50

[0042] Then, using the calculated index, perform a dot product between the corresponding parameter and the input value X according to the following formula:

[0043] F(X)=Relu(X×params

index

[0044] In summary, convolutional neural networks achieve temporal and spatial similarity between predicted and labeled values ​​through dot product and multiple parameter adjustments.

[0045] After acquiring the public database, the images and their corresponding action potential responses from the database are used as input and supervision signals, respectively, to train the created convolutional neural network (CNN) until it can accurately predict action potential responses for different images. Then, the parameters from the CNN are transferred to a spiking neural network (SNN), while the functionality of each layer remains largely unchanged. Finally, the input to the SNN is changed to a pulse sequence corresponding to the external scene to avoid accuracy loss during the encoding of the external scene signal after parameter transfer.

[0046] Step 102: Input the pulse sequence into the pre-trained spiking neural network to obtain the response prediction result.

[0047] Specifically, after the control chip of the visual prosthesis acquires the pulse sequence corresponding to the external scene, it inputs the pulse sequence into a pre-trained spiking neural network, calls the spiking neural network to process the pulse sequence, and obtains the response prediction result. The response prediction result is the predicted action potential response of the neurons in the primary visual cortex to the external scene.

[0048] In one example, the spiking neural network obtains the response prediction result in the following way: based on the pulse sequence, it obtains the membrane potential change value of each neuron within a preset time; for each neuron, the sum of the membrane potential change value and the initial membrane potential is taken as the current membrane potential; if the current membrane potential is greater than a preset threshold, the number of firing peaks of the neuron is determined according to the ratio of the current membrane potential to the preset threshold; the number of firing peaks of each neuron is counted to obtain the response prediction result.

[0049] Specifically, when predicting action potential responses of neurons in the primary visual cortex, the spiking neural network uses depthwise separable convolutional layers to replace the processing function of the primary visual cortex. The pulse sequence is input into the spiking neural network, which processes the data to obtain the membrane potential changes of each neuron in the primary visual cortex within a preset time period. For a single neuron, the sum of the membrane potential change and the initial membrane potential is used as the current membrane potential. This current membrane potential is then compared to a preset threshold. If the current membrane potential is greater than the preset threshold, the ratio between the current membrane potential and the preset threshold is calculated and rounded. This rounding can be done by rounding down, rounding up, or using other rounding methods. The number of firing spikes of the neuron is then determined based on the rounded ratio. Finally, a pulse counting layer counts the number of firing spikes of each neuron to obtain the number of action potential responses generated by neurons in the primary visual cortex to the external scene, thus obtaining the response prediction result.

[0050] For example, for a single neuron, the change in the ratio of membrane potential to a preset threshold over multiple preset time periods is as follows: Figure 3 As shown, the membrane potential is 0 within the first preset duration, and the ratio is 0; within the second preset duration, the membrane potential accumulates to half of the preset threshold, and the ratio is 0.5; within the third preset duration, the membrane potential accumulates to four-fifths of the preset threshold; and within the fourth preset duration, the membrane potential accumulates to 2.2 times the preset threshold, and the ratio is 2.2. Therefore, if the current mode potential within the fourth preset duration is greater than the preset threshold, after rounding down the ratio, two discharge spikes are sequentially output using an accumulation and distribution network with multi-pulse emission rate within the fourth preset duration, and the membrane potential is reset.

[0051] In other words, in a spiking neural network, the accumulation distribution network with a single-pulse emission frequency is replaced with an accumulation distribution network with a multi-pulse emission rate. Then, the replacement accumulation distribution network is used to count whether neurons emit pulses and the number of pulses emitted. By using the accumulation distribution network with a multi-pulse emission rate, the number of firing peaks of neurons is determined based on the ratio of membrane potential to a preset threshold. This is more in line with the response characteristics of neurons under natural conditions, greatly improving the feature extraction ability and response prediction accuracy of spiking neural networks for neuronal responses in the primary visual cortex, and further enhancing the ability of spiking neural networks to replace retinal functions.

[0052] It is worth mentioning that when the other layers of the spiking neural network perform functional substitution for the cells or receptors at the corresponding levels, the accumulation and distribution network with multiple pulse emission frequencies can also be used to extract and output the action potential response of the cells or receptors, thereby improving the feature extraction capability of the entire spiking neural network.

[0053] Furthermore, the spiking neural network acquires the membrane potential change value of each neuron within a preset time period, including: for each neuron, acquiring the number of pulses from several preceding neurons connected to the neuron within the preset time period; and weighting and summing the number of pulses according to the preset weights corresponding to each preceding neuron to obtain the membrane potential change value. Specifically, based on the connection relationship between neurons, the preceding neurons connected to each neuron are determined. When acquiring the membrane potential change value of a neuron, the number of pulses output by each preceding neuron within the preset time period is counted, and then, combined with the preset weights corresponding to each preceding neuron, the number of pulses output by each preceding neuron is weighted and summed. The product of each pulse number and the preset weight is then summed to obtain the membrane potential change value of the neuron within the preset time period. By using a weighted summation method, the membrane potential change value of the neuron within the preset time period is accurately acquired.

[0054] It is worth mentioning that the preset weights of each preceding neuron can be determined based on factors such as the functional importance of the preceding neuron and the interval distance, and this embodiment does not impose any restrictions on this.

[0055] Step 103: Generate corresponding stimulus pulses based on the response prediction results and transmit the stimulus pulses to ganglion cells in the primary visual cortex or retina.

[0056] Specifically, after the visual prosthesis calls the spiking neural network to obtain the predicted action potential response of the primary visual cortex to the external scene, it uses the Poisson distribution to determine the timing and number of pulses occurring within a continuous time period based on the predicted response. Then, it generates the stimulus pulses corresponding to the predicted response and transmits the stimulus pulses to the ganglion cells of the primary visual cortex or retina, so that the visual cortex can receive the stimulus signals caused by the external scene and form an image of the external scene based on the received stimulus pulses.

[0057] Figure 4 This graph compares the action potential responses of normal primary visual cortex neurons to external scenes over a certain time period with the pulse sequences predicted by a spiking neural network (SNN). `Data` represents the action potential response of normal primary visual cortex neurons to external scenes, and `SNN` represents the action potential response predicted by the spiking neural network based on the external scene. In the graph showing the relationship between the number of pulses and time, the first black line segment in each statistical data set represents the number of pulses from normal primary visual cortex neurons, and the second gray line segment represents the number of pulses predicted by the spiking neural network. Small rectangles indicate a single stimulus pulse generated using a Poisson distribution based on the number of pulses from normal primary visual cortex neurons, while large rectangles indicate 10 repeated stimulus pulses. Based on the results of 10 repetitions, it can be observed that the spiking neural network can effectively predict the action potential vector corresponding to the external scene.

[0058] Furthermore, it should be understood that the step divisions of the various methods described above are only for clarity. In practice, they can be combined into one step or some steps can be split into multiple steps. As long as they include the same logical relationship, they are all within the scope of protection of this patent. Adding insignificant modifications or introducing insignificant designs to the algorithm or process, but without changing the core design of the algorithm and process, are also within the scope of protection of this patent.

[0059] Another aspect of the embodiments of this application relates to a pulse stimulation device, see reference Figure 5 ,include:

[0060] The acquisition module 501 is used to acquire the pulse sequence corresponding to the external scene.

[0061] The prediction module 502 is used to input the pulse sequence into a pre-trained spiking neural network to obtain the response prediction result; wherein, the response prediction result is the predicted action potential response of the neurons in the primary visual cortex to the external scene.

[0062] The stimulation module 503 is used to generate corresponding stimulation pulses based on the response prediction results and transmit the stimulation pulses to the ganglion cells of the primary visual cortex or retina.

[0063] It is not difficult to see that this embodiment is a device embodiment corresponding to the method embodiment, and this embodiment can be implemented in conjunction with the method embodiment. The relevant technical details mentioned in the method embodiment are still valid in this embodiment, and will not be repeated here to reduce repetition. Correspondingly, the relevant technical details mentioned in this embodiment can also be applied to the method embodiment.

[0064] It is worth mentioning that all modules involved in this embodiment are logical modules. In practical applications, a logical unit can be a physical unit, a part of a physical unit, or a combination of multiple physical units. Furthermore, to highlight the innovative aspects of this application, this embodiment does not introduce units that are not closely related to solving the technical problems proposed in this application; however, this does not mean that other units are absent in this embodiment.

[0065] Embodiments of this application also provide an electronic device, with reference to Figure 6 The method includes: at least one processor 601; and a memory 602 communicatively connected to at least one processor 601; wherein the memory 602 stores instructions executable by at least one processor 601, which are executed by at least one processor 601 to enable at least one processor 601 to perform the pulse stimulation method described in any of the above method embodiments.

[0066] The memory 602 and processor 601 are connected via a bus, which may include any number of interconnecting buses and bridges. The bus connects various circuits of one or more processors 601 and memory 602 together. The bus can also connect various other circuits, such as peripheral devices, voltage regulators, and power management circuits, which are well known in the art and therefore will not be described further herein. A bus interface provides an interface between the bus and the transceiver. The transceiver can be a single element or multiple elements, such as multiple receivers and transmitters, providing a unit for communicating with various other devices over a transmission medium. Data processed by processor 601 is transmitted over a wireless medium via an antenna, which further receives data and transmits it to processor 601.

[0067] Processor 601 is responsible for managing the bus and general processing, and can also provide various functions, including timing, peripheral interfaces, voltage regulation, power management, and other control functions. Memory 602 can be used to store data used by processor 601 during operation.

[0068] Another aspect of this application provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, it implements the above-described method embodiments.

[0069] That is, those skilled in the art will understand that all or part of the steps in the methods of the above embodiments can be implemented by a program instructing related hardware. This program is stored in a storage medium and includes several instructions to cause a device (which may be a microcontroller, chip, etc.) or processor to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, a portable hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0070] Those skilled in the art will understand that the above embodiments are specific embodiments for implementing this application, and in practical applications, various changes can be made to them in form and detail without departing from the spirit and scope of this application.

Claims

1. A pulse stimulation method, characterized in that, The method includes: a control chip for a visual prosthesis or an external terminal capable of communicating with the visual prosthesis control chip; and the method further includes: Obtain the pulse sequence corresponding to the external scene; The pulse sequence is input into a pre-trained spiking neural network to obtain a response prediction result; wherein, the response prediction result is the predicted action potential response of neurons in the primary visual cortex to the external scene; Generate corresponding stimulus pulses based on the response prediction results; The step of obtaining the pulse sequence corresponding to the external scene includes: Perform pixel detection on the external scene; A corresponding pulse event is generated for the pixel point whose pixel value changes in the external scene; wherein, the pulse event includes the pixel value change time, pixel point coordinates and pixel value change polarity; The pulse sequence is generated based on all the pulse events in the external scene; The spiking neural network obtains the response prediction result in the following manner: Based on the pulse sequence, obtain the membrane potential change value of each neuron within a preset time period; For each neuron, the sum of the membrane potential change and the initial membrane potential is taken as the current membrane potential; If the current membrane potential is greater than a preset threshold, the number of firing spikes of the neuron is determined based on the ratio of the current membrane potential to the preset threshold. The number of firing spikes in each neuron is counted to obtain the response prediction result; The step of generating the corresponding stimulus pulse based on the response prediction result includes: Using the Poisson distribution and the response prediction results, the timing and number of pulse occurrences within a continuous time period are determined, and the stimulus pulses corresponding to the response prediction results are generated.

2. The pulse stimulation method according to claim 1, characterized in that, Before acquiring the pulse sequence corresponding to the external scene, the method further includes: The initial spiking neural network is trained based on data from the response database of the primary visual cortex to obtain the pre-trained spiking neural network; wherein, the response database includes multiple images, the pulse sequence corresponding to each image, and the action potential response corresponding to each image.

3. The pulse stimulation method according to claim 2, characterized in that, Before training the initial spiking neural network based on data from the response database of the primary visual cortex, the method further includes: The pre-acquired images are smoothly moved along a preset path in a preset display, and the pulse sequence generated by each image during the movement is acquired. For each pulse sequence, the pulse sequence and the image corresponding to the pulse sequence are associated and stored.

4. The pulse stimulation method according to claim 3, characterized in that, During the movement of each image along the preset path, it shifts simultaneously in both the horizontal and vertical directions.

5. The pulse stimulation method according to claim 3, characterized in that, Before associating and storing the pulse sequence and the image corresponding to the pulse sequence, the method further includes: The pulse sequences are filtered according to the preset display refresh rate. The step of associating and storing the pulse sequence and the image corresponding to the pulse sequence includes: The filtered pulse sequences and the corresponding images are associated and stored.

6. The pulse stimulation method according to claim 2, characterized in that, The step of training the initial spiking neural network based on data from the response database of the primary visual cortex includes: For each pulse sequence, the pulse sequence is divided according to a preset time window to generate several pulse sub-sequences; In chronological order, each of the pulse subsequences is input into the initial spiking neural network to train the initial spiking neural network.

7. The pulse stimulation method according to claim 1, characterized in that, The step of obtaining the membrane potential change values ​​of each neuron within a preset time period includes: For each neuron, the number of pulses from several preceding neurons connected to the neuron within the preset time period is obtained; Based on the preset weights corresponding to each of the preceding neurons, the number of pulses is weighted and summed to obtain the membrane potential change value.

8. A pulse stimulation device, characterized in that, include: The acquisition module is used to acquire the pulse sequence corresponding to the external scene through the control chip of the visual prosthesis or an external terminal that can communicate with the control chip of the visual prosthesis; The step of obtaining the pulse sequence corresponding to the external scene includes: performing pixel detection on the external scene; generating corresponding pulse events for pixels whose pixel values ​​change in the external scene; wherein, the pulse event includes pixel value change time, pixel coordinates, and pixel value change polarity; and generating the pulse sequence based on all the pulse events of the external scene. A prediction module is used to input the pulse sequence into a pre-trained spiking neural network to obtain a response prediction result. The spiking neural network obtains the response prediction result in the following ways: Based on the pulse sequence, it obtains the membrane potential change value of each neuron within a preset time period; for each neuron, it uses the sum of the membrane potential change value and the initial membrane potential as the current membrane potential; if the current membrane potential is greater than a preset threshold, it determines the number of firing spikes of the neuron based on the ratio of the current membrane potential to the preset threshold; it counts the number of firing spikes of each neuron to obtain the response prediction result; wherein, the response prediction result is the predicted action potential response of the neurons in the primary visual cortex to the external scene. The stimulation module is used to generate corresponding stimulation pulses based on the response prediction result; the generation of corresponding stimulation pulses based on the response prediction result includes: using a Poisson distribution to determine the pulse occurrence time and number within a continuous time period based on the response prediction result, and generating stimulation pulses corresponding to the response prediction result.

9. An electronic device, characterized in that, include: At least one processor; as well as, A memory communicatively connected to the at least one processor; wherein, The memory stores instructions executable by the at least one processor, which, when executed by the at least one processor, enables the at least one processor to perform the pulse stimulation method as described in any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the pulse stimulation method according to any one of claims 1 to 7.

Images