Primordial Language-Based Brain-Inspired Visual Prosthesis System, Zoned Electrical Stimulation Method and Device

Abstract

The present invention provides a primitives-based brain-inspired visual prosthesis system, a method and device for regional electrical stimulation, which relate to the technical field of visual prostheses. The primitives-based brain-inspired visual sensor collects spatio-temporal differential images and color intensity images, and performs primitive extraction to obtain multiple primitive information in multi-bit form. The pulse encoder encodes them using different encoding methods to obtain multiple primitive information in single-bit form. The processor controls the implantable electrode arrays in different regions to generate electrical stimulation signals based on each primitive information in multi-bit form or single-bit form, and generates a regulation signal based on the feedback electroencephalogram signal, and adjusts the attribute parameters of the brain-inspired visual sensor based on the regulation signal. Based on the characteristics of low bandwidth, high robustness, and pulse coding of the brain-inspired visual sensor, the brain-inspired visual prosthesis system of the present invention can efficiently perceive the external environment and perform regional electrical stimulation based on primitive processing, improving the user's visual experience.

Description

Technical Field

[0001] The present invention relates to the technical field of visual prostheses, and in particular, to a brain-like visual prosthesis system based on primitives, a method and device for zonal electrical stimulation. Background Art

[0002] Vision is one of the most important senses for humans to perceive the world. However, retinal degenerative diseases can cause photoreceptors (cone cells and rod cells) to gradually apoptose, leading to irreversible vision loss or even total blindness in patients. A visual prosthesis is a device that treats severe vision loss by stimulating nerve cells at any position in the visual pathway (usually using electrical pulses).

[0003] In related technologies, as Figure 1 shown, a visual prosthesis system usually uses a patient-worn camera to collect videos of the external environment. The processor analyzes the videos to generate analog nerve signals, and applies electrical stimulation to the V1 area of the visual cortex or the retinal area through electrode arrays such as the Utah Electrode Array (UEA) or the Wireless Floating Microelectrode Array (WFMA). The human eye then stimulates the initial visual cortex V1 area through the lateral geniculate body, and the initial visual cortex V1 area further stimulates other areas of the visual cortex, causing the patient to experience phosphene phenomena. Additionally, the processor can also perform phosphene simulation based on the videos and display the results of the phosphene simulation through a display to know what the patient sees.

[0004] However, in the above-mentioned related technologies, collecting the external environment through a traditional camera has limited information collection, which has a large gap from the information seen by the human eye. When there are high-speed moving objects in the external environment, motion blur will occur, and underexposure or overexposure is likely to occur in high dynamic range scenes. Moreover, the repetition rate between adjacent frames in the collected videos is relatively high, resulting in low bandwidth utilization of data transmission, thus leading to relatively inefficient perception of the visual prosthesis system. Summary of the Invention

[0005] The present invention provides a brain-like visual prosthesis system based on primitives, a method and device for zonal electrical stimulation, so as to solve the defect that the perception of the visual prosthesis system in the prior art is relatively inefficient.

[0006] The present invention provides a brain-like visual prosthesis system based on primitives, including a brain-like visual sensor based on primitives, an implantable electrode array, and a processor. The brain-like visual sensor based on primitives and the implantable electrode array are both connected to the processor;

[0007] The brain-inspired vision sensor is used to collect the external environment to obtain spatio-temporal differential images and color intensity images, extract primitives from the spatio-temporal differential images and the color intensity images to obtain multiple primitive information in multi-bit form, and encode the multiple primitive information in multi-bit form through a pulse encoder using different encoding methods according to the type of each primitive information to obtain multiple primitive information in single-bit form, and send the multiple primitive information in multi-bit form or the multiple primitive information in single-bit form to the processor;

[0008] The processor is used to control implantable electrode arrays implanted in different regions of the human visual processing pathway to generate electrical stimulation signals based on each primitive information in multi-bit form or each primitive information in single-bit form, and the electrical stimulation signals are used to stimulate the corresponding regions through the implantable electrode arrays implanted in the corresponding regions;

[0009] The implantable electrode array is used to collect the electroencephalogram signals of the corresponding region and send the electroencephalogram signals to the processor;

[0010] The processor is further used to generate a regulation signal based on the electroencephalogram signals through an electroencephalogram decoding feedback regulator, and adjust the attribute parameters of the brain-inspired vision sensor based on the regulation signal, so as to re-collect the external environment based on the adjusted brain-inspired vision sensor and stimulate the corresponding region based on the re-collected images.

[0011] According to a primitive-based brain-inspired vision prosthesis system provided by the present invention, the multiple primitive information in multi-bit form includes contour information, change information, color information, and light intensity information;

[0012] The brain-inspired vision sensor is specifically used to extract primitives from the spatio-temporal differential images to obtain the contour information and change information of the external environment, extract primitives from the color intensity images to obtain the color information and light intensity information of the external environment, and use the pulse encoder to perform binary encoding on the contour information to obtain first primitive information in single-bit form, perform time encoding on the change information to obtain second primitive information in single-bit form, and perform rate encoding on the color information and the light intensity information to obtain third primitive information in single-bit form. According to a primitive-based brain-inspired vision prosthesis system provided by the present invention, the processor is specifically used to control a first implantable electrode array implanted in the retinal region to generate a first electrical stimulation signal encoded with the first primitive information and the second primitive information based on the first primitive information and the second primitive information, and the first electrical stimulation signal is used to stimulate the rod cells in the retinal region;

[0013] The processor is specifically configured to control the second implanted electrode array implanted in the retinal region to generate a second electrical stimulation signal encoding the third primitive information based on the third primitive information, and the second electrical stimulation signal is used to stimulate cone cells in the retinal region.

[0014] According to a primitive-based brain-like visual prosthesis system provided by the present invention, the processor is specifically configured to control the third implanted electrode array implanted in the primary visual cortex V1 region to generate a third electrical stimulation signal encoding the first primitive information, the second primitive information, and the third primitive information based on the first primitive information, the second primitive information, and the third primitive information, and the third electrical stimulation signal is used to stimulate the primary visual cortex V1 region.

[0015] According to a primitive-based brain-like visual prosthesis system provided by the present invention, the processor is specifically configured to perform feature extraction on multiple primitive information in a multi-bit form to obtain semantic information;

[0016] The processor is specifically configured to control the fourth implanted electrode array implanted in the primary visual cortex V1 region to generate a fourth electrical stimulation signal encoding the first primitive information and the third primitive information based on the first primitive information and the third primitive information, and the fourth electrical stimulation signal is used to stimulate the primary visual cortex V1 region;

[0017] The processor is specifically configured to control the fifth implanted electrode array implanted in the visual cortex V2 region to generate a fifth electrical stimulation signal encoding the second primitive information based on the second primitive information, and the fifth electrical stimulation signal is used to stimulate the visual cortex V2 region;

[0018] The processor is specifically configured to control the sixth implanted electrode array implanted in the visual cortex V4 region to generate a sixth electrical stimulation signal encoding the semantic information based on the semantic information, and the sixth electrical stimulation signal is used to stimulate the visual cortex V4 region.

[0019] According to a primitive-based brain-like visual prosthesis system provided by the present invention, the processor is specifically configured to determine motion attention information based on the second primitive information through an electroencephalogram signal predictor, generate contrast enhancement information based on the third primitive information, and predict simulated neural signals based on the motion attention information, the contrast enhancement information, and the first primitive information;

[0020] The processor is specifically configured to determine stimulation parameters executable by the implanted electrode array based on the simulated neural signals, and control the implanted electrode array to generate the electrical stimulation signal based on the stimulation parameters.

[0021] A brain-inspired visual prosthesis system based on primitives according to the present invention, wherein the brain-inspired visual sensor is configured to collect the depth of the external environment to obtain depth information and send the depth information to the processor;

[0022] The processor is further configured to generate a reconstructed image based on the first primitive information, the second primitive information, and the third primitive information through an image reconstructor, segment the reconstructed image through an image segmenter to obtain a target segmentation mask, and generate a depth image based on the depth information and the reconstructed image through a depth estimator;

[0023] The processor is further configured to predict the analog neural signals based on the first primitive information, the second primitive information, the third primitive information, the depth image, the reconstructed image, and the target segmentation mask through the electroencephalogram signal predictor.

[0024] A brain-inspired visual prosthesis system based on primitives according to the present invention, wherein the processor is specifically configured to extract features from the electroencephalogram signals through an electroencephalogram decoding feedback regulator to obtain intensity matching features and frequency synchronization features;

[0025] The processor is specifically configured to generate the regulation signal when at least one of the following preset conditions is not satisfied, and the preset conditions include: the intensity matching feature indicates that the pulse intensity of the electrical stimulation signal reaches the neural activation threshold, and the frequency synchronization feature indicates that the pulse frequency of the electrical stimulation signal resonates with the neural oscillation.

[0026] A brain-inspired visual prosthesis system based on primitives according to the present invention, wherein the processor is specifically configured to determine the identifier of the implantable electrode array and the target primitive combination information type corresponding to the target region based on the correspondence between the region, the electrode array identifier, and the primitive combination information type, and determine the information corresponding to the target primitive combination information type from the first primitive information, the second primitive information, and the third primitive information to obtain target primitive combination information;

[0027] The processor is specifically configured to control the implantable electrode array to generate the electrical stimulation signal based on the target primitive combination information.

[0028] The present invention also provides a partitioned electrical stimulation method based on primitives, which is applied to any of the above brain-inspired visual prosthesis systems based on primitives, and the method includes:

[0029] Receive multiple primitive information in multi-bit form or multiple primitive information in single-bit form sent by a primitive-based brain-inspired vision sensor. The multiple primitive information in multi-bit form is obtained by the brain-inspired vision sensor through primitive extraction of the acquired spatio-temporal differential image and color intensity image. The multiple primitive information in single-bit form is obtained by the brain-inspired vision sensor through a pulse encoder using different coding methods for different types of primitive information;

[0030] Based on each piece of primitive information in multi-bit form or each piece of primitive information in single-bit form, control an implantable electrode array implanted in different regions of the human visual processing pathway to generate an electrical stimulation signal, which is used to stimulate the corresponding region through the implantable electrode array implanted in the corresponding region;

[0031] Receive the electroencephalogram signals of the corresponding region collected by the implantable electrode array;

[0032] Through an electroencephalogram decoding feedback regulator, generate a regulation signal based on the electroencephalogram signals, and adjust the attribute parameters of the brain-inspired vision sensor based on the regulation signal, so as to re-collect the external environment based on the adjusted brain-inspired vision sensor, and stimulate the corresponding region based on the re-collected image.

[0033] The present invention also provides a primitive-based zonal electrical stimulation device, including:

[0034] A first receiving unit, configured to receive multiple primitive information in multi-bit form or multiple primitive information in single-bit form sent by a primitive-based brain-inspired vision sensor. The multiple primitive information in multi-bit form is obtained by the brain-inspired vision sensor through primitive extraction of the acquired spatio-temporal differential image and color intensity image. The multiple primitive information in single-bit form is obtained by the brain-inspired vision sensor through a pulse encoder using different coding methods for different types of primitive information;

[0035] A control unit, configured to control an implantable electrode array implanted in different regions of the human visual processing pathway to generate an electrical stimulation signal based on each piece of primitive information in multi-bit form or each piece of primitive information in single-bit form, and the electrical stimulation signal is used to stimulate the corresponding region through the implantable electrode array implanted in the corresponding region;

[0036] A second receiving unit, configured to receive the electroencephalogram signals of the corresponding region collected by the implantable electrode array;

[0037] An adjustment unit is configured to generate a regulation signal based on the electroencephalogram signal through an electroencephalogram decoding feedback regulator, and adjust the attribute parameters of the brain-like vision sensor based on the regulation signal, so as to re-collect the external environment based on the adjusted brain-like vision sensor, and stimulate the corresponding area based on the re-collected image.

[0038] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the computer program, the method for partitioned electrical stimulation based on primitives as described in any one of the above is implemented.

[0039] The present invention also provides a non-transitory computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the method for partitioned electrical stimulation based on primitives as described in any one of the above is implemented.

[0040] The present invention also provides a computer program product, including a computer program. When the computer program is executed by a processor, the method for partitioned electrical stimulation based on primitives as described in any one of the above is implemented.

[0041] The present invention provides a primitive-based brain-inspired visual prosthesis system, a zonal electrical stimulation method and device. The system includes a primitive-based brain-inspired visual sensor, an implantable electrode array, and a processor. The brain-inspired visual sensor is used to collect the external environment to obtain spatio-temporal differential images and color intensity images, extract primitives from the spatio-temporal differential images and color intensity images to obtain a plurality of primitive information in multi-bit form, and use a pulse encoder to encode each primitive information type using different coding methods to obtain a plurality of primitive information in single-bit form. The processor is used to control the implantable electrode array implanted in different regions of the human visual processing pathway to generate electrical stimulation signals based on each primitive information in multi-bit form or each primitive information in single-bit form. The electrical stimulation signals are used to stimulate the corresponding regions through the implantable electrode array implanted in the corresponding regions. The processor is also used to generate a regulation signal through a brain electrical decoding feedback regulator based on the brain electrical signals of the corresponding regions collected by the implantable electrode array, and adjust the attribute parameters of the brain-inspired visual sensor based on the regulation signal, so as to re-collect the external environment based on the adjusted brain-inspired visual sensor, and stimulate the corresponding regions based on the re-collected images. It can be seen that the present invention realizes the extraction and combination of primitives of the external environment by using a primitive-based brain-inspired visual sensor to control the implantable electrode array implanted in different regions of the human visual processing pathway to generate electrical stimulation signals for stimulating the corresponding regions. Since the primitive-based brain-inspired visual sensor has the functions of low power consumption, low bandwidth, high robustness, high frame rate, high dynamic range, and pulse coding output, the brain-inspired visual prosthesis system can efficiently perceive the external environment and perform zonal electrical stimulation based on primitive processing, thereby improving the perception and computational efficiency of the brain-inspired visual prosthesis system, especially suitable for coping with the challenge of difficult deployment of a large number of communication and computing resources on the human body and enhancing the user's visual experience. In addition, the brain-inspired visual sensor provided by the present invention can output a plurality of primitive information in multi-bit form and single-bit form. The plurality of primitive information in single-bit form is more in line with the characteristics of human nerve signals, so it can greatly reduce the computational complexity of the subsequent conversion process from image information to electrical stimulation signals in the visual prosthesis system. Moreover, the present invention proposes a two-way control framework. From the brain-inspired visual sensor to the human brain, based on the brain-inspired visual sensor perceiving the external environment, the predicted analog nerve signals are used to realize electrical stimulation of multiple positions in the visual pathway through the implantable electrode array. From the human brain to the brain-inspired visual sensor, based on the implantable electrode array perceiving the brain electrical signals, the brain electrical signals are decoded to generate a regulation signal, and the attribute parameters of the brain-inspired visual sensor are adjusted based on the regulation signal, so that the brain-inspired visual sensor re-collects the external environment based on the adjusted attribute parameters, and then generates new electrical stimulation signals based on the re-collected spatio-temporal differential images and color intensity images, making the new electrical stimulation signals more in line with the user and achieving the mutual coordination and adaptation between humans and machines. Description of the Drawings

[0042] To more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for the description of the embodiments or the prior art. Obviously, the drawings in the following description are some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.

[0043] Figure 1 It is a schematic structural diagram of a visual prosthesis system in the prior art.

[0044] Figure 2 It is a schematic diagram of the imaging effect of an active pixel array in the prior art.

[0045] Figure 3 It is a schematic diagram of the imaging effect of a differential pixel array in the prior art.

[0046] Figure 4 It is one of the schematic structural diagrams of a primitives-based brain-inspired visual prosthesis system provided by an embodiment of the present invention.

[0047] Figure 5 It is another schematic structural diagram of a primitives-based brain-inspired visual prosthesis system provided by an embodiment of the present invention.

[0048] Figure 6 It is yet another schematic structural diagram of a primitives-based brain-inspired visual prosthesis system provided by an embodiment of the present invention.

[0049] Figure 7 It is still another schematic structural diagram of a primitives-based brain-inspired visual prosthesis system provided by an embodiment of the present invention.

[0050] Figure 8 It shows a comparison between the original image and the electrical stimulation signal schematic diagram in the prior art.

[0051] Figure 9 It is another schematic structural diagram of a primitives-based brain-inspired visual prosthesis system provided by an embodiment of the present invention.

[0052] Figure 10 It is yet another schematic structural diagram of a primitives-based brain-inspired visual prosthesis system provided by an embodiment of the present invention.

[0053] Figure 11 It is a schematic flowchart of a partitioned electrical stimulation method based on primitives provided by an embodiment of the present invention.

[0054] Figure 12 It is a schematic diagram of the physical structure of an electronic device provided by an embodiment of the present invention. Detailed implementation manners

[0055] To make the objectives, technical solutions, and advantages of the present invention more clear, the following will, in conjunction with the accompanying drawings in the present invention, clearly and completely describe the technical solutions in the present invention. Apparently, the described embodiments are some, rather than all, of the embodiments of the present invention. All other embodiments obtained by those of ordinary skill in the art based on the embodiments in the present invention without making creative efforts fall within the scope of protection of the present invention.

[0056] Before introducing the primal-based brain-inspired visual prosthesis system of the present invention, the content related to the present invention will be introduced first:

[0057] In the human visual processing pathway, the splitting and combination of visual primitives play a crucial role. Initially, photoreceptor cells (cone cells and rod cells) sense the light entering the eye and convert the light signal into an electrical signal. These electrical signals are gradually transmitted and integrated through complex neuronal networks in the retina, such as bipolar cells and ganglion cells, to form visual primitives, including basic visual features such as edges, changes, colors, and intensities. When the signal is transmitted to the primary visual cortex (V1) area, the extraction of primitives further unfolds. There are a large number of functionally specialized neurons in the V1 area, and these specialized neurons have selective responses to different visual primitives. For example, some neurons only have a strong response to edges in a specific direction, thus disassembling a complex visual scene into discrete primitive representations. Subsequently, the signal encoded with primitive information is transmitted to higher-level visual cortex areas, such as the secondary visual cortex (V2), visual area 4 (V4), and inferotemporal cortex (IT) area, etc. In these areas, primitive combination and depth processing are realized. Neurons in different areas cooperate to combine primitives. Neurons in the V2 area are sensitive to simple shapes formed by the combination of multiple edges, and neurons in the IT area can specifically encode complex object shapes and categories. At the same time, the brain also calibrates the combination and processing results of primitives with the help of context information, past memories, etc., and finally constructs a complete perception and understanding of the three-dimensional depth, semantic understanding, motion tracking, etc. of the complex visual world in the external environment.

[0058] A brain-computer interface (BCI) is a direct communication pathway between the electrical activity of the brain and an external device, most commonly a computer or a robotic limb. BCI is typically used in areas such as mapping, enhancing or repairing human cognition, and enhancing or repairing human sensorimotor functions. Based on the distance between the electrodes and the brain tissue, the implementation scope of BCI can be divided into non-invasive, such as electroencephalography (EEG), magnetoencephalography (MEG), electrooculography (EOG), magnetic resonance imaging (MRI), partially invasive, such as electrocorticography (ECoG) and intravascular, and invasive (microelectrode arrays). BCI enables the brain to communicate with the outside world without relying on peripheral neuromuscular tissues. For example, an intracortical visual prosthesis (ICVP) is implanted in the visual cortex of blind patients. ICVP is a type of implanted visual prosthesis (IVP). By using ICVP to bypass the damaged area and directly stimulating the V1 area through an electrode array, patients can have visual perception.

[0059] Modern mainstream image sensors mainly adopt active pixel sensor (APS) technology, and the typical representative is the complementary metal-oxide-semiconductor (CMOS) image sensor. The core working principle of such sensors is that each pixel unit integrates a photodiode and an active amplifier, and realizes the conversion of photoelectric signals through the global shutter or rolling shutter mechanism of line-by-line scanning. However, the APS architecture has several limitations: the dynamic range is usually low, and there is motion blur when shooting moving objects, etc.

[0060] With the development of vision sensors, traditional sensors based on active pixel arrays face problems such as low frame rate and low dynamic range. To address these issues, event-based vision sensors (EVS), inspired by bipolar cells in the human visual pathway, have been proposed, including dynamic vision sensors (DVS), dynamic and active-pixel vision sensors (DAVIS), etc. These sensors adopt a special difference pixel (DP) design, where each pixel has an independent and asynchronous low-latency response to changes in light intensity, featuring high temporal resolution, high dynamic range, and low information redundancy. Among them, the DAVIS series of event cameras use a hybrid pixel array of difference pixels (DP) and active pixels (AP), enabling spatially aligned multimodal image perception. However, some event cameras, such as the DVXplorer event camera (DVXplorer Event Camera, DVXplorer) and Prophesee EVK4, only use difference pixels (DP) and cannot obtain RGB images. Nevertheless, the event stream accuracy of these sensors is relatively low, mostly ±1 bit (bit), and the event stream is asynchronously triggered, making it difficult for their signal characteristics to be directly compatible with mainstream frame-based processing algorithms. In addition, the triggering of the event stream depends on external motion, so it lacks stability. Figure 2 is a schematic diagram of the imaging effect of the active pixel array in the prior art, Figure 3 is a schematic diagram of the imaging effect of the difference pixel array in the prior art.

[0061] Inspired by the primitive splitting and combination mechanism in the human visual processing pathway, brain-inspired vision sensors based on primitives have been proposed. When perceiving the external environment, these sensors split the visual information of the external environment into visual primitives such as color, intensity, edge, change, depth, etc., and then combine and further process the primitives, enabling superior performance of high precision, high resolution, high frame rate, high dynamic range, and low bandwidth and low power consumption. For example, the brain-inspired vision sensor based on primitives can be Tianmou Core. On the one hand, it realizes the temporal and spatial differences of light intensity based on difference pixels (DP) to sparsely perceive the edges and changes in the external environment. On the other hand, it can perceive the color and light intensity of the external environment based on active pixels (AP), with the advantages of high dynamic range (>130 dB), high speed (up to 10,000 fps), and low bandwidth. Through further processing, the multi-channel primitive information is complementarily combined.

[0062] Based on this, the present invention introduces a primitive-based brain-inspired vision sensor into a brain-inspired vision prosthesis system. By utilizing the primitive-based brain-inspired vision sensor, primitive extraction and combination of the external environment are realized to control an implantable electrode array implanted in different regions of the human visual processing pathway to generate electrical stimulation signals for stimulating corresponding regions. Since the primitive-based brain-inspired vision sensor has the performance of low bandwidth, high frame rate, and high dynamic range, the brain-inspired vision prosthesis system can efficiently perceive the external environment and perform zonal electrical stimulation based on primitive processing, thereby improving the perception efficiency of the brain-inspired vision prosthesis system and enhancing the user's visual experience. In addition, the brain-inspired vision sensor provided by the present invention can output multiple primitive information in multi-bit form and single-bit form. The multiple primitive information in single-bit form is more in line with the characteristics of human neural signals, so the computational complexity of the subsequent conversion process from image information to electrical stimulation signals can be greatly reduced in the vision prosthesis system. Moreover, the present invention proposes a two-way control framework. From the brain-inspired vision sensor to the human brain, based on the brain-inspired vision sensor perceiving the external environment, the predicted analog neural signals are used to realize electrical stimulation of multiple positions in the visual pathway through the implantable electrode array. From the human brain to the brain-inspired vision sensor, based on the implantable electrode array perceiving the electroencephalogram signals, the electroencephalogram signals are decoded to generate a regulation signal, and based on the regulation signal, the attribute parameters of the brain-inspired vision sensor are adjusted, so that the brain-inspired vision sensor re-collects the external environment based on the adjusted attribute parameters, and then based on the spatio-temporal differential image and color intensity image obtained from the re-collection, a new electrical stimulation signal is generated, making the new electrical stimulation signal more in line with the user and achieving the mutual coordination and adaptation between humans and machines.

[0063] Figure 4 FIG. 4 is one of the schematic structural diagrams of the primitive-based brain-inspired vision prosthesis system provided by an embodiment of the present invention. As Figure 4 shown, the primitive-based brain-inspired vision prosthesis system includes a primitive-based brain-inspired vision sensor, an implantable electrode array, and a processor. The primitive-based brain-inspired vision sensor and the implantable electrode array are both connected to the processor;

[0064] The brain-inspired vision sensor is configured to collect the external environment to obtain a spatio-temporal differential image and a color intensity image, perform primitive extraction on the spatio-temporal differential image and the color intensity image to obtain multiple primitive information in multi-bit form, and perform encoding on each type of primitive information through a pulse encoder using different encoding methods to obtain multiple primitive information in single-bit form, and send the multiple primitive information in multi-bit form or the multiple primitive information in single-bit form to the processor;

[0065] The processor is configured to control implantable electrode arrays implanted in different regions of the human visual processing pathway to generate electrical stimulation signals based on the primitive information in multi-bit form or the primitive information in single-bit form, and the electrical stimulation signals are used to stimulate the corresponding regions through the implantable electrode arrays implanted in the corresponding regions;

[0066] The implantable electrode arrays are configured to collect electroencephalogram (EEG) signals of the corresponding regions and send the EEG signals to the processor;

[0067] The processor is further configured to generate a regulation signal based on the EEG signals through an EEG decoding feedback regulator, and adjust the attribute parameters of the brain-like vision sensor based on the regulation signal, so as to re-collect the external environment based on the adjusted brain-like vision sensor, and stimulate the corresponding regions based on the re-collected images.

[0068] Among them, the brain-like vision sensor based on primitives can be, for example, Tianmou Core, or a pulse encoder can be added on the basis of Tianmou Core. The pulse encoder encodes the types of the primitive information in multi-bit form by using different encoding methods to obtain multiple primitive information in single-bit form; the implantable electrode arrays can be one or multiple. When there are multiple implantable electrode arrays, the multiple implantable electrode arrays can be implanted in different regions. For example, including two implantable electrode arrays, one implantable electrode array can be implanted in the retina region and the other implantable electrode array can be implanted in the primary visual cortex V1 region. The present invention does not limit this.

[0069] Exemplarily, in practical applications, an electronic device configured with a brain-like vision prosthesis system based on primitives can be worn on a user. The external environment is collected through the brain-like vision sensor based on primitives to obtain spatio-temporal differential images and color intensity images. The spatio-temporal differential images and color intensity images are subjected to primitive extraction to obtain multiple primitive information in multi-bit form, and the pulse encoder encodes the types of the primitive information by using different encoding methods to obtain multiple primitive information in single-bit form. The multiple primitive information in multi-bit form or the multiple primitive information in single-bit form is sent to the processor. The processor can predict the analog neural signals in different regions of the human visual processing pathway based on the primitive information in multi-bit form and the combination of at least two types of information in the primitive information, and control the implantable electrode arrays implanted in the corresponding regions to generate electrical stimulation signals based on the analog neural signals, and stimulate the corresponding regions through the electrical stimulation signals, realizing zonal stimulation, and finally enabling the user to generate visual perception.

[0070] In addition, considering the individual differences of each user, the same set of attribute parameters of the brain-inspired vision sensor may not be suitable for all users. Therefore, a two-way human-machine interaction is considered. For each region, the electroencephalogram (EEG) signals of the high-level visual cortex are collected through the implanted electrode array in the corresponding region, and the collected EEG signals are sent to the processor. The processor decodes the EEG signals and analyzes the spatio-temporal patterns of neural activities (such as specific frequency band energy, neuron cluster firing rate, or decoded visual features, etc.) to generate a regulation signal. Here, the regulation signal includes a signal for modifying the attribute parameters of the brain-inspired vision sensor, and the modified attribute parameters can be the frame rate, exposure, gain, and pixel value threshold of the brain-inspired vision sensor, etc., so that the brain-inspired vision sensor re-collects the external environment based on the adjusted attribute parameters, and then generates a new electrical stimulation signal based on the spatio-temporal differential image and color intensity image obtained from the re-collection, and stimulates the corresponding region based on the new electrical stimulation signal, making the new electrical stimulation signal more suitable for the user, achieving the mutual coordination and adaptation between humans and machines, and providing high-quality visual stimulation that conforms to the individual characteristics of the user.

[0071] It should be noted that when the processor controls the implanted electrode array to generate corresponding electrical stimulation signals based on each primitive information in multi-bit form, it can support computer science-oriented applications. When the processor controls the implanted electrode array to generate corresponding electrical stimulation signals based on each primitive information in single-bit form, it can support neuroscience-oriented applications.

[0072] It should be noted that the selection of the implanted electrode array includes but is not limited to UEA and WFMA, and the deployment positions of the implanted electrode array include but are not limited to the retina region, the optic nerve region, and multiple different visual signal perception and processing regions in the initial visual cortex V1 region. The present invention does not make any limitations in this regard.

[0073] It should be noted that the acquisition of the external environment by the brain-inspired vision sensor based on primitives includes but is not limited to multiple visual primitives such as the color, contour, light intensity, change, and depth of the external environment. The present invention does not make any limitations in this regard.

[0074] It should be noted that the brain-inspired vision sensor based on primitives can be implemented through chip hybrid pixel array integration, or through a beam splitter to align multi-modal discrete vision sensors, or through multi-sensor calibration, or through a binocular vision system; the present invention utilizes the advantages of high frame rate, high dynamic range, low bandwidth, and low power consumption of the brain-inspired vision sensor based on primitives that exceed those of traditional cameras, as well as the similarity between the signal characteristics under the brain-inspired vision primitive processing mechanism and the characteristics of human eye perception and processing, which helps to achieve high-quality perception and long battery life operation deployed at the processor end.

[0075] It should be noted that the processor in the present invention can be set separately or integrated in a brain-inspired vision sensor based on primitives. When selecting the processor, a neuromorphic chip can be considered. Especially for dual-channel heterogeneous data input, a neuromorphic chip that supports both artificial neural networks and spiking neural networks, such as Tianjic, can achieve low-power and efficient multi-modal data processing at the computational level. The present invention does not limit this.

[0076] The brain-inspired visual prosthesis system based on primitives provided by the present invention includes a brain-inspired visual sensor based on primitives, an implantable electrode array, and a processor. The brain-inspired visual sensor is used to collect the external environment to obtain spatio-temporal difference images and color intensity images, extract primitives from the spatio-temporal difference images and color intensity images to obtain multiple primitive information in multi-bit form, and use a pulse encoder to encode each primitive information type using different encoding methods to obtain multiple primitive information in single-bit form. The processor is used to control the implantable electrode array implanted in different regions of the human visual processing pathway to generate electrical stimulation signals based on each primitive information in multi-bit form or each primitive information in single-bit form. The electrical stimulation signals are used to stimulate the corresponding regions through the implantable electrode array implanted in the corresponding regions. The processor is also used to generate a regulation signal through a brain electrical decoding feedback regulator based on the brain electrical signals of the corresponding regions collected by the implantable electrode array, and adjust the attribute parameters of the brain-inspired visual sensor based on the regulation signal, so as to re-collect the external environment based on the adjusted brain-inspired visual sensor, and stimulate the corresponding regions based on the re-collected images. It can be seen that the present invention realizes the extraction and combination of primitives of the external environment by using a brain-inspired visual sensor based on primitives, so as to control the implantable electrode array implanted in different regions of the human visual processing pathway to generate electrical stimulation signals for stimulating the corresponding regions. Since the brain-inspired visual sensor based on primitives has the functions of low power consumption, low bandwidth, high robustness, high frame rate, high dynamic range, and pulse coding output, the brain-inspired visual prosthesis system can efficiently perceive the external environment and perform zonal electrical stimulation based on primitive processing, thereby improving the perception and computational efficiency of the brain-inspired visual prosthesis system, especially suitable for coping with the challenge of difficult deployment of a large number of communication and computing resources on the human body and enhancing the user's visual experience. In addition, the brain-inspired visual sensor provided by the present invention can output multiple primitive information in multi-bit form and single-bit form. The multiple primitive information in single-bit form is more in line with the characteristics of human nerve signals, so it can greatly reduce the computational complexity of the subsequent conversion process from image information to electrical stimulation signals in the visual prosthesis system. Moreover, the present invention proposes a two-way control framework. From the brain-inspired visual sensor to the human brain, based on the brain-inspired visual sensor perceiving the external environment, the predicted analog nerve signals are used to realize electrical stimulation of multiple positions in the visual pathway through the implantable electrode array. From the human brain to the brain-inspired visual sensor, based on the implantable electrode array perceiving the brain electrical signals, the brain electrical signals are decoded to generate a regulation signal, and the attribute parameters of the brain-inspired visual sensor are adjusted based on the regulation signal, so that the brain-inspired visual sensor re-collects the external environment based on the adjusted attribute parameters, and then generates new electrical stimulation signals based on the re-collected spatio-temporal difference images and color intensity images, making the new electrical stimulation signals more in line with the user and achieving the mutual coordination and adaptation between humans and machines.

[0077] In one embodiment, the multiple primitive information in multi-bit form includes contour information, change information, color information, and light intensity information.

[0078] The brain-like vision sensor is specifically configured to perform primitive extraction on the spatio-temporal differential image to obtain the contour information and change information of the external environment, perform primitive extraction on the color intensity image to obtain the color information and light intensity information of the external environment, and use the pulse encoder to perform binary encoding on the contour information to obtain the first primitive information in single-bit form, perform time encoding on the change information to obtain the second primitive information in single-bit form, and perform rate encoding on the color information and the light intensity information to obtain the third primitive information in single-bit form.

[0079] Among them, the contour information is used to characterize the light intensity difference at different spatial positions at the same moment, and the change information is used to characterize the difference in light intensity at the same spatial position over time.

[0080] Exemplarily, the brain-like vision sensor can perform primitive extraction on the spatio-temporal differential image based on algorithms such as artificial neural network algorithms, spiking neural network algorithms, or hybrid neural network algorithms to obtain the contour information and change information of the external environment, and perform primitive extraction on the color intensity image to obtain the color information and light intensity information of the external environment. Furthermore, the pulse encoder is used to perform binary encoding on the contour information to obtain the first primitive information in single-bit form, perform time encoding on the change information to obtain the second primitive information in single-bit form, and perform rate encoding on the color information and the light intensity information to obtain the third primitive information in single-bit form.

[0081] In this embodiment, the pulse encoder included in the brain-like vision sensor is used to perform binary encoding on the contour information to obtain the first primitive information in single-bit form, perform time encoding on the change information to obtain the second primitive information in single-bit form, and perform rate encoding on the color information and the light intensity information to obtain the third primitive information in single-bit form, realizing the generation of different types of primitive information in single-bit form. The multiple primitive information in single-bit form is more in line with the characteristics of human nerve signals, so it can greatly reduce the computational complexity of the subsequent conversion process from image information to electrical stimulation signals in the visual prosthesis system.

[0082] In one embodiment, the processor is specifically configured to control the first implantable electrode array implanted in the retinal region to generate a first electrical stimulation signal encoded with the first primitive information and the second primitive information based on the first primitive information and the second primitive information, and the first electrical stimulation signal is used to stimulate the rod cells in the retinal region.

[0083] The processor is specifically configured to control the second implanted electrode array implanted in the retinal region to generate a second electrical stimulation signal encoded with the third primitive information based on the third primitive information, and the second electrical stimulation signal is used to stimulate cone cells in the retinal region.

[0084] Exemplarily, Figure 5 FIG. 2 is a second schematic structural diagram of a brain-inspired visual prosthesis system based on primitives according to an embodiment of the present invention. As Figure 5 shown, Figure 5 FIG. 2 shows a system solution for using a brain-inspired visual sensor and implanting an implanted electrode array in a retinal region to stimulate cone cells and rod cells. At the brain-inspired visual sensor end, the brain-inspired visual sensor provides dual-channel data collected from the external environment, including high-frame-rate, sparse spatio-temporal difference images and low-frame-rate, dense color intensity images. The spatio-temporal difference images focus on perceiving changes and contours in the external environment, and the color intensity images focus on finely perceiving the color and light intensity of the external environment. This dual-channel data is consistent with the functional characteristics of rod cells and cone cells in the retina. Therefore, in the present invention, spatio-temporal difference images and color intensity images for the external environment are obtained through the brain-inspired visual sensor, a first analog neural signal is predicted based on first primitive information and second primitive information determined from the spatio-temporal difference images, and a first stimulation parameter executable by the first implanted electrode array implanted in the retinal region is determined based on the first analog neural signal. Then, the first stimulation parameter is sent to the first implanted electrode array, so that the first implanted electrode array generates a first electrical stimulation signal encoded with the first primitive information and the second primitive information based on the first stimulation parameter, and the rod cells in the retinal region are stimulated through the first electrical stimulation signal. Figure 5 FIG. 2 shows that the human eye includes a retina, a fovea, a blind spot, etc. The cells shown in the green box are indicated by green arrows, that is, they include cone cells and rod cells.

[0085] A second analog neural signal is predicted based on third primitive information determined from the color intensity images, and a second stimulation parameter executable by the second implanted electrode array implanted in the retinal region is determined based on the second analog neural signal. Then, the second stimulation parameter is sent to the second implanted electrode array, so that the second implanted electrode array generates a second electrical stimulation signal encoded with the third primitive information based on the second stimulation parameter, and the cone cells in the retinal region are stimulated through the second electrical stimulation signal, thereby achieving targeted and efficient electrical stimulation.

[0086] It should be noted that the first implantable electrode array and the second implantable electrode array can be the same or different. When they are the same, some electrode points in the first implantable electrode array can be used to stimulate rod cells, and another part of the electrode points in the first implantable electrode array can be used to stimulate cone cells. When the first implantable electrode array and the second implantable electrode array are different, rod cells and cone cells can be stimulated separately through the first implantable electrode array and the second implantable electrode array in parallel.

[0087] In this embodiment, based on the first primitive information and the second primitive information, it is possible to control the first implantable electrode array implanted in the retinal region to generate a first electrical stimulation signal encoded with the first primitive information and the second primitive information to stimulate the rod cells in the retinal region. It is also possible to control the second implantable electrode array implanted in the retinal region to generate a second electrical stimulation signal encoded with the third primitive information based on the third primitive information to stimulate the cone cells in the retinal region, thereby achieving targeted electrical stimulation of the retinal region.

[0088] In one embodiment, the processor is specifically configured to control a third implantable electrode array implanted in the primary visual cortex V1 region to generate a third electrical stimulation signal encoded with the first primitive information, the second primitive information, and the third primitive information based on the first primitive information, the second primitive information, and the third primitive information, and the third electrical stimulation signal is used to stimulate the primary visual cortex V1 region.

[0089] Exemplarily, Figure 6 is the third schematic diagram of the structure of the brain-like visual prosthesis system based on primitives provided by the embodiments of the present invention. As Figure 6 shown, Figure 6 It is a system solution for stimulating the primary visual cortex V1 region through an implantable electrode array by using a brain-like visual sensor based on primitives. This solution controls a third implantable electrode array implanted in the primary visual cortex V1 region to generate a third electrical stimulation signal encoded with the first primitive information, the second primitive information, and the third primitive information based on the first primitive information, the second primitive information, and the third primitive information, providing basic visual primitives such as edges, changes, colors, and light intensities for the superficial visual processing brain region, and providing an information source for further processing and processing primitive information in subsequent brain regions. Compared with electrical stimulation at the retinal level, it can better address the problem of blindness caused by optic nerve damage.

[0090] It should be noted that the present invention can generate a first electrical stimulation signal, a second electrical stimulation signal, and a third electrical stimulation signal simultaneously. The rod cells in the retinal region are stimulated by the first electrical stimulation signal, the cone cells in the retinal region are stimulated by the second electrical stimulation signal, and the third electrical stimulation signal is used to stimulate the primary visual cortex V1 region. The present invention does not limit this.

[0091] In one embodiment, the processor is specifically configured to extract features from multiple primitive information in multi-bit form to obtain semantic information.

[0092] The processor is specifically configured to control a fourth implantable electrode array implanted in the primary visual cortex V1 region to generate a fourth electrical stimulation signal encoded with the first primitive information and the third primitive information based on the first primitive information and the third primitive information, and the fourth electrical stimulation signal is used to stimulate the primary visual cortex V1 region.

[0093] The processor is specifically configured to control a fifth implantable electrode array implanted in the visual cortex V2 region to generate a fifth electrical stimulation signal encoded with the second primitive information based on the second primitive information, and the fifth electrical stimulation signal is used to stimulate the visual cortex V2 region.

[0094] The processor is specifically configured to control a sixth implantable electrode array implanted in the visual cortex V4 region to generate a sixth electrical stimulation signal encoded with the semantic information based on the semantic information, and the sixth electrical stimulation signal is used to stimulate the visual cortex V4 region.

[0095] Exemplarily, Figure 7 is the fourth structural schematic diagram of the primitive-based brain-like visual prosthesis system provided by the embodiment of the present invention, as Figure 7 shown. Figure 7A system solution for two-way brain-machine interaction that implants an implantable electrode array into different visual cortex regions by using a brain-inspired visual sensor based on the primitive. Due to the limited resolution of the implantable electrode array, simply downsampling is difficult to fully retain the features of the original spatio-temporal differential image and color intensity image. Therefore, the present invention adopts a strategy of compensating for resolution with information dimension to perform multi-level image feature extraction on the spatio-temporal differential image and color intensity image from low to high. Specifically: overall feature extraction is performed on multiple primitive information in multi-bit form obtained by primitive extraction of the spatio-temporal differential image and color intensity image to obtain semantic information, and the semantic information is used as high-level features, the first primitive information and the third primitive information representing contour information, color information, and light intensity information are used as low-level features, and the second primitive information representing change information is used as middle-level features; furthermore, based on the first primitive information and the third primitive information representing contour information, color information, and light intensity information, which are low-level features, the fourth implantable electrode array implanted in the primary visual cortex V1 region is controlled to generate a fourth electrical stimulation signal encoded with the first primitive information and the third primitive information to stimulate the primary visual cortex V1 region; based on the second primitive information representing change information, which is the middle-level feature, the fifth implantable electrode array implanted in the visual cortex V2 region is controlled to generate a fifth electrical stimulation signal encoded with the second primitive information to stimulate the visual cortex V2 region; at the same time, based on the semantic information, which is the high-level feature, the sixth implantable electrode array implanted in the visual cortex V4 region is controlled to generate a sixth electrical stimulation signal encoded with the semantic information to stimulate the visual cortex V4 region.

[0096] It should be noted that as Figure 7 shown, considering the individual differences of each user, the same set of attribute parameters of the brain-inspired visual sensor may not be suitable for all users. Therefore, considering two-way human-machine interaction, the electroencephalogram signals of the high-level visual cortex are collected through the implantable electrode array, the collected electroencephalogram signals are sent to the processor, the electroencephalogram signals are decoded by the processor, and the attribute parameters of the brain-inspired visual sensor are regulated based on the decoding results to achieve mutual adaptation and regulation under human-machine cooperation.

[0097] It should be noted that in the case of limited resolution of the implantable electrode array, inputting the processed high-level features (such as the semantics of the external environment, etc.) can more effectively trigger clear perception than directly inputting the downsampled low-level images. Figure 8 Shows a comparison of the schematic diagrams of the original image and the electrical stimulation signal in the prior art. As Figure 8 shown, the left figure is the original image collected by the camera, and the right figure is the schematic diagram of the electrical stimulation signal generated after downsampling the original image. It can be seen that due to the limited resolution of the implantable electrode array, the important information such as the edge contour is lost in the electrical stimulation signal obtained by directly downsampling the original image.

[0098] In this embodiment, the information dimension is expanded under the limitation of the low-resolution implantable electrode array, and a strategy of compensating the resolution with the information dimension is adopted. Multi-level image feature extraction from low to high is performed on the spatiotemporal difference image and the color intensity image, and different areas are stimulated in a targeted manner based on information at different levels, thereby achieving higher quality visual information stimulation that is more in line with low resolution, and improving the user's visual perception experience in actual scenes.

[0099] In one embodiment, the processor is specifically used to determine motion attention information based on the second primitive information through an EEG signal predictor, generate contrast enhancement information based on the third primitive information, and predict a simulated neural signal based on the motion attention information, the contrast enhancement information and the first primitive information.

[0100] The processor is specifically used to determine stimulation parameters executable by the implantable electrode array based on the simulated neural signal, and control the implantable electrode array to generate the electrical stimulation signal based on the stimulation parameters.

[0101] For example, Figure 9 FIG. 5 is a schematic diagram of the structure of a primitive-based brain-like visual prosthesis system provided in an embodiment of the present invention. Figure 9 As shown, Figure 9A strategic framework for using a brain-inspired visual sensor based on the primitive language to achieve the extraction of various basic visual features, thereby realizing the electrical stimulation control of the primary visual processing area. The primary visual information processing of the human brain mainly covers basic primitives such as color, contour, change, and light intensity. In this invention, the contour information and change information of the external environment are extracted from the spatio-temporal differential images collected by the brain-inspired visual sensor, and the color information and light intensity information of the external environment are extracted from the color intensity images collected by the brain-inspired visual sensor. The second primitive information representing the change information is input into the electroencephalogram signal predictor, and the change signals (such as the displacement of a moving object, etc.) in adjacent frames or spatial regions are detected through the electroencephalogram signal predictor, and finally the motion attention information is obtained to highlight the motion area in the external environment; the third primitive information representing the color information and light intensity information is input into the electroencephalogram signal predictor, and the electroencephalogram signal predictor distinguishes different color regions based on the color information, enhances the color contrast, and optimizes the overall contrast by adjusting the local brightness difference, and finally generates the contrast enhancement information; further, a complete shape representation is synthesized based on the motion attention information, the contrast enhancement information, and the first primitive information representing the contour information. These features (the motion attention information, the contrast enhancement information, and the first primitive information representing the contour information) are transformed into the spatio-temporal pattern of neural signals, that is, the motion attention information is encoded as high-frequency pulses, such as in Visual Area 5 (V5), and the contrast enhancement information and the first primitive information representing the contour information are mapped to the tuning responses of cortical neurons (such as the orientation and shape selectivity in V1 / V2 regions), thereby simulating the hierarchical neural activities of the biological visual system, that is, obtaining the simulated neural signals, and then determining the stimulation parameters that can be executed by the implantable electrode array based on the simulated neural signals. Here, the stimulation parameters include pulse amplitude, pulse frequency, and electrode points to be activated, etc., and the stimulation parameters are sent to the implantable electrode array, so that the implantable electrode array generates corresponding electrical stimulation signals based on the received stimulation parameters.

[0102] In one embodiment, the processor is specifically configured to extract features from the electroencephalogram signal through an electroencephalogram decoding feedback regulator to obtain an intensity matching feature and a frequency synchronization feature.

[0103] The processor is specifically configured to generate the regulation signal when at least one of the following preset conditions is not satisfied. The preset conditions include: the intensity matching feature indicates that the pulse intensity of the electrical stimulation signal reaches the neural activation threshold, and the frequency synchronization feature indicates that the pulse frequency of the electrical stimulation signal resonates with the neural oscillation.

[0104] Exemplarily, the processor extracts features from the collected EEG signals through an EEG decoding feedback regulator, analyzes the intensity matching features and frequency synchronization features, determines whether the pulse intensity of the generated electrical stimulation signal reaches the nerve activation threshold based on the intensity matching features, and determines whether the pulse frequency of the generated electrical stimulation signal resonates with the nerve oscillation based on the frequency synchronization features. When the pulse intensity of the generated electrical stimulation signal does not reach the nerve activation threshold, and / or the pulse frequency of the generated electrical stimulation signal does not resonate with the nerve oscillation, it indicates that the generated electrical stimulation signal does not fully match the user. Then, a regulation signal is generated to adjust the attribute parameters of the brain-like vision sensor through the regulation signal, so as to re-collect the external environment based on the adjusted brain-like vision sensor. Furthermore, based on the spatio-temporal differential image and color intensity image obtained from the re-collection, a new electrical stimulation signal is generated, making the new electrical stimulation signal match the user and achieving the mutual coordination and adaptation between humans and machines. When the pulse intensity of the generated electrical stimulation signal reaches the nerve activation threshold and the pulse frequency of the generated electrical stimulation signal resonates with the nerve oscillation, it indicates that the generated electrical stimulation signal matches the user, and there is no need to generate a regulation signal.

[0105] In this embodiment, it is determined whether the generated electrical stimulation signal matches the user based on the intensity matching features and frequency synchronization features. When it is determined that the generated electrical stimulation signal does not fully match the user, the attribute parameters of the brain-like vision sensor are dynamically adjusted based on the regulation signal to ensure that the newly generated electrical stimulation signal matches the user, thereby realizing precise closed-loop regulation of nerve activities.

[0106] In one embodiment, the brain-like vision sensor is used to collect the depth of the external environment to obtain depth information and send the depth information to the processor.

[0107] The processor is further configured to generate a reconstructed image based on the first primitive information, the second primitive information, and the third primitive information through an image reconstructor, segment the reconstructed image through an image segmenter to obtain a target segmentation mask, and generate a depth image based on the depth information and the reconstructed image through a depth estimator.

[0108] The processor is further configured to predict the simulated nerve signal through the EEG signal predictor based on the first primitive information, the second primitive information, the third primitive information, the depth image, the reconstructed image, and the target segmentation mask.

[0109] Exemplarily, Figure 10 is the sixth structural schematic diagram of the primitive-based brain-like vision prosthesis system provided by the embodiments of the present invention, as Figure 10 shown, Figure 10To implement multi-level visual feature extraction from low to high using various visual primitives of a brain-inspired visual sensor based on the original language, thereby realizing a strategy framework for electrically stimulating different processing regions from shallow to deep in the human visual information processing pathway. Considering that the processing of brain visual signals is a process from shallow to deep, and limited by the low resolution of the implanted electrode array, it is difficult for shallow image features to exhibit intuitive visual effects at low resolution. The present invention proposes an algorithmic process for image processing from low-level features to high-level features, and predicts and simulates neural signals through an electroencephalogram signal predictor for different levels of features, which are conducted to regions from the retina to the V4 region of the visual cortex, and even more advanced visual information processing regions, to achieve multi-site electrical stimulation, so as to achieve a better visual perception effect with the assistance of multi-level features at low resolution. Among them, the advantages of the brain-inspired visual sensor based on the original language in terms of depth, color, light intensity, frame rate, dynamic range, etc. can input the first primitive information representing contour information, the second primitive information representing change information, and the third primitive information representing color information and light intensity information into an image reconstructor, and the image reconstructor outputs a high-speed and high-dynamic-range reconstructed image with a maximum of 10,000 fps. Then, the reconstructed image is segmented using an image segmenter based on a color histogram, etc., and a target segmentation mask for the external environment can be obtained; when the brain-inspired visual sensor is implemented based on a binocular vision system, the depth of the external environment can be collected by the brain-inspired visual sensor to obtain depth information, and both the depth information and the reconstructed image are input into a depth estimator, and a depth image output by the depth estimator is obtained. The primitive combination is realized through the reconstructed image, and the semantic information is represented through the target segmentation mask. Finally, based on the first primitive information representing contour information, the second primitive information representing change information, the third primitive information representing color information and light intensity information, the depth image, the reconstructed image, and the target segmentation mask, multi-modal feature fusion and hierarchical coding are used to predict and simulate neural signals, and then generate multi-region electrical stimulation signals from shallow to deep to electrically stimulate different regions, specifically including electrical stimulation of the retina region, the optic nerve region, and the visual cortex region, etc.

[0110] In this embodiment, basic features such as color information, contour information, change information, and light intensity information from spatio-temporal difference images and color intensity images at the low level can be obtained, a high-frame-rate and high-dynamic-range reconstructed image after fusion can be obtained, and cognitive-level features of the target segmentation mask and spatial perception features including depth can be obtained. The zoned stimulation of multi-level features helps to achieve high-quality visual perception at low resolution.

[0111] In one embodiment, the processor is specifically configured to determine the identifier of the implantable electrode array and the target primitive combination information type corresponding to the target region based on the correspondence between the region, the electrode array identifier, and the primitive combination information type, and determine the information corresponding to the target primitive combination information type from the first primitive information, the second primitive information, and the third primitive information, so as to obtain the target primitive combination information.

[0112] The processor is specifically configured to control the implantable electrode array to generate the electrical stimulation signal based on the target primitive combination information.

[0113] Exemplarily, the correspondence between the region, the electrode array identifier, and the primitive combination information type may be pre-stored. When the identifier of the implantable electrode array implanted in the target region is obtained, the target primitive combination information type corresponding to the target region and the identifier of the implantable electrode array is searched for in the correspondence. Then, the information matching the target primitive combination information type is screened out from the first primitive information representing the contour information, the second primitive information representing the change information, and the third primitive information representing the color information and the light intensity information, and these matching information is used as the target primitive combination information. Then, based on the target primitive combination information, the implantable electrode array implanted in the target region is controlled to generate an electrical stimulation signal. For example, if the target region is the retina region and the identifier of the corresponding implantable electrode array is 1, and the correspondence stores the correspondence between the electrode array identifier 1, the retina region, the light intensity, and the color type, then the target primitive combination information is determined to be the third primitive information representing the combination of the light intensity information and the color information.

[0114] It should be noted that the present invention can also adjust the primitive combination information type in the pre-set correspondence based on the functional partition of the brain and the feedback electroencephalogram signal, and can also adjust the stimulation parameters executable by the implantable electrode array based on the feedback electroencephalogram signal, so as to control the implantable electrode array to generate electrical stimulation signals with different strengths to adapt to individuals. The present invention does not limit this.

[0115] In summary, the present invention designs a software and hardware loop for human-machine two-way interaction including "multi-primitive and primitive combination signal input based on primitive-based brain-inspired vision sensors - parallel implanted electrode array partitioned stimulation - electroencephalogram signal sensing feedback - adaptive adjustment of brain-inspired vision sensors". By using primitive-based brain-inspired vision sensors, it realizes the primitive splitting perception and combined processing of visual information in the external environment, and then constructs a robust perception performance for high-dynamic range scenes and high-speed moving targets, as well as a low-redundancy and efficient extraction ability for information such as color, edges, and motion, achieving sensor signal input closer to the human visual perception and processing mechanism, while taking into account the deployment requirements of low-power and low-bandwidth processors and low-latency human-machine two-way adaptive regulation. A partitioned stimulation strategy is proposed: Utilizing the complementarity and difference of multiple visual primitives, considering the cell diversity functional partitions of the human retina and visual cortex of the brain, and making full use of the characteristics of visual primitives and their combinations to stimulate different cells in the retinal area or different brain regions in a targeted and parallel manner. For example, cone cells are more sensitive to light intensity and color, and the implanted electrode array will apply electrical stimulation signals encoded with light intensity information and color information to cone cells in a targeted manner. The primitive-based brain-inspired visual prosthesis system can achieve visual perception effects beyond the traditional camera-based prosthesis vision paradigm through primitive-based perception, processing, and targeted partitioned stimulation.

[0116] Figure 11 FIG. 4 is a schematic flowchart of a primitive-based partitioned electrical stimulation method provided by an embodiment of the present invention. As Figure 11 shown, it is applied to the above-mentioned primitive-based brain-inspired visual prosthesis system. The primitive-based partitioned electrical stimulation method includes the following steps:

[0117] Step 1101: Receive multiple primitive information in multi-bit form or multiple primitive information in single-bit form sent by a primitive-based brain-inspired vision sensor. The multiple primitive information in multi-bit form is obtained by the brain-inspired vision sensor through primitive extraction of the acquired spatio-temporal differential image and color intensity image. The multiple primitive information in single-bit form is obtained by the brain-inspired vision sensor through a pulse encoder using different encoding methods for each type of primitive information.

[0118] Step 1102: Based on each primitive information in multi-bit form or each primitive information in single-bit form, control an implanted electrode array implanted in different regions of the human visual processing pathway to generate an electrical stimulation signal, and the electrical stimulation signal is used to stimulate the corresponding region through the implanted electrode array implanted in the corresponding region.

[0119] Step 1103: Receive the electroencephalogram signal of the corresponding region collected by the implanted electrode array.

[0120] Step 1104: Based on the EEG signal, generate a regulation signal through an EEG decoding feedback regulator, and adjust the attribute parameters of the brain-inspired vision sensor based on the regulation signal, so as to re-collect the external environment based on the adjusted brain-inspired vision sensor, and stimulate the corresponding area based on the re-collected image.

[0121] The method for partitioned electrical stimulation based on primitives provided by the present invention receives multiple primitive information in multi-bit form or multiple primitive information in single-bit form sent by a brain-inspired vision sensor based on primitives, and controls implantable electrode arrays implanted in different areas of the human visual processing pathway to generate electrical stimulation signals based on each primitive information in multi-bit form or each primitive information in single-bit form. The electrical stimulation signals are used to stimulate the corresponding area through the implantable electrode arrays implanted in the corresponding area; and a regulation signal is generated through an EEG decoding feedback regulator based on the EEG signals of the corresponding area collected by the implantable electrode arrays, and the attribute parameters of the brain-inspired vision sensor are adjusted based on the regulation signal, so as to re-collect the external environment based on the adjusted brain-inspired vision sensor, and stimulate the corresponding area based on the re-collected image. It can be seen that the present invention realizes the extraction and combination of primitives of the external environment by using a brain-inspired vision sensor based on primitives, so as to control implantable electrode arrays implanted in different areas of the human visual processing pathway to generate electrical stimulation signals for stimulating the corresponding area. Since the brain-inspired vision sensor based on primitives has the functions of low power consumption, low bandwidth, high robustness, high frame rate, high dynamic range, and pulse-coded output, the brain-inspired vision prosthesis system can efficiently perceive the external environment and perform partitioned electrical stimulation based on primitive processing, thereby improving the perception and computational efficiency of the brain-inspired vision prosthesis system, especially suitable for coping with the challenge of difficult deployment of a large number of communication and computing resources on the human body and enhancing the user's visual experience; in addition, the brain-inspired vision sensor provided by the present invention can output multiple primitive information in multi-bit form and single-bit form, and the multiple primitive information in single-bit form is more in line with the characteristics of human neural signals, so the computational complexity of the subsequent conversion process from image information to electrical stimulation signals can be greatly reduced in the vision prosthesis system; moreover, the present invention proposes a two-way control framework. From the brain-inspired vision sensor to the human brain, based on the brain-inspired vision sensor perceiving the external environment, the predicted analog neural signals are used to realize electrical stimulation of multiple positions in the visual pathway through the implantable electrode arrays; from the human brain to the brain-inspired vision sensor, based on the implantable electrode arrays perceiving the EEG signals, the EEG signals are decoded to generate a regulation signal, and the attribute parameters of the brain-inspired vision sensor are adjusted based on the regulation signal, so that the brain-inspired vision sensor re-collects the external environment based on the adjusted attribute parameters, and then based on the spatio-temporal differential image and color intensity image obtained by the re-collection, a new electrical stimulation signal is generated, so that the new electrical stimulation signal is more in line with the user, achieving the mutual coordination and adaptation between humans and machines.

[0122] In one embodiment, the multiple primitive information in the multi-bit form includes contour information, variation information, color information, and light intensity information.

[0123] In one embodiment, step 1102 above controls implantable electrode arrays implanted in different regions of the human visual processing pathway to generate electrical stimulation signals based on each piece of primitive information in the multi-bit form or each piece of primitive information in the single-bit form. The electrical stimulation signals are used to stimulate the corresponding regions through the implantable electrode arrays implanted in the corresponding regions, and can be specifically implemented in the following manner:

[0124] Based on the first primitive information and the second primitive information, control the first implantable electrode array implanted in the retinal region to generate a first electrical stimulation signal encoding the first primitive information and the second primitive information. The first electrical stimulation signal is used to stimulate the rod cells in the retinal region; and based on the third primitive information, control the second implantable electrode array implanted in the retinal region to generate a second electrical stimulation signal encoding the third primitive information. The second electrical stimulation signal is used to stimulate the cone cells in the retinal region.

[0125] Among them, the first primitive information is obtained by the brain-like vision sensor using binary encoding for the contour information through a pulse encoder, the second primitive information is obtained by the brain-like vision sensor using time encoding for the variation information through a pulse encoder, and the third primitive information is obtained by the brain-like vision sensor using rate encoding for the color information and the light intensity information through a pulse encoder.

[0126] In one embodiment, step 1102 above controls implantable electrode arrays implanted in different regions of the human visual processing pathway to generate electrical stimulation signals based on each piece of primitive information in the multi-bit form or each piece of primitive information in the single-bit form. The electrical stimulation signals are used to stimulate the corresponding regions through the implantable electrode arrays implanted in the corresponding regions, and can be specifically implemented in the following manner:

[0127] Based on the first primitive information, the second primitive information, and the third primitive information, control the third implantable electrode array implanted in the primary visual cortex V1 region to generate a third electrical stimulation signal encoding the first primitive information, the second primitive information, and the third primitive information. The third electrical stimulation signal is used to stimulate the primary visual cortex V1 region.

[0128] In one embodiment, step 1102 above controls implantable electrode arrays implanted in different regions of the human visual processing pathway to generate electrical stimulation signals based on each piece of primitive information in the multi-bit form or each piece of primitive information in the single-bit form. The electrical stimulation signals are used to stimulate the corresponding regions through the implantable electrode arrays implanted in the corresponding regions, and can be specifically implemented in the following manner:

[0129] Feature extraction is performed on multiple primitive information in multi-bit form to obtain semantic information; based on the first primitive information and the third primitive information, control the fourth implanted electrode array implanted in the primary visual cortex V1 region to generate a fourth electrical stimulation signal encoded with the first primitive information and the third primitive information, and the fourth electrical stimulation signal is used to stimulate the primary visual cortex V1 region; based on the second primitive information, control the fifth implanted electrode array implanted in the visual cortex V2 region to generate a fifth electrical stimulation signal encoded with the second primitive information, and the fifth electrical stimulation signal is used to stimulate the visual cortex V2 region; based on the semantic information, control the sixth implanted electrode array implanted in the visual cortex V4 region to generate a sixth electrical stimulation signal encoded with the semantic information, and the sixth electrical stimulation signal is used to stimulate the visual cortex V4 region.

[0130] In one embodiment, based on each primitive information in single-bit form, controlling the implanted electrode arrays implanted in different regions of the human visual processing pathway to generate electrical stimulation signals can be specifically implemented in the following manner:

[0131] The electroencephalogram signal predictor determines the motion attention information based on the second primitive information, generates the contrast enhancement information based on the third primitive information, and predicts the simulated neural signal based on the motion attention information, the contrast enhancement information, and the first primitive information; determines the stimulation parameters that the implanted electrode array can execute based on the simulated neural signal, and controls the implanted electrode array to generate the electrical stimulation signal based on the stimulation parameters.

[0132] In one embodiment, the primitive-based partitioned electrical stimulation method further includes the following steps:

[0133] Receive the depth information of the external environment collected by the brain-like visual sensor, generate a reconstructed image based on the first primitive information, the second primitive information, and the third primitive information through an image reconstructor, segment the reconstructed image through an image segmenter to obtain a target segmentation mask, and generate a depth image based on the depth information and the reconstructed image through a depth estimator; finally, predict the simulated neural signal based on the first primitive information, the second primitive information, the third primitive information, the depth image, the reconstructed image, and the target segmentation mask through the electroencephalogram signal predictor.

[0134] In one embodiment, generating a regulation signal based on the electroencephalogram signal through an electroencephalogram decoding feedback regulator can be specifically implemented in the following manner:

[0135] Feature extraction is performed on the EEG signals through an EEG decoding feedback regulator to obtain an intensity matching feature and a frequency synchronization feature; when at least one of the following preset conditions is not satisfied, the regulation signal is generated, and the preset conditions include: the intensity matching feature indicates that the pulse intensity of the electrical stimulation signal reaches the neural activation threshold, and the frequency synchronization feature indicates that the pulse frequency of the electrical stimulation signal resonates with the neural oscillation.

[0136] In one embodiment, the method for partitioned electrical stimulation based on primitives further includes the following steps:

[0137] Based on the correspondence relationship between the region, the electrode array identifier, and the type of primitive combination information, determine the identifier of the implantable electrode array and the type of target primitive combination information corresponding to the target region, and determine the information corresponding to the type of target primitive combination information from the first primitive information, the second primitive information, and the third primitive information to obtain the target primitive combination information; and control the implantable electrode array to generate the electrical stimulation signal based on the target primitive combination information.

[0138] It should be noted that for the specific implementation process of the method for partitioned electrical stimulation based on primitives, reference can be made to the specific description in the above-mentioned primitive-based brain-like visual prosthesis system, and the present invention will not elaborate herein.

[0139] Next, the primitive-based partitioned electrical stimulation device provided by the present invention will be described. The primitive-based partitioned electrical stimulation device described below can be mutually corresponding and referred to the primitive-based partitioned electrical stimulation method described above.

[0140] The primitive-based partitioned electrical stimulation device includes a first receiving unit, a control unit, a second receiving unit, and an adjustment unit, where:

[0141] The first receiving unit is configured to receive multiple primitive information in multi-bit form or multiple primitive information in single-bit form sent by a primitive-based brain-like visual sensor. The multiple primitive information in multi-bit form is obtained by the brain-like visual sensor performing primitive extraction on the collected spatio-temporal differential image and color intensity image. The multiple primitive information in single-bit form is obtained by the brain-like visual sensor encoding each type of primitive information using different encoding methods through a pulse encoder;

[0142] The control unit is configured to control implantable electrode arrays implanted in different regions of the human visual processing pathway to generate electrical stimulation signals based on each primitive information in multi-bit form or each primitive information in single-bit form. The electrical stimulation signals are used to stimulate the corresponding regions through the implantable electrode arrays implanted in the corresponding regions;

[0143] A second receiving unit, configured to receive electroencephalogram signals of a corresponding region collected by the implantable electrode array;

[0144] An adjustment unit, configured to generate a control signal based on the electroencephalogram signals through an electroencephalogram decoding feedback regulator, and adjust attribute parameters of the brain-like vision sensor based on the control signal, so as to re-collect the external environment based on the adjusted brain-like vision sensor, and stimulate the corresponding region based on the re-collected image.

[0145] Figure 12 is a schematic physical structure diagram of an electronic device provided by an embodiment of the present invention. As Figure 12 shown, the electronic device may include: a processor 1210, a communication interface 1220, a memory 1230, and a communication bus 1240. Among them, the processor 1210, the communication interface 1220, and the memory 1230 communicate with each other through the communication bus 1240. The processor 1210 may call logic instructions in the memory 1230 to execute a primitives-based partitioned electrical stimulation method, which includes: receiving multiple primitives information in a multi-bit form or multiple primitives information in a single-bit form sent by a brain-like vision sensor based on primitives, where the multiple primitives information in the multi-bit form is obtained by the brain-like vision sensor performing primitives extraction on the collected spatio-temporal differential image and color intensity image, and the multiple primitives information in the single-bit form is obtained by the brain-like vision sensor encoding each type of primitives information through a pulse encoder using different encoding methods;

[0146] Based on each piece of primitives information in the multi-bit form or each piece of primitives information in the single-bit form, controlling an implantable electrode array implanted in different regions of the human visual processing pathway to generate an electrical stimulation signal, where the electrical stimulation signal is used to stimulate the corresponding region through the implantable electrode array implanted in the corresponding region;

[0147] Receiving electroencephalogram signals of a corresponding region collected by the implantable electrode array;

[0148] Generating a control signal based on the electroencephalogram signals through an electroencephalogram decoding feedback regulator, and adjusting attribute parameters of the brain-like vision sensor based on the control signal, so as to re-collect the external environment based on the adjusted brain-like vision sensor, and stimulate the corresponding region based on the re-collected image.

[0149] In addition, when the logical instructions in the above-mentioned memory 1230 are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a part of this technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in various embodiments of the present invention. The foregoing storage medium includes: various media such as USB flash drives, mobile hard disks, read-only memories (ROM, Read-Only Memory), random access memories (RAM, Random Access Memory), magnetic disks, or optical discs that can store program codes.

[0150] On the other hand, the present invention also provides a computer program product. The computer program product includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the primitives-based partitioned electrical stimulation method provided by the above-mentioned various methods. The method includes: receiving multiple primitives information in multi-bit form or multiple primitives information in single-bit form sent by a primitives-based brain-inspired vision sensor, where the multiple primitives information in multi-bit form is obtained by the brain-inspired vision sensor performing primitives extraction on the collected spatio-temporal difference image and color intensity image, and the multiple primitives information in single-bit form is obtained by the brain-inspired vision sensor encoding each type of primitives information using different encoding methods through a pulse encoder;

[0151] Based on each primitives information in multi-bit form or each primitives information in single-bit form, controlling an implantable electrode array implanted in different regions of the human visual processing pathway to generate an electrical stimulation signal, where the electrical stimulation signal is used to stimulate the corresponding region through the implantable electrode array implanted in the corresponding region;

[0152] Receiving the electroencephalogram signals of the corresponding region collected by the implantable electrode array;

[0153] Generating a regulation signal through an electroencephalogram decoding feedback regulator based on the electroencephalogram signals, and adjusting the attribute parameters of the brain-inspired vision sensor based on the regulation signal, so as to re-collect the external environment based on the adjusted brain-inspired vision sensor, and stimulate the corresponding region based on the re-collected image.

[0154] In another aspect, the present invention also provides a non-transitory computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, it implements a primitive-based partitioned electrical stimulation method provided by the above-mentioned various methods. The method includes: receiving a plurality of primitive information in multi-bit form or a plurality of primitive information in single-bit form sent by a primitive-based brain-inspired vision sensor. The plurality of primitive information in multi-bit form is obtained by the brain-inspired vision sensor performing primitive extraction on the acquired spatio-temporal differential image and color intensity image. The plurality of primitive information in single-bit form is obtained by the brain-inspired vision sensor encoding each type of primitive information through a pulse encoder using different encoding methods;

[0155] Based on each piece of primitive information in multi-bit form or each piece of primitive information in single-bit form, controlling an implantable electrode array implanted in different regions of the human visual processing pathway to generate an electrical stimulation signal, and the electrical stimulation signal is used to stimulate the corresponding region through the implantable electrode array implanted in the corresponding region;

[0156] Receiving the electroencephalogram signal of the corresponding region collected by the implantable electrode array;

[0157] Through an electroencephalogram decoding feedback regulator, generating a regulation signal based on the electroencephalogram signal, and adjusting the attribute parameters of the brain-inspired vision sensor based on the regulation signal, so as to re-collect the external environment based on the adjusted brain-inspired vision sensor, and stimulate the corresponding region based on the re-collected image.

[0158] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separated. The components shown as units may or may not be physical units, that is, they may be located in one place or distributed to multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment. A person of ordinary skill in the art can understand and implement it without creative labor.

[0159] Through the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on such an understanding, the above technical solution, in essence, or the part that contributes to the prior art can be embodied in the form of a software product. The computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disc, etc., and includes several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0160] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions described in the foregoing embodiments, or perform equivalent replacements for some of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A primitives-based brain-like visual prosthesis system, characterized in that It includes a primitive-based brain-inspired vision sensor, an implantable electrode array, and a processor. The primitive-based brain-inspired vision sensor and the implantable electrode array are both connected to the processor; The brain-inspired vision sensor is configured to collect the external environment to obtain a spatio-temporal difference image and a color intensity image, perform primitive extraction on the spatio-temporal difference image and the color intensity image to obtain multiple primitive information in multi-bit form, and use different coding methods for each type of primitive information through a pulse encoder to encode and obtain multiple primitive information in single-bit form, and send the multiple primitive information in multi-bit form or the multiple primitive information in single-bit form to the processor; The processor is configured to, based on each primitive information in multi-bit form or each primitive information in single-bit form, control the implantable electrode array implanted in different regions of the human visual processing pathway to generate an electrical stimulation signal, and the electrical stimulation signal is used to stimulate the corresponding region through the implantable electrode array implanted in the corresponding region; The implantable electrode array is configured to collect the electroencephalogram signal of the corresponding region and send the electroencephalogram signal to the processor; The processor is further configured to generate a regulation signal based on the electroencephalogram signal through an electroencephalogram decoding feedback regulator, and adjust the attribute parameters of the brain-inspired vision sensor based on the regulation signal, so as to re-collect the external environment based on the adjusted brain-inspired vision sensor and stimulate the corresponding region based on the re-collected image.

2. The brain-like vision prosthesis system based on the source language according to claim 1, wherein The multiple primitive information in multi-bit form includes contour information, change information, color information, and light intensity information; The brain-inspired vision sensor is specifically configured to perform primitive extraction on the spatio-temporal difference image to obtain the contour information and change information of the external environment, perform primitive extraction on the color intensity image to obtain the color information and light intensity information of the external environment, and use binary coding for the contour information through the pulse encoder to obtain the first primitive information in single-bit form, use time coding for the change information to obtain the second primitive information in single-bit form, and use rate coding for the color information and the light intensity information to obtain the third primitive information in single-bit form.

3. The primitive-based brain-inspired vision prosthesis system according to claim 2, wherein The processor is specifically configured to, based on the first primitive information and the second primitive information, control the first implantable electrode array implanted in the retinal region to generate a first electrical stimulation signal encoded with the first primitive information and the second primitive information, and the first electrical stimulation signal is used to stimulate the rod cells in the retinal region; The processor is specifically configured to, based on the third primitive information, control the second implantable electrode array implanted in the retinal region to generate a second electrical stimulation signal encoded with the third primitive information, and the second electrical stimulation signal is used to stimulate the cone cells in the retinal region.

4. The primitive-based brain-inspired vision prosthesis system according to claim 2, wherein The processor is specifically configured to control a third implantable electrode array implanted in the primary visual cortex V1 region to generate a third electrical stimulation signal encoding the first primitive information, the second primitive information, and the third primitive information based on the first primitive information, the second primitive information, and the third primitive information, and the third electrical stimulation signal is used to stimulate the primary visual cortex V1 region.

5. The primitive-based brain-like visual prosthesis system according to claim 2, wherein the processor is specifically configured to perform feature extraction on a plurality of primitive information in a multi-bit form to obtain semantic information; the processor is specifically configured to control a fourth implantable electrode array implanted in the primary visual cortex V1 region to generate a fourth electrical stimulation signal encoding the first primitive information and the third primitive information based on the first primitive information and the third primitive information, and the fourth electrical stimulation signal is used to stimulate the primary visual cortex V1 region; the processor is specifically configured to control a fifth implantable electrode array implanted in the visual cortex V2 region to generate a fifth electrical stimulation signal encoding the second primitive information based on the second primitive information, and the fifth electrical stimulation signal is used to stimulate the visual cortex V2 region; the processor is specifically configured to control a sixth implantable electrode array implanted in the visual cortex V4 region to generate a sixth electrical stimulation signal encoding the semantic information based on the semantic information, and the sixth electrical stimulation signal is used to stimulate the visual cortex V4 region.

6. The primitive-based brain-like visual prosthesis system according to claim 2, wherein the processor is specifically configured to determine motion attention information based on the second primitive information through an electroencephalogram signal predictor, generate contrast enhancement information based on the third primitive information, and predict simulated neural signals based on the motion attention information, the contrast enhancement information, and the first primitive information; the processor is specifically configured to determine stimulation parameters executable by the implantable electrode array based on the simulated neural signals, and control the implantable electrode array to generate the electrical stimulation signal based on the stimulation parameters.

7. The primitive-based brain-like visual prosthesis system according to claim 6, wherein the brain-like visual sensor is configured to collect the depth of the external environment to obtain depth information and send the depth information to the processor; the processor is further configured to generate a reconstructed image based on the first primitive information, the second primitive information, and the third primitive information through an image reconstructor, segment the reconstructed image through an image segmenter to obtain a target segmentation mask, and generate a depth image based on the depth information and the reconstructed image through a depth estimator; the processor is further configured to predict the simulated neural signals through the electroencephalogram signal predictor based on the first primitive information, the second primitive information, the third primitive information, the depth image, the reconstructed image, and the target segmentation mask.

8. The primitive-based brain-like visual prosthesis system according to claim 1, wherein The processor is specifically configured to extract features from the EEG signals through an EEG decoding feedback regulator to obtain intensity matching features and frequency synchronization features; The processor is specifically configured to generate the regulation signal when at least one of the following preset conditions is not satisfied. The preset conditions include: the intensity matching feature indicates that the pulse intensity of the electrical stimulation signal reaches the nerve activation threshold, and the frequency synchronization feature indicates that the pulse frequency of the electrical stimulation signal resonates with the neural oscillation.

9. The brain-like visual prosthesis system based on primitives according to any one of claims 2-7, wherein The processor is specifically configured to determine the identifier of the implantable electrode array and the target primitive combination information type corresponding to the target area based on the correspondence between the area, the electrode array identifier, and the primitive combination information type, and determine the information corresponding to the target primitive combination information type from the first primitive information, the second primitive information, and the third primitive information to obtain the target primitive combination information; The processor is specifically configured to control the implantable electrode array to generate the electrical stimulation signal based on the target primitive combination information.

10. A primitives-based partitioned electrical stimulation device, characterized in that, Comprising: A first receiving unit, configured to receive multiple primitive information in a multi-bit form or multiple primitive information in a single-bit form sent by a brain-like visual sensor based on primitives. The multiple primitive information in the multi-bit form is obtained by the brain-like visual sensor performing primitive extraction on the acquired spatio-temporal differential image and color intensity image. The multiple primitive information in the single-bit form is obtained by the brain-like visual sensor encoding each type of primitive information using different encoding methods through a pulse encoder; A control unit, configured to control implantable electrode arrays implanted in different areas of the human visual processing pathway to generate electrical stimulation signals based on each primitive information in the multi-bit form or each primitive information in the single-bit form. The electrical stimulation signals are used to stimulate the corresponding areas through the implantable electrode arrays implanted in the corresponding areas; A second receiving unit, configured to receive the EEG signals of the corresponding areas collected by the implantable electrode arrays; An adjustment unit, configured to generate a regulation signal based on the EEG signals through an EEG decoding feedback regulator, and adjust the attribute parameters of the brain-like visual sensor based on the regulation signal, so as to re-collect the external environment based on the adjusted brain-like visual sensor, and stimulate the corresponding areas based on the re-collected images.

11. An electronic device, comprising a memory, a processor, and a computer program stored on the memory and running on the processor, characterized in that, When the processor executes the computer program, the following steps are implemented: Receive multiple primitive information in a multi-bit form or multiple primitive information in a single-bit form sent by a brain-like visual sensor based on primitives. The multiple primitive information in the multi-bit form is obtained by the brain-like visual sensor performing primitive extraction on the acquired spatio-temporal differential image and color intensity image. The multiple primitive information in the single-bit form is obtained by the brain-like visual sensor encoding each type of primitive information using different encoding methods through a pulse encoder; Based on the primitive information in multi-bit form or the primitive information in single-bit form, control the implantable electrode arrays implanted in different regions of the human visual processing pathway to generate electrical stimulation signals, and the electrical stimulation signals are used to stimulate the corresponding regions through the implantable electrode arrays implanted in the corresponding regions; Receive the electroencephalogram signals of the corresponding regions collected by the implantable electrode arrays; Based on the electroencephalogram signals, generate a regulation signal through an electroencephalogram decoding feedback regulator, and adjust the attribute parameters of the brain-like visual sensor based on the regulation signal, so as to re-collect the external environment based on the adjusted brain-like visual sensor, and stimulate the corresponding regions based on the re-collected images.

12. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, the following steps are implemented: Receive multiple primitive information in multi-bit form or multiple primitive information in single-bit form sent by a brain-like visual sensor based on primitives. The multiple primitive information in multi-bit form is obtained by the brain-like visual sensor performing primitive extraction on the collected spatio-temporal differential images and color intensity images, and the multiple primitive information in single-bit form is obtained by the brain-like visual sensor encoding each type of primitive information through a pulse encoder using different encoding methods; Based on the primitive information in multi-bit form or the primitive information in single-bit form, control the implantable electrode arrays implanted in different regions of the human visual processing pathway to generate electrical stimulation signals, and the electrical stimulation signals are used to stimulate the corresponding regions through the implantable electrode arrays implanted in the corresponding regions; Receive the electroencephalogram signals of the corresponding regions collected by the implantable electrode arrays; Based on the electroencephalogram signals, generate a regulation signal through an electroencephalogram decoding feedback regulator, and adjust the attribute parameters of the brain-like visual sensor based on the regulation signal, so as to re-collect the external environment based on the adjusted brain-like visual sensor, and stimulate the corresponding regions based on the re-collected images.

13. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, the following steps are implemented: Receive multiple primitive information in multi-bit form or multiple primitive information in single-bit form sent by a brain-like visual sensor based on primitives. The multiple primitive information in multi-bit form is obtained by the brain-like visual sensor performing primitive extraction on the collected spatio-temporal differential images and color intensity images, and the multiple primitive information in single-bit form is obtained by the brain-like visual sensor encoding each type of primitive information through a pulse encoder using different encoding methods; Based on the primitive information in multi-bit form or the primitive information in single-bit form, control the implantable electrode arrays implanted in different regions of the human visual processing pathway to generate electrical stimulation signals, and the electrical stimulation signals are used to stimulate the corresponding regions through the implantable electrode arrays implanted in the corresponding regions; Receive the electroencephalogram signals of the corresponding regions collected by the implantable electrode arrays; Based on the EEG signal, the EEG decoding feedback regulator generates a regulation signal, and adjusts the attribute parameters of the brain-inspired vision sensor based on the regulation signal, so as to re-collect the external environment based on the adjusted brain-inspired vision sensor, and stimulate the corresponding area based on the re-collected image.

Images