Primitive-based brain-vision-like prosthesis system and partitioned electrical stimulation method and device

Abstract

The invention provides a primitive-based brain-like visual prosthesis system, a partitioned electrical stimulation method and a partitioned electrical stimulation device, and relates to the technical field of visual prostheses. A primitive-based brain-like visual sensor acquires a space-time difference image and a color intensity image, performs primitive extraction to obtain multiple pieces of primitive information in a multi-bit form; encoding by adopting different encoding modes through a pulse encoder to obtain a plurality of pieces of primitive information in a single-bit form; and the processor controls the implantable electrode arrays in different areas to generate electrical stimulation signals based on the primitive information in a multi-bit form or a single-bit form, generates regulation and control signals based on the fed-back electroencephalogram signals, and adjusts attribute parameters of the brain-like vision sensor based on the regulation and control signals. Based on the characteristics of low bandwidth, high robustness and pulse coding of the brain-like vision sensor, the brain-like vision prosthesis system can efficiently perceive the external environment and perform partitioned electrical stimulation based on primitive processing, so that the visual experience of a user is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of visual prostheses, and in particular to a brain-like visual prosthesis system based on primitives, a method and device for zonal electrical stimulation. Background Art

[0002] Vision is one of the most important senses for humans to perceive the world. However, retinal degenerative diseases can cause photoreceptors (cone cells and rod cells) to gradually apoptose, leading patients to irreversible vision loss or even total blindness. A visual prosthesis is a device that treats severe vision loss by stimulating nerve cells at any position in the visual pathway (usually using electrical pulses).

[0003] In related technologies, as Figure 1 shown, a visual prosthesis system usually uses a patient-worn camera to collect videos of the external environment. The videos are analyzed by a processor to generate analog nerve signals, and electrical stimulation is applied to the V1 area of the visual cortex or the retinal area through an electrode array such as a Utah Electrode Array (UEA) or a Wireless Floating Microelectrode Array (WFMA). The human eye then stimulates the initial visual cortex V1 area through the lateral geniculate body, and the initial visual cortex V1 area further stimulates other areas of the visual cortex, causing the patient to have a phosphene phenomenon. In addition, the processor can also perform phosphene simulation based on the video and display the result of the phosphene simulation through a display to know the content seen by the patient.

[0004] However, in the above related technologies, the information collected by a traditional camera for the external environment is limited, and there is a large gap from the information seen by the human eye. When there are high-speed moving objects in the external environment, motion blur will occur, and underexposure or overexposure is likely to occur in high-dynamic range scenes. Moreover, the repetition rate between adjacent two frames of images in the collected videos is relatively high, resulting in low bandwidth utilization of data transmission, thus leading to relatively inefficient perception of the visual prosthesis system. Summary of the Invention

[0005] The present invention provides a brain-like visual prosthesis system based on primitives, a method and device for zonal electrical stimulation, so as to solve the defect that the perception of the visual prosthesis system in the prior art is relatively inefficient.

[0006] The present invention provides a brain-like visual prosthesis system based on primitives, including a brain-like visual sensor based on primitives, an implantable electrode array, and a processor. The brain-like visual sensor based on primitives and the implantable electrode array are both connected to the processor; The brain-inspired vision sensor is used to collect the external environment to obtain spatio-temporal difference images and color intensity images, extract primitives from the spatio-temporal difference images and the color intensity images to obtain multiple primitive information in multi-bit form, and encode the multiple primitive information in single-bit form by using different encoding methods for each type of primitive information through a pulse encoder, and send the multiple primitive information in multi-bit form or the multiple primitive information in single-bit form to the processor; The processor is used to control an implantable electrode array implanted in different regions of the human visual processing pathway to generate an electrical stimulation signal based on each primitive information in multi-bit form or each primitive information in single-bit form, and the electrical stimulation signal is used to stimulate the corresponding region through the implantable electrode array implanted in the corresponding region; The implantable electrode array is used to collect the electroencephalogram signals of the corresponding region and send the electroencephalogram signals to the processor; The processor is further used to generate a regulation signal based on the electroencephalogram signals through an electroencephalogram decoding feedback regulator, and adjust the attribute parameters of the brain-inspired vision sensor based on the regulation signal, so as to re-collect the external environment based on the adjusted brain-inspired vision sensor and stimulate the corresponding region based on the re-collected images.

[0007] According to a primitive-based brain-inspired vision prosthesis system provided by the present invention, the multiple primitive information in multi-bit form includes contour information, change information, color information, and light intensity information; The brain-inspired vision sensor is specifically used to extract primitives from the spatio-temporal difference images to obtain the contour information and change information of the external environment, extract primitives from the color intensity images to obtain the color information and light intensity information of the external environment, and use binary encoding for the contour information through the pulse encoder to obtain first primitive information in single-bit form, use time encoding for the change information to obtain second primitive information in single-bit form, and use rate encoding for the color information and the light intensity information to obtain third primitive information in single-bit form. According to a primitive-based brain-inspired vision prosthesis system provided by the present invention, the processor is specifically used to control a first implantable electrode array implanted in the retinal region to generate a first electrical stimulation signal encoded with the first primitive information and the second primitive information based on the first primitive information and the second primitive information, and the first electrical stimulation signal is used to stimulate the rod cells in the retinal region; The processor is specifically used to control a second implantable electrode array implanted in the retinal region to generate a second electrical stimulation signal encoded with the third primitive information based on the third primitive information, and the second electrical stimulation signal is used to stimulate the cone cells in the retinal region.

[0008] According to a primitives-based brain-inspired visual prosthesis system provided by the present invention, the processor is specifically configured to control a third implantable electrode array implanted in the primary visual cortex V1 region to generate a third electrical stimulation signal encoding the first primitive information, the second primitive information, and the third primitive information based on the first primitive information, the second primitive information, and the third primitive information, and the third electrical stimulation signal is used to stimulate the primary visual cortex V1 region.

[0009] According to a primitives-based brain-inspired visual prosthesis system provided by the present invention, the processor is specifically configured to perform feature extraction on a plurality of primitive information in a multi-bit form to obtain semantic information; The processor is specifically configured to control a fourth implantable electrode array implanted in the primary visual cortex V1 region to generate a fourth electrical stimulation signal encoding the first primitive information and the third primitive information based on the first primitive information and the third primitive information, and the fourth electrical stimulation signal is used to stimulate the primary visual cortex V1 region; The processor is specifically configured to control a fifth implantable electrode array implanted in the visual cortex V2 region to generate a fifth electrical stimulation signal encoding the second primitive information based on the second primitive information, and the fifth electrical stimulation signal is used to stimulate the visual cortex V2 region; The processor is specifically configured to control a sixth implantable electrode array implanted in the visual cortex V4 region to generate a sixth electrical stimulation signal encoding the semantic information based on the semantic information, and the sixth electrical stimulation signal is used to stimulate the visual cortex V4 region.

[0010] According to a primitives-based brain-inspired visual prosthesis system provided by the present invention, the processor is specifically configured to determine motion attention information based on the second primitive information through an electroencephalogram signal predictor, generate contrast enhancement information based on the third primitive information, and predict analog neural signals based on the motion attention information, the contrast enhancement information, and the first primitive information; The processor is specifically configured to determine stimulation parameters executable by the implantable electrode array based on the analog neural signals, and control the implantable electrode array to generate the electrical stimulation signal based on the stimulation parameters.

[0011] According to a primitives-based brain-inspired visual prosthesis system provided by the present invention, the brain-inspired visual sensor is configured to collect the depth of the external environment to obtain depth information, and send the depth information to the processor; The processor is further configured to generate a reconstructed image based on the first primitive information, the second primitive information, and the third primitive information through an image reconstructor, segment the reconstructed image through an image segmenter to obtain a target segmentation mask, and generate a depth image based on the depth information and the reconstructed image through a depth estimator; The processor is further configured to predict the simulated neural signal through the electroencephalogram signal predictor based on the first primitive information, the second primitive information, the third primitive information, the depth image, the reconstructed image, and the target segmentation mask.

[0012] According to a primitive-based brain-like visual prosthesis system provided by the present invention, the processor is specifically configured to extract features from the electroencephalogram signal through an electroencephalogram decoding feedback regulator to obtain an intensity matching feature and a frequency synchronization feature; The processor is specifically configured to generate the regulation signal when at least one of the following preset conditions is not satisfied, where the preset conditions include: the intensity matching feature indicates that the pulse intensity of the electrical stimulation signal reaches the nerve activation threshold, and the frequency synchronization feature indicates that the pulse frequency of the electrical stimulation signal resonates with the nerve oscillation.

[0013] According to a primitive-based brain-like visual prosthesis system provided by the present invention, the processor is specifically configured to determine the identifier of the implantable electrode array and the target primitive combination information type corresponding to the target area based on the correspondence relationship between the area, the electrode array identifier, and the primitive combination information type, and determine the information corresponding to the target primitive combination information type from the first primitive information, the second primitive information, and the third primitive information to obtain the target primitive combination information; The processor is specifically configured to control the implantable electrode array to generate the electrical stimulation signal based on the target primitive combination information.

[0014] The present invention further provides a primitive-based partitioned electrical stimulation method applied to any of the above-mentioned primitive-based brain-like visual prosthesis systems, and the method includes: Receiving multiple primitive information in a multi-bit form or multiple primitive information in a single-bit form sent by a primitive-based brain-like visual sensor, where the multiple primitive information in the multi-bit form is obtained by the brain-like visual sensor performing primitive extraction on the acquired spatio-temporal differential image and color intensity image, and the multiple primitive information in the single-bit form is obtained by the brain-like visual sensor encoding each primitive information type through a pulse encoder using different encoding methods; Based on each primitive information in multi-bit form or each primitive information in single-bit form, control the implantable electrode array implanted in different regions of the human visual processing pathway to generate an electrical stimulation signal, where the electrical stimulation signal is used to stimulate the corresponding region through the implantable electrode array implanted in the corresponding region; Receive the electroencephalogram signal of the corresponding region collected by the implantable electrode array; Based on the electroencephalogram signal, generate a regulation signal through an electroencephalogram decoding feedback regulator, and adjust the attribute parameters of the brain-like visual sensor based on the regulation signal, so as to re-collect the external environment based on the adjusted brain-like visual sensor, and stimulate the corresponding region based on the re-collected image.

[0015] The present invention also provides a primitive-based partitioned electrical stimulation device, including: A first receiving unit, configured to receive multiple primitive information in multi-bit form or multiple primitive information in single-bit form sent by a brain-like visual sensor based on primitives, where the multiple primitive information in multi-bit form is obtained by the brain-like visual sensor performing primitive extraction on the collected spatio-temporal differential image and color intensity image, and the multiple primitive information in single-bit form is obtained by the brain-like visual sensor encoding each type of primitive information through a pulse encoder using different encoding methods; A control unit, configured to control the implantable electrode array implanted in different regions of the human visual processing pathway to generate an electrical stimulation signal based on each primitive information in multi-bit form or each primitive information in single-bit form, where the electrical stimulation signal is used to stimulate the corresponding region through the implantable electrode array implanted in the corresponding region; A second receiving unit, configured to receive the electroencephalogram signal of the corresponding region collected by the implantable electrode array; An adjustment unit, configured to generate a regulation signal through an electroencephalogram decoding feedback regulator based on the electroencephalogram signal, and adjust the attribute parameters of the brain-like visual sensor based on the regulation signal, so as to re-collect the external environment based on the adjusted brain-like visual sensor, and stimulate the corresponding region based on the re-collected image.

[0016] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored on the memory and executable on the processor, where when the processor executes the computer program, it implements the primitive-based partitioned electrical stimulation method as described in any one of the above.

[0017] The present invention also provides a non-transitory computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, it implements the primitive-based partitioned electrical stimulation method as described in any one of the above.

[0018] The present invention also provides a computer program product, including a computer program which, when executed by a processor, implements the primitive-based partitioned electrical stimulation method as described in any one of the above.

[0019] The brain-inspired visual prosthesis system, zoned electrical stimulation method and device provided by the present invention. The system includes a brain-inspired visual sensor based on primitives, an implantable electrode array, and a processor. The brain-inspired visual sensor is used to collect the external environment to obtain spatio-temporal differential images and color intensity images, extract primitives from the spatio-temporal differential images and color intensity images to obtain multiple primitive information in multi-bit form, and use a pulse encoder to encode each primitive information type using different encoding methods to obtain multiple primitive information in single-bit form. The processor is used to control the implantable electrode array implanted in different regions of the human visual processing pathway to generate electrical stimulation signals based on each primitive information in multi-bit form or each primitive information in single-bit form. The electrical stimulation signals are used to stimulate the corresponding regions through the implantable electrode array implanted in the corresponding regions. The processor is also used to generate a regulation signal through a brain electrical decoding feedback regulator based on the brain electrical signals of the corresponding regions collected by the implantable electrode array, and adjust the attribute parameters of the brain-inspired visual sensor based on the regulation signal, so as to re-collect the external environment based on the adjusted brain-inspired visual sensor, and stimulate the corresponding regions based on the re-collected images. It can be seen that the present invention realizes the extraction and combination of primitives of the external environment by using a brain-inspired visual sensor based on primitives, so as to control the implantable electrode array implanted in different regions of the human visual processing pathway to generate electrical stimulation signals for stimulating the corresponding regions. Since the brain-inspired visual sensor based on primitives has the functions of low power consumption, low bandwidth, high robustness, high frame rate, high dynamic range, and pulse coding output, the brain-inspired visual prosthesis system can efficiently perceive the external environment and perform zoned electrical stimulation based on primitive processing, thereby improving the perception and computational efficiency of the brain-inspired visual prosthesis system, especially suitable for coping with the challenge of difficult deployment of a large number of communication and computing resources on the human body, and enhancing the user's visual experience. In addition, the brain-inspired visual sensor provided by the present invention can output multiple primitive information in multi-bit form and single-bit form. The multiple primitive information in single-bit form is more in line with the characteristics of human neural signals, so it can greatly reduce the computational complexity of the subsequent conversion process from image information to electrical stimulation signals in the visual prosthesis system. Moreover, the present invention proposes a two-way control framework. From the brain-inspired visual sensor to the human brain, based on the brain-inspired visual sensor perceiving the external environment, the predicted analog neural signals are used to realize electrical stimulation of multiple positions in the visual pathway through the implantable electrode array. From the human brain to the brain-inspired visual sensor, based on the implantable electrode array perceiving the brain electrical signals, the brain electrical signals are decoded to generate a regulation signal, and the attribute parameters of the brain-inspired visual sensor are adjusted based on the regulation signal, so that the brain-inspired visual sensor re-collects the external environment based on the adjusted attribute parameters, and then generates new electrical stimulation signals based on the re-collected spatio-temporal differential images and color intensity images, making the new electrical stimulation signals more in line with the user and achieving the mutual coordination and adaptation between humans and machines. Description of the Drawings

[0020] To more clearly illustrate the technical solutions in the present invention or the prior art, the following will briefly introduce the accompanying drawings required for the description of the embodiments or the prior art. Obviously, the accompanying drawings in the following description are some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other accompanying drawings can also be obtained based on these drawings.

[0021] Figure 1 It is a schematic structural diagram of a visual prosthesis system in the prior art.

[0022] Figure 2 It is a schematic diagram of the imaging effect of an active pixel array in the prior art.

[0023] Figure 3 It is a schematic diagram of the imaging effect of a differential pixel array in the prior art.

[0024] Figure 4 It is one of the schematic structural diagrams of a primitive-based brain-like visual prosthesis system provided by an embodiment of the present invention.

[0025] Figure 5 It is another schematic structural diagram of a primitive-based brain-like visual prosthesis system provided by an embodiment of the present invention.

[0026] Figure 6 It is the third schematic structural diagram of a primitive-based brain-like visual prosthesis system provided by an embodiment of the present invention.

[0027] Figure 7 It is the fourth schematic structural diagram of a primitive-based brain-like visual prosthesis system provided by an embodiment of the present invention.

[0028] Figure 8 It shows a comparison of a schematic diagram of an original image and an electrical stimulation signal in the prior art.

[0029] Figure 9 It is the fifth schematic structural diagram of a primitive-based brain-like visual prosthesis system provided by an embodiment of the present invention.

[0030] Figure 10 It is the sixth schematic structural diagram of a primitive-based brain-like visual prosthesis system provided by an embodiment of the present invention.

[0031] Figure 11 It is a schematic flowchart of a primitive-based partitioned electrical stimulation method provided by an embodiment of the present invention.

[0032] Figure 12 It is a schematic physical structure diagram of an electronic device provided by an embodiment of the present invention. Detailed implementation manners

[0033] To make the objectives, technical solutions, and advantages of the present invention clearer, the following will clearly and completely describe the technical solutions in the present invention in conjunction with the accompanying drawings in the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all of the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments in the present invention without making creative efforts fall within the protection scope of the present invention.

[0034] Before introducing the original-language-based brain-inspired visual prosthesis system of the present invention, the content related to the present invention will be introduced first: In the human visual processing pathway, the splitting and combination of visual primitives play a crucial role. Initially, photoreceptor cells (cone cells and rod cells) sense the light entering the eye and convert the light signal into an electrical signal. These electrical signals are gradually transmitted and integrated through the complex neuronal network in the retina, such as bipolar cells and ganglion cells, to form visual primitives, including basic visual features such as edges, changes, colors, and intensities. When the signal is transmitted to the primary visual cortex (V1) region, the primitive extraction further unfolds. There are a large number of functionally specialized neurons in the V1 region, and these specialized neurons have selective responses to different visual primitives. For example, some neurons only have a strong response to edges in a specific direction, thus disassembling the complex visual scene into discrete primitive representations. Subsequently, the signal encoded with primitive information is transmitted to higher-level visual cortex regions, such as the secondary visual cortex (V2), visual area 4 (V4), and inferotemporal cortex (IT) regions. In these regions, primitive combination and depth processing are realized. Neurons in different regions cooperate to combine the primitives. Neurons in the V2 region are sensitive to simple shapes formed by the combination of multiple edges, and neurons in the IT region can specifically encode complex object shapes and categories. At the same time, the brain also calibrates the primitive combination and processing results with the help of context information, past memories, etc., and finally constructs a complete perception and understanding of the three-dimensional depth, semantic understanding, motion tracking, etc. of the complex visual world in the external environment.

[0035] Brain-Computer Interface (BCI) is a direct communication pathway between the electrical activities of the brain and external devices (most commonly computers or robotic limbs). BCI is typically used in areas such as mapping, enhancing or repairing human cognition, and enhancing or repairing human sensorimotor functions. Based on the distance between the electrodes and the brain tissue, the implementation scope of BCI can be divided into non-invasive, such as electroencephalography (EEG), magnetoencephalography (MEG), electrooculography (EOG), magnetic resonance imaging (MRI), partially invasive, such as electrocorticography (ECoG) and intravascular, and invasive (microelectrode arrays). BCI enables the brain to communicate with the outside world without relying on peripheral neuromuscular tissues. For example, an intracortical visual prosthesis (ICVP) is implanted in the visual cortex of blind patients. ICVP is a type of implanted visual prosthesis (IVP). By using ICVP to bypass the damaged area and directly stimulating the V1 area through an electrode array, patients can generate visual perception.

[0036] Modern mainstream image sensors mainly adopt Active Pixel Sensor (APS) technology, and the typical representative is the Complementary Metal-Oxide-Semiconductor (CMOS) image sensor. The core working principle of such sensors is that each pixel unit integrates a photodiode and an active amplifier, and realizes the conversion of photoelectric signals through a global shutter or rolling shutter mechanism of progressive scanning. However, the APS architecture has several limitations: the dynamic range is usually low, and there is motion blur when shooting moving objects, etc.

[0037] With the development of vision sensors, traditional sensors based on active pixel arrays face problems such as low frame rate and low dynamic range. To address these issues, event-based vision sensors (EVS), inspired by bipolar cells in the human visual pathway, have been proposed, including dynamic vision sensors (DVS), dynamic and active-pixel vision sensors (DAVIS), etc. These sensors adopt a special difference pixel (DP) design, where each pixel has an independent and asynchronous low-latency response to changes in light intensity, featuring high temporal resolution, high dynamic range, and low information redundancy. Among them, the DAVIS series of event cameras use a hybrid pixel array of difference pixels (DP) and active pixels (AP), enabling spatially aligned multimodal image perception. However, some event cameras, such as the DVXplorer event camera (DVXplorer Event Camera, DVXplorer) and Prophesee EVK4, only use difference pixels (DP) and cannot obtain RGB images. However, the event stream accuracy of these sensors is relatively low, mostly ±1 bit (bit), and the event stream is asynchronously triggered, making it difficult for their signal characteristics to be directly compatible with mainstream frame-based processing algorithms. In addition, the triggering of the event stream depends on external motion, so there is a lack of stability. Figure 2 It is a schematic diagram of the imaging effect of the active pixel array in the prior art. Figure 3 It is a schematic diagram of the imaging effect of the difference pixel array in the prior art.

[0038] Inspired by the primitive splitting and combination mechanism in the human visual processing pathway, brain-inspired vision sensors based on primitives have been proposed. When perceiving the external environment, these sensors split the visual information of the external environment into visual primitives such as color, intensity, edges, changes, and depth, and then combine and further process the primitives, enabling superior performance of high precision, high resolution, high frame rate, high dynamic range, low bandwidth, and low power consumption. For example, the brain-inspired vision sensor based on primitives can be Tianmou Core. On the one hand, it realizes the temporal and spatial differences of light intensity based on difference pixels (DP) to sparsely perceive the edges and changes in the external environment. On the other hand, it can perceive the color and light intensity of the external environment based on active pixels (AP), with the advantages of high dynamic range (>130dB), high speed (up to 10,000fps), and low bandwidth. Through further processing, the multi-channel primitive information is complementarily combined.

[0039] Based on this, the present invention introduces a primitive-based brain-inspired vision sensor into the brain-inspired vision prosthesis system. By utilizing the primitive-based brain-inspired vision sensor, the extraction and combination of primitives from the external environment are realized to control the implanted electrode array implanted in different regions of the human visual processing pathway to generate electrical stimulation signals for stimulating corresponding regions. Since the primitive-based brain-inspired vision sensor has the performance of low bandwidth, high frame rate, and high dynamic range, the brain-inspired vision prosthesis system can efficiently perceive the external environment and perform zonal electrical stimulation based on primitive processing, thereby improving the perception efficiency of the brain-inspired vision prosthesis system and enhancing the user's visual experience. In addition, the brain-inspired vision sensor provided by the present invention can output multiple primitive information in multi-bit form and single-bit form. The multiple primitive information in single-bit form is more in line with the characteristics of human neural signals, so it can greatly reduce the computational complexity of the subsequent conversion process from image information to electrical stimulation signals in the vision prosthesis system. Moreover, the present invention proposes a two-way control framework. From the brain-inspired vision sensor to the human brain, based on the brain-inspired vision sensor perceiving the external environment, the predicted analog neural signals are used to realize electrical stimulation of multiple locations in the visual pathway through the implanted electrode array. From the human brain to the brain-inspired vision sensor, based on the implanted electrode array perceiving the brain electrical signals, the brain electrical signals are decoded to generate a regulation signal, and based on the regulation signal, the attribute parameters of the brain-inspired vision sensor are adjusted, so that the brain-inspired vision sensor re-collects the external environment based on the adjusted attribute parameters, and then based on the spatio-temporal difference image and color intensity image obtained from the re-collection, a new electrical stimulation signal is generated, making the new electrical stimulation signal more in line with the user and achieving the mutual coordination and adaptation between humans and machines.

[0040] Figure 4 is one of the structural schematic diagrams of the primitive-based brain-inspired vision prosthesis system provided by the embodiments of the present invention. As Figure 4 shown, the primitive-based brain-inspired vision prosthesis system includes a primitive-based brain-inspired vision sensor, an implanted electrode array, and a processor. The primitive-based brain-inspired vision sensor and the implanted electrode array are both connected to the processor; The brain-inspired vision sensor is configured to collect the external environment to obtain a spatio-temporal difference image and a color intensity image, extract primitives from the spatio-temporal difference image and the color intensity image to obtain multiple primitive information in multi-bit form, and encode each primitive information type through a pulse encoder using different encoding methods to obtain multiple primitive information in single-bit form, and send the multiple primitive information in multi-bit form or the multiple primitive information in single-bit form to the processor; The processor is configured to control an implantable electrode array implanted in different regions of the human visual processing pathway to generate an electrical stimulation signal based on each primitive information in multi-bit form or each primitive information in single-bit form, and the electrical stimulation signal is used to stimulate the corresponding region through the implantable electrode array implanted in the corresponding region; The implantable electrode array is configured to collect electroencephalogram signals of the corresponding region and send the electroencephalogram signals to the processor; The processor is further configured to generate a regulation signal based on the electroencephalogram signals through an electroencephalogram decoding feedback regulator, and adjust the attribute parameters of the brain-like vision sensor based on the regulation signal, so as to re-collect the external environment based on the adjusted brain-like vision sensor, and stimulate the corresponding region based on the re-collected image.

[0041] Among them, the brain-like vision sensor based on primitives can be, for example, Tianmou Core, or a pulse encoder can be added on the basis of Tianmou Core. The pulse encoder encodes each primitive information in multi-bit form using different encoding methods to obtain multiple primitive information in single-bit form; the implantable electrode array can be one or multiple. When there are multiple implantable electrode arrays, the multiple implantable electrode arrays can be implanted in different regions. For example, including two implantable electrode arrays, one implantable electrode array can be implanted in the retinal region and the other implantable electrode array can be implanted in the primary visual cortex V1 region. The present invention does not make any limitation in this regard.

[0042] Exemplarily, in practical applications, an electronic device configured with a brain-like vision prosthesis system based on primitives can be worn on the user. The external environment is collected through the brain-like vision sensor based on primitives to obtain a spatio-temporal differential image and a color intensity image. Primitive extraction is performed on the spatio-temporal differential image and the color intensity image to obtain multiple primitive information in multi-bit form, and the pulse encoder encodes each primitive information using different encoding methods to obtain multiple primitive information in single-bit form. The multiple primitive information in multi-bit form or the multiple primitive information in single-bit form is sent to the processor. The processor can predict the simulated neural signals in different regions of the human visual processing pathway based on each primitive information in multi-bit form and the combination of at least two types of information in each primitive information, and control the implantable electrode array implanted in the corresponding region to generate an electrical stimulation signal based on the simulated neural signals, and stimulate the corresponding region through the electrical stimulation signal, realizing zonal stimulation, and finally enabling the user to generate visual perception.

[0043] In addition, considering the individual differences of each user, the same set of attribute parameters of the brain-inspired vision sensor may not be suitable for all users. Therefore, a two-way human-machine interaction is considered. For each region, the electroencephalogram (EEG) signals of the high-level visual cortex are collected through the implantable electrode array in the corresponding region, and the collected EEG signals are sent to the processor. The processor decodes the EEG signals and analyzes the spatio-temporal patterns of neural activities (such as specific frequency band energy, neuron cluster firing rate, or decoded visual features, etc.) to generate a regulation signal. Here, the regulation signal includes a signal for modifying the attribute parameters of the brain-inspired vision sensor, and the modified attribute parameters can be the frame rate, exposure, gain, and pixel value threshold of the brain-inspired vision sensor, etc., so that the brain-inspired vision sensor re-collects the external environment based on the adjusted attribute parameters. Then, based on the spatio-temporal difference image and color intensity image obtained from the re-collection, a new electrical stimulation signal is generated, and the corresponding region is stimulated based on the new electrical stimulation signal, so that the new electrical stimulation signal is more in line with the user, achieving the mutual coordination and adaptation between humans and machines, and providing high-quality visual stimulation that conforms to the individual characteristics of the user.

[0044] It should be noted that when the processor controls the implantable electrode array to generate corresponding electrical stimulation signals based on each primitive information in the multi-bit form, it can support computer science-oriented applications. When the processor controls the implantable electrode array to generate corresponding electrical stimulation signals based on each primitive information in the single-bit form, it can support neuroscience-oriented applications.

[0045] It should be noted that the selection of the implantable electrode array includes but is not limited to UEA and WFMA. The deployment locations of the implantable electrode array include but are not limited to the retina region, the optic nerve region, and multiple different visual signal perception and processing regions in the initial visual cortex V1 region. The present invention does not make any limitations in this regard.

[0046] It should be noted that the acquisition of the external environment by the brain-inspired vision sensor based on primitives includes but is not limited to multiple visual primitives such as the color, contour, light intensity, change, and depth of the external environment. The present invention does not make any limitations in this regard.

[0047] It should be noted that the brain-inspired vision sensor based on primitives can be implemented through chip hybrid pixel array integration, or through spectroscope alignment of multi-modal discrete vision sensors, or through multi-sensor calibration, or through a binocular vision system; the present invention utilizes the advantages of high frame rate, high dynamic range, low bandwidth, and low power consumption of the brain-inspired vision sensor based on primitives that exceed traditional cameras, as well as the similarity between the signal characteristics under the brain-inspired vision primitive processing mechanism and the characteristics of human eye perception and processing, which helps to achieve high-quality perception and long battery life operation deployed at the processor end.

[0048] It should be noted that the processor in the present invention can be set separately or integrated into a brain-inspired vision sensor based on primitives. When selecting the processor, a neuromorphic chip can be considered. In particular, for the dual-channel heterogeneous data input, a neuromorphic chip that supports both artificial neural networks and spiking neural networks, such as Tianjic, can be used to achieve low-power and efficient multi-modal data processing at the computational level. The present invention does not make any limitations in this regard.

[0049] The brain-inspired visual prosthesis system based on primitives provided by the present invention includes a brain-inspired visual sensor based on primitives, an implantable electrode array, and a processor. The brain-inspired visual sensor is used to collect the external environment to obtain spatio-temporal differential images and color intensity images, extract primitives from the spatio-temporal differential images and color intensity images to obtain multiple primitive information in multi-bit form, and use a pulse encoder to encode each primitive information type in different encoding methods to obtain multiple primitive information in single-bit form. The processor is used to control the implantable electrode array implanted in different regions of the human visual processing pathway to generate electrical stimulation signals based on each primitive information in multi-bit form or each primitive information in single-bit form. The electrical stimulation signals are used to stimulate the corresponding regions through the implantable electrode array implanted in the corresponding regions. The processor is also used to generate a regulation signal through a brain electrical decoding feedback regulator based on the brain electrical signals of the corresponding regions collected by the implantable electrode array, and adjust the attribute parameters of the brain-inspired visual sensor based on the regulation signal, so as to re-collect the external environment based on the adjusted brain-inspired visual sensor, and stimulate the corresponding regions based on the re-collected images. It can be seen that the present invention realizes the extraction and combination of primitives of the external environment by using a brain-inspired visual sensor based on primitives, so as to control the implantable electrode array implanted in different regions of the human visual processing pathway to generate electrical stimulation signals for stimulating the corresponding regions. Since the brain-inspired visual sensor based on primitives has the functions of low power consumption, low bandwidth, high robustness, high frame rate, high dynamic range, and pulse coding output, the brain-inspired visual prosthesis system can efficiently perceive the external environment and perform zonal electrical stimulation based on primitive processing, thereby improving the perception and computational efficiency of the brain-inspired visual prosthesis system, especially suitable for coping with the challenge of difficult deployment of a large number of communication and computing resources on the human body, and enhancing the user's visual experience. In addition, the brain-inspired visual sensor provided by the present invention can output multiple primitive information in multi-bit form and single-bit form. The multiple primitive information in single-bit form is more in line with the characteristics of human neural signals, so it can greatly reduce the computational complexity of the subsequent conversion process from image information to electrical stimulation signals in the visual prosthesis system. Moreover, the present invention proposes a two-way control framework. From the brain-inspired visual sensor to the human brain, based on the brain-inspired visual sensor perceiving the external environment, the predicted analog neural signals are used to achieve electrical stimulation of multiple positions in the visual pathway through the implantable electrode array. From the human brain to the brain-inspired visual sensor, based on the implantable electrode array perceiving the brain electrical signals, the brain electrical signals are decoded to generate a regulation signal, and the attribute parameters of the brain-inspired visual sensor are adjusted based on the regulation signal, so that the brain-inspired visual sensor re-collects the external environment based on the adjusted attribute parameters, and then generates new electrical stimulation signals based on the re-collected spatio-temporal differential images and color intensity images, making the new electrical stimulation signals more in line with the user and achieving the mutual coordination and adaptation between humans and machines.

[0050] In one embodiment, the multiple primitive information in the multi-bit form includes contour information, change information, color information, and light intensity information.

[0051] The brain-like vision sensor is specifically configured to perform primitive extraction on the spatio-temporal difference image to obtain the contour information and change information of the external environment, perform primitive extraction on the color intensity image to obtain the color information and light intensity information of the external environment, and use the pulse encoder to perform binary coding on the contour information to obtain the first primitive information in a single-bit form, perform time coding on the change information to obtain the second primitive information in a single-bit form, and perform rate coding on the color information and the light intensity information to obtain the third primitive information in a single-bit form.

[0052] Among them, the contour information is used to characterize the light intensity difference at different spatial positions at the same moment, and the change information is used to characterize the difference in light intensity at the same spatial position over time.

[0053] Exemplarily, the brain-like vision sensor can perform primitive extraction on the spatio-temporal difference image based on artificial neural network algorithms, spiking neural network algorithms, or hybrid neural network algorithms, etc., to obtain the contour information and change information of the external environment, and perform primitive extraction on the color intensity image to obtain the color information and light intensity information of the external environment. Furthermore, the pulse encoder is used to perform binary coding on the contour information to obtain the first primitive information in a single-bit form, perform time coding on the change information to obtain the second primitive information in a single-bit form, and perform rate coding on the color information and the light intensity information to obtain the third primitive information in a single-bit form.

[0054] In this embodiment, the pulse encoder included in the brain-like vision sensor is used to perform binary coding on the contour information to obtain the first primitive information in a single-bit form, perform time coding on the change information to obtain the second primitive information in a single-bit form, and perform rate coding on the color information and the light intensity information to obtain the third primitive information in a single-bit form, realizing the generation of different types of primitive information in a single-bit form. The multiple primitive information in a single-bit form is more in line with the characteristics of human nerve signals, so it can greatly reduce the computational complexity of the subsequent conversion process from image information to electrical stimulation signals in the visual prosthesis system.

[0055] In one embodiment, the processor is specifically configured to control the first implantable electrode array implanted in the retinal area to generate a first electrical stimulation signal encoded with the first primitive information and the second primitive information based on the first primitive information and the second primitive information, and the first electrical stimulation signal is used to stimulate the rod cells in the retinal area.

[0056] The processor is specifically configured to control the second implanted electrode array implanted in the retinal region to generate a second electrical stimulation signal encoded with the third primitive information based on the third primitive information, and the second electrical stimulation signal is used to stimulate cone cells in the retinal region.

[0057] Exemplarily, Figure 5 FIG. 2 is a second schematic structural diagram of a brain-inspired visual prosthesis system based on primitives according to an embodiment of the present invention. As Figure 5 shown, Figure 5 FIG. 2 shows a system solution for using a brain-inspired visual sensor and implanting an implanted electrode array in a retinal region to stimulate cone cells and rod cells. At the brain-inspired visual sensor end, the brain-inspired visual sensor provides dual-channel data collected from the external environment, including high-frame-rate, sparse spatio-temporal differential images and low-frame-rate, dense color intensity images. The spatio-temporal differential images focus on perceiving changes and contours in the external environment, and the color intensity images focus on finely perceiving the color and light intensity of the external environment. This dual-channel data is consistent with the functional characteristics of rod cells and cone cells in the retina. Therefore, in the present invention, spatio-temporal differential images and color intensity images of the external environment are obtained through the brain-inspired visual sensor, a first analog neural signal is predicted based on first primitive information and second primitive information determined from the spatio-temporal differential images, and a first stimulation parameter executable by the first implanted electrode array implanted in the retinal region is determined based on the first analog neural signal. Then, the first stimulation parameter is sent to the first implanted electrode array, so that the first implanted electrode array generates a first electrical stimulation signal encoded with the first primitive information and the second primitive information based on the first stimulation parameter, and the rod cells in the retinal region are stimulated through the first electrical stimulation signal. Figure 5 FIG. 2 shows that the human eye includes a retina, a fovea, a blind spot, etc. The cells shown in the green box are indicated by green arrows, that is, they include cone cells and rod cells.

[0058] A second analog neural signal is predicted based on third primitive information determined from the color intensity images, and a second stimulation parameter executable by the second implanted electrode array implanted in the retinal region is determined based on the second analog neural signal. Then, the second stimulation parameter is sent to the second implanted electrode array, so that the second implanted electrode array generates a second electrical stimulation signal encoded with the third primitive information based on the second stimulation parameter, and the cone cells in the retinal region are stimulated through the second electrical stimulation signal, thereby achieving targeted and efficient electrical stimulation.

[0059] It should be noted that the first implantable electrode array and the second implantable electrode array can be the same or different. When they are the same, some electrode points in the first implantable electrode array can be used to stimulate rod cells, and another part of the electrode points in the first implantable electrode array can be used to stimulate cone cells. When the first implantable electrode array and the second implantable electrode array are different, the rod cells and cone cells can be stimulated separately through the first implantable electrode array and the second implantable electrode array in parallel.

[0060] In this embodiment, based on the first primitive information and the second primitive information, the first implantable electrode array implanted in the retinal area can be controlled to generate a first electrical stimulation signal encoded with the first primitive information and the second primitive information to stimulate the rod cells in the retinal area. Also, based on the third primitive information, the second implantable electrode array implanted in the retinal area can be controlled to generate a second electrical stimulation signal encoded with the third primitive information to stimulate the cone cells in the retinal area, thereby realizing targeted electrical stimulation of the retinal area.

[0061] In one embodiment, the processor is specifically configured to control a third implantable electrode array implanted in the primary visual cortex V1 area to generate a third electrical stimulation signal encoded with the first primitive information, the second primitive information, and the third primitive information based on the first primitive information, the second primitive information, and the third primitive information, and the third electrical stimulation signal is used to stimulate the primary visual cortex V1 area.

[0062] Exemplarily, Figure 6 is the third schematic diagram of the structure of the brain-like visual prosthesis system based on primitives provided by the embodiments of the present invention. As Figure 6 shown, Figure 6 is a system solution for stimulating the primary visual cortex V1 area through an implantable electrode array by using a brain-like visual sensor based on primitives. This solution controls a third implantable electrode array implanted in the primary visual cortex V1 area to generate a third electrical stimulation signal encoded with the first primitive information, the second primitive information, and the third primitive information based on the first primitive information, the second primitive information, and the third primitive information, providing basic visual primitives such as edges, changes, colors, and light intensities for the superficial visual processing brain area, and providing an information source for further processing and processing primitive information in subsequent brain areas. Compared with electrical stimulation at the retinal level, it can better cope with blindness caused by damaged optic nerves.

[0063] It should be noted that the present invention can simultaneously generate a first electrical stimulation signal, a second electrical stimulation signal, and a third electrical stimulation signal. The rod cells in the retinal region are stimulated by the first electrical stimulation signal, the cone cells in the retinal region are stimulated by the second electrical stimulation signal, and the third electrical stimulation signal is used to stimulate the primary visual cortex V1 region. The present invention does not make any limitations in this regard.

[0064] In one embodiment, the processor is specifically configured to extract features from multiple primitive information in multi-bit form to obtain semantic information.

[0065] The processor is specifically configured to control a fourth implantable electrode array implanted in the primary visual cortex V1 region to generate a fourth electrical stimulation signal encoded with the first primitive information and the third primitive information based on the first primitive information and the third primitive information, and the fourth electrical stimulation signal is used to stimulate the primary visual cortex V1 region.

[0066] The processor is specifically configured to control a fifth implantable electrode array implanted in the visual cortex V2 region to generate a fifth electrical stimulation signal encoded with the second primitive information based on the second primitive information, and the fifth electrical stimulation signal is used to stimulate the visual cortex V2 region.

[0067] The processor is specifically configured to control a sixth implantable electrode array implanted in the visual cortex V4 region to generate a sixth electrical stimulation signal encoded with the semantic information based on the semantic information, and the sixth electrical stimulation signal is used to stimulate the visual cortex V4 region.

[0068] Exemplarily, Figure 7 is the fourth structural schematic diagram of the primitive-based brain-like visual prosthesis system provided by the embodiment of the present invention. As Figure 7 shown, Figure 7A system solution for two-way brain-machine interaction that implants an implantable electrode array into different visual cortex regions by using a brain-like visual sensor based on the primitive. Due to the limited resolution of the implantable electrode array, simply downsampling is difficult to fully retain the features of the original spatio-temporal differential image and color intensity image. Therefore, the present invention adopts a strategy of compensating for the resolution with information dimension, and performs multi-level image feature extraction on the spatio-temporal differential image and color intensity image from low to high. Specifically: overall feature extraction is performed on multiple primitive information in multi-bit form obtained by primitive extraction of the spatio-temporal differential image and color intensity image to obtain semantic information, and the semantic information is used as the high-level feature, and the first primitive information and the third primitive information representing contour information, color information, and light intensity information are used as the low-level features, and the second primitive information representing change information is used as the middle-level feature; furthermore, based on the first primitive information and the third primitive information representing contour information, color information, and light intensity information, these low-level features, the fourth implantable electrode array implanted in the primary visual cortex V1 region is controlled to generate a fourth electrical stimulation signal encoded with the first primitive information and the third primitive information to stimulate the primary visual cortex V1 region; based on the second primitive information representing change information, this middle-level feature, the fifth implantable electrode array implanted in the visual cortex V2 region is controlled to generate a fifth electrical stimulation signal encoded with the second primitive information to stimulate the visual cortex V2 region; at the same time, based on the semantic information, this high-level feature, the sixth implantable electrode array implanted in the visual cortex V4 region is controlled to generate a sixth electrical stimulation signal encoded with the semantic information to stimulate the visual cortex V4 region.

[0069] It should be noted that, as Figure 7 shown, considering the individual differences of each user, the same set of attribute parameters of the brain-like visual sensor may not be suitable for all users. Therefore, considering two-way human-machine interaction, the electroencephalogram signals of the high-level visual cortex are collected through the implantable electrode array, the collected electroencephalogram signals are sent to the processor, the electroencephalogram signals are decoded by the processor, and the attribute parameters of the brain-like visual sensor are regulated based on the decoding results to achieve mutual adaptation and regulation under human-machine cooperation.

[0070] It should be noted that in the case of limited resolution of the implantable electrode array, inputting the processed high-level features (such as the semantics of the external environment, etc.) can more effectively trigger clear perception than directly inputting the downsampled low-level image. Figure 8 shows a comparison of the schematic diagrams of the original image and the electrical stimulation signal in the prior art, as Figure 8 shown, the left figure is the original image collected by the camera, and the right figure is the schematic diagram of the electrical stimulation signal generated after downsampling the original image. It can be seen that due to the limited resolution of the implantable electrode array, the electrical stimulation signal obtained by directly downsampling the original image loses important information such as the edge contour.

[0071] In this embodiment, the information dimension is expanded under the limitation of the low-resolution implantable electrode array, and a strategy of compensating the resolution with the information dimension is adopted. Multi-level image feature extraction from low to high is performed on the spatiotemporal difference image and the color intensity image, and different areas are stimulated in a targeted manner based on information at different levels, thereby achieving higher quality visual information stimulation that is more in line with low resolution, and improving the user's visual perception experience in actual scenes.

[0072] In one embodiment, the processor is specifically used to determine motion attention information based on the second primitive information through an EEG signal predictor, generate contrast enhancement information based on the third primitive information, and predict a simulated neural signal based on the motion attention information, the contrast enhancement information and the first primitive information.

[0073] The processor is specifically used to determine stimulation parameters executable by the implantable electrode array based on the simulated neural signal, and control the implantable electrode array to generate the electrical stimulation signal based on the stimulation parameters.

[0074] For example, Figure 9 FIG. 5 is a schematic diagram of the structure of a primitive-based brain-like visual prosthesis system provided in an embodiment of the present invention. Figure 9 As shown, Figure 9A strategic framework for using a brain-inspired visual sensor based on the primitive language to achieve the extraction of various basic visual features, thereby realizing the electrical stimulation control of the primary visual processing area. The primary visual information processing of the human brain mainly covers basic primitives such as color, contour, change, and light intensity. In this invention, the contour information and change information of the external environment are extracted from the spatio-temporal differential image collected by the brain-inspired visual sensor, and the color information and light intensity information of the external environment are extracted from the color intensity image collected by the brain-inspired visual sensor. The second primitive information representing the change information is input into the electroencephalogram signal predictor, and the change signal in adjacent frames or spatial regions (such as the displacement of a moving object, etc.) is detected through the electroencephalogram signal predictor, and finally the motion attention information is obtained to highlight the motion area in the external environment; the third primitive information representing the color information and light intensity information is input into the electroencephalogram signal predictor, and the electroencephalogram signal predictor distinguishes different color regions based on the color information, enhances the color contrast, and optimizes the overall contrast by adjusting the local brightness difference, and finally generates the contrast enhancement information; further, a complete shape representation is synthesized based on the motion attention information, the contrast enhancement information, and the first primitive information representing the contour information. These features (the motion attention information, the contrast enhancement information, and the first primitive information representing the contour information) are converted into the spatio-temporal pattern of neural signals, that is, the motion attention information is encoded as high-frequency pulses, such as in Visual Area 5 (V5), and the contrast enhancement information and the first primitive information representing the contour information are mapped to the tuning responses of cortical neurons (such as the orientation and shape selectivity of the V1 / V2 area), thereby simulating the hierarchical neural activity of the biological visual system, that is, obtaining the simulated neural signals, and then determining the stimulation parameters that the implantable electrode array can execute based on the simulated neural signals. The stimulation parameters here include pulse amplitude, pulse frequency, and electrode points to be activated, etc., and the stimulation parameters are sent to the implantable electrode array, so that the implantable electrode array generates corresponding electrical stimulation signals based on the received stimulation parameters.

[0075] In one embodiment, the processor is specifically configured to extract features from the electroencephalogram signal through an electroencephalogram decoding feedback regulator to obtain an intensity matching feature and a frequency synchronization feature.

[0076] The processor is specifically configured to generate the regulation signal when at least one of the following preset conditions is not satisfied. The preset conditions include: the intensity matching feature represents that the pulse intensity of the electrical stimulation signal reaches the neural activation threshold, and the frequency synchronization feature represents that the pulse frequency of the electrical stimulation signal resonates with the neural oscillation.

[0077] Exemplarily, the processor extracts features from the collected EEG signals through an EEG decoding feedback regulator, analyzes the intensity matching features and frequency synchronization features, determines whether the pulse intensity of the generated electrical stimulation signal reaches the nerve activation threshold based on the intensity matching features, and determines whether the pulse frequency of the generated electrical stimulation signal resonates with the nerve oscillation based on the frequency synchronization features. When the pulse intensity of the generated electrical stimulation signal does not reach the nerve activation threshold, and / or the pulse frequency of the generated electrical stimulation signal does not resonate with the nerve oscillation, it indicates that the generated electrical stimulation signal does not fully match the user. Then, a regulation signal is generated to adjust the attribute parameters of the brain-like vision sensor through the regulation signal, so as to re-collect the external environment based on the adjusted brain-like vision sensor. Furthermore, based on the spatio-temporal difference image and color intensity image obtained from the re-collection, a new electrical stimulation signal is generated, so that the new electrical stimulation signal matches the user, achieving the mutual coordination and adaptation between humans and machines. When the pulse intensity of the generated electrical stimulation signal reaches the nerve activation threshold and the pulse frequency of the generated electrical stimulation signal resonates with the nerve oscillation, it indicates that the generated electrical stimulation signal matches the user, and there is no need to generate a regulation signal.

[0078] In this embodiment, it is determined whether the generated electrical stimulation signal matches the user based on the intensity matching features and frequency synchronization features. When it is determined that the generated electrical stimulation signal does not fully match the user, the attribute parameters of the brain-like vision sensor are dynamically adjusted based on the regulation signal to ensure that the newly generated electrical stimulation signal matches the user, thereby achieving precise closed-loop regulation of nerve activities.

[0079] In one embodiment, the brain-like vision sensor is used to collect the depth of the external environment to obtain depth information and send the depth information to the processor.

[0080] The processor is further configured to generate a reconstructed image based on the first primitive information, the second primitive information, and the third primitive information through an image reconstructor, segment the reconstructed image through an image segmenter to obtain a target segmentation mask, and generate a depth image based on the depth information and the reconstructed image through a depth estimator.

[0081] The processor is further configured to predict the simulated nerve signal through the EEG signal predictor based on the first primitive information, the second primitive information, the third primitive information, the depth image, the reconstructed image, and the target segmentation mask.

[0082] Exemplarily, Figure 10 is the sixth structural schematic diagram of the primitive-based brain-like vision prosthesis system provided by the embodiments of the present invention. As Figure 10 shown, Figure 10To implement the extraction of multi-level visual features from low to high using various visual primitives of the primitive-based brain-inspired vision sensor, thereby realizing the strategy framework of electrically stimulating and controlling different processing regions from shallow to deep in the human visual information processing pathway. Considering that the processing of brain visual signals is a process from shallow to deep, and limited by the low resolution of the implanted electrode array, it is difficult for shallow image features to show intuitive visual effects at low resolutions. The present invention proposes an algorithmic process for image processing from low-level features to high-level features, and predicts and simulates neural signals through an electroencephalogram signal predictor for different levels of features, which are conducted to regions from the retina to the V4 region of the visual cortex, and even higher-level visual information processing regions, to achieve multi-site electrical stimulation, so as to achieve a better visual perception effect with the assistance of multi-level features at low resolutions. Among them, the advantages of the primitive-based brain-inspired vision sensor in terms of depth, color, light intensity, frame rate, dynamic range, etc. can input the first primitive information representing contour information, the second primitive information representing change information, and the third primitive information representing color information and light intensity information into an image reconstructor, and the image reconstructor outputs a high-speed and high-dynamic range reconstructed image with a maximum of 10,000 fps. Then, using an image segmenter based on a color histogram, etc., to segment the reconstructed image, a target segmentation mask for the external environment can be obtained; when the brain-inspired vision sensor is implemented based on a binocular vision system, the depth of the external environment can be collected through the brain-inspired vision sensor to obtain depth information, and both the depth information and the reconstructed image are input into a depth estimator, and the depth image output by the depth estimator is obtained. The primitive combination is realized through the reconstructed image, and the semantic information is represented through the target segmentation mask. Finally, based on the first primitive information representing contour information, the second primitive information representing change information, the third primitive information representing color information and light intensity information, the depth image, the reconstructed image, and the target segmentation mask, multi-modal feature fusion and hierarchical coding are used to predict and simulate neural signals, and then generate multi-region electrical stimulation signals from shallow to deep to electrically stimulate different regions, specifically including electrical stimulation of the retina region, the optic nerve region, and the visual cortex region, etc.

[0083] In this embodiment, both basic features such as color information, contour information, change information, and light intensity information from spatio-temporal difference images and color intensity images at a low level can be obtained, and a reconstructed image with a high frame rate and high dynamic range after fusion can also be obtained, and cognitive-level features of the target segmentation mask and spatial perception features including depth can be obtained. The partitioned stimulation of multi-level features helps to achieve high-quality visual perception at low resolutions.

[0084] In one embodiment, the processor is specifically configured to determine the identifier of the implantable electrode array and the target primitive combination information type corresponding to the target area based on the correspondence between the area, the electrode array identifier, and the primitive combination information type, and determine the information corresponding to the target primitive combination information type from the first primitive information, the second primitive information, and the third primitive information to obtain the target primitive combination information.

[0085] The processor is specifically configured to control the implantable electrode array to generate the electrical stimulation signal based on the target primitive combination information.

[0086] Exemplarily, the correspondence between the area, the electrode array identifier, and the primitive combination information type may be pre-stored. When the identifier of the implantable electrode array implanted in the target area is obtained, the target primitive combination information type corresponding to the target area and the identifier of the implantable electrode array is searched for in the correspondence. Then, the information matching the target primitive combination information type is screened out from the first primitive information representing contour information, the second primitive information representing change information, and the third primitive information representing color information and light intensity information, and these matching information is used as the target primitive combination information. Then, based on the target primitive combination information, the implantable electrode array implanted in the target area is controlled to generate an electrical stimulation signal. For example, if the target area is the retinal area and the identifier of the corresponding implantable electrode array is 1, and the correspondence stores the correspondence between electrode array identifier 1, retinal area, light intensity, and color type, then the target primitive combination information is determined to be the third primitive information representing the combination of light intensity information and color information.

[0087] It should be noted that the present invention may also adjust the primitive combination information type in the pre-set correspondence based on the functional partition of the brain and the feedback electroencephalogram signal, and may also adjust the stimulation parameters executable by the implantable electrode array based on the feedback electroencephalogram signal to control the implantable electrode array to generate electrical stimulation signals with different strengths to adapt to the individual. The present invention does not limit this.

[0088] In summary, the present invention designs a software and hardware circuit for two-way human-machine interaction including "multi-primitive and primitive combination signal input based on the primitive-based brain-like vision sensor - parallel implanted electrode array partition stimulation - electroencephalogram signal perception feedback - adaptive adjustment of the brain-like vision sensor". By using the primitive-based brain-like vision sensor, it realizes the primitive splitting perception and combined processing of visual information of the external environment, and then constructs a robust perception performance for high-dynamic-range scenes and high-speed moving targets, as well as a low-redundancy and efficient extraction ability for information such as color, edges, and motion, realizing sensor signal input closer to the human visual perception and processing mechanism, while taking into account the deployment requirements of low-power and low-bandwidth processors and the low-latency two-way human-machine adaptive regulation. A partition stimulation strategy is proposed: Utilizing the complementarity and difference of multiple visual primitives, considering the cell diversity functional partitions of the human retina and the visual cortex of the brain, and making full use of the characteristics of visual primitives and their combinations to stimulate different cells or different brain regions of the retina area in a targeted and parallel manner. For example, cone cells are more sensitive to light intensity and color, and the implanted electrode array will apply electrical stimulation signals encoding light intensity information and color information to cone cells in a targeted manner. The primitive-based brain-like visual prosthesis system can achieve a visual perception effect that exceeds the visual prosthesis paradigm based on traditional cameras through primitive-based perception, processing, and targeted partition stimulation.

[0089] Figure 11 It is a schematic flowchart of the primitive-based partition electrical stimulation method provided by an embodiment of the present invention. As Figure 11 shown, it is applied to the above-mentioned primitive-based brain-like visual prosthesis system. The primitive-based partition electrical stimulation method includes the following steps: Step 1101: Receive multiple primitive information in multi-bit form or multiple primitive information in single-bit form sent by the primitive-based brain-like vision sensor. The multiple primitive information in multi-bit form is obtained by the brain-like vision sensor performing primitive extraction on the collected spatio-temporal differential image and color intensity image, and the multiple primitive information in single-bit form is obtained by the brain-like vision sensor encoding each primitive information type using different encoding methods through a pulse encoder.

[0090] Step 1102: Based on each primitive information in multi-bit form or each primitive information in single-bit form, control the implanted electrode array in different regions of the human visual processing pathway to generate an electrical stimulation signal, and the electrical stimulation signal is used to stimulate the corresponding region through the implanted electrode array implanted in the corresponding region.

[0091] Step 1103: Receive the electroencephalogram signal of the corresponding region collected by the implanted electrode array.

[0092] Step 1104: Based on the EEG signal, generate a regulation signal through an EEG decoding feedback regulator, and adjust the attribute parameters of the brain-inspired vision sensor based on the regulation signal, so as to re-collect the external environment based on the adjusted brain-inspired vision sensor, and stimulate the corresponding area based on the re-collected image.

[0093] The method for partitioned electrical stimulation based on primitives provided by the present invention receives multiple primitive information in the form of multiple bits or multiple primitive information in the form of single bits sent by a brain-inspired vision sensor based on primitives, and controls implantable electrode arrays implanted in different areas of the human visual processing pathway to generate electrical stimulation signals based on each primitive information in the form of multiple bits or each primitive information in the form of single bits. The electrical stimulation signals are used to stimulate the corresponding area through the implantable electrode array implanted in the corresponding area; and generate a regulation signal through an EEG decoding feedback regulator based on the EEG signal of the corresponding area collected by the implantable electrode array, and adjust the attribute parameters of the brain-inspired vision sensor based on the regulation signal, so as to re-collect the external environment based on the adjusted brain-inspired vision sensor, and stimulate the corresponding area based on the re-collected image. It can be seen that the present invention realizes the extraction and combination of primitives of the external environment by using a brain-inspired vision sensor based on primitives, so as to control implantable electrode arrays implanted in different areas of the human visual processing pathway to generate electrical stimulation signals for stimulating the corresponding areas. Since the brain-inspired vision sensor based on primitives has the functions of low power consumption, low bandwidth, high robustness, high frame rate, high dynamic range, and pulse-coded output, the brain-inspired visual prosthesis system can efficiently perceive the external environment and perform partitioned electrical stimulation based on primitive processing, thereby improving the perception and computational efficiency of the brain-inspired visual prosthesis system, especially suitable for coping with the challenge of difficult deployment of a large number of communication and computing resources on the human body and enhancing the user's visual experience; in addition, the brain-inspired vision sensor provided by the present invention can output multiple primitive information in the form of multiple bits and multiple primitive information in the form of single bits, and the multiple primitive information in the form of single bits is more in line with the characteristics of human nerve signals, so it can greatly reduce the computational complexity of the subsequent conversion process from image information to electrical stimulation signals in the visual prosthesis system; moreover, the present invention proposes a framework for two-way control. From the brain-inspired vision sensor to the human brain, based on the brain-inspired vision sensor perceiving the external environment, the predicted analog nerve signals are used to realize electrical stimulation of multiple positions in the visual pathway through the implantable electrode array; from the human brain to the brain-inspired vision sensor, based on the implantable electrode array perceiving the EEG signal, decoding the EEG signal to generate a regulation signal, and adjusting the attribute parameters of the brain-inspired vision sensor based on the regulation signal, so that the brain-inspired vision sensor re-collects the external environment based on the adjusted attribute parameters, and then generates a new electrical stimulation signal based on the re-collected spatio-temporal differential image and color intensity image, making the new electrical stimulation signal more in line with the user and achieving the mutual coordination and adaptation between humans and machines.

[0094] In one embodiment, the multiple primitive information in the multi-bit form includes contour information, variation information, color information, and light intensity information.

[0095] In one embodiment, step 1102 controls an implantable electrode array implanted in different regions of the human visual processing pathway to generate an electrical stimulation signal based on each piece of primitive information in the multi-bit form or each piece of primitive information in the single-bit form. The electrical stimulation signal is used to stimulate the corresponding region through the implantable electrode array implanted in the corresponding region. Specifically, it can be implemented in the following manner: Based on the first primitive information and the second primitive information, control the first implantable electrode array implanted in the retinal region to generate a first electrical stimulation signal encoding the first primitive information and the second primitive information. The first electrical stimulation signal is used to stimulate the rod cells in the retinal region; and based on the third primitive information, control the second implantable electrode array implanted in the retinal region to generate a second electrical stimulation signal encoding the third primitive information. The second electrical stimulation signal is used to stimulate the cone cells in the retinal region.

[0096] Among them, the first primitive information is obtained by the brain-like visual sensor using binary encoding for the contour information through a pulse encoder, the second primitive information is obtained by the brain-like visual sensor using time encoding for the variation information through a pulse encoder, and the third primitive information is obtained by the brain-like visual sensor using rate encoding for the color information and the light intensity information through a pulse encoder.

[0097] In one embodiment, step 1102 controls an implantable electrode array implanted in different regions of the human visual processing pathway to generate an electrical stimulation signal based on each piece of primitive information in the multi-bit form or each piece of primitive information in the single-bit form. The electrical stimulation signal is used to stimulate the corresponding region through the implantable electrode array implanted in the corresponding region. Specifically, it can be implemented in the following manner: Based on the first primitive information, the second primitive information, and the third primitive information, control the third implantable electrode array implanted in the primary visual cortex V1 region to generate a third electrical stimulation signal encoding the first primitive information, the second primitive information, and the third primitive information. The third electrical stimulation signal is used to stimulate the primary visual cortex V1 region.

[0098] In one embodiment, step 1102 controls an implantable electrode array implanted in different regions of the human visual processing pathway to generate an electrical stimulation signal based on each piece of primitive information in the multi-bit form or each piece of primitive information in the single-bit form. The electrical stimulation signal is used to stimulate the corresponding region through the implantable electrode array implanted in the corresponding region. Specifically, it can be implemented in the following manner: Feature extraction is performed on multiple primitive information in multi-bit form to obtain semantic information; based on the first primitive information and the third primitive information, control the fourth implanted electrode array implanted in the primary visual cortex V1 region to generate a fourth electrical stimulation signal encoded with the first primitive information and the third primitive information, and the fourth electrical stimulation signal is used to stimulate the primary visual cortex V1 region; based on the second primitive information, control the fifth implanted electrode array implanted in the visual cortex V2 region to generate a fifth electrical stimulation signal encoded with the second primitive information, and the fifth electrical stimulation signal is used to stimulate the visual cortex V2 region; based on the semantic information, control the sixth implanted electrode array implanted in the visual cortex V4 region to generate a sixth electrical stimulation signal encoded with the semantic information, and the sixth electrical stimulation signal is used to stimulate the visual cortex V4 region.

[0099] In one embodiment, the above-mentioned control of the implanted electrode array implanted in different regions of the human visual processing pathway to generate electrical stimulation signals based on each primitive information in single-bit form can be specifically implemented in the following manner: The electroencephalogram signal predictor determines the motion attention information based on the second primitive information, generates the contrast enhancement information based on the third primitive information, and predicts the simulated neural signal based on the motion attention information, the contrast enhancement information, and the first primitive information; determines the stimulation parameters that the implanted electrode array can execute based on the simulated neural signal, and controls the implanted electrode array to generate the electrical stimulation signal based on the stimulation parameters.

[0100] In one embodiment, the primitive-based partitioned electrical stimulation method further includes the following steps: Receive the depth information of the external environment collected by the brain-like visual sensor, generate a reconstructed image based on the first primitive information, the second primitive information, and the third primitive information through an image reconstructor, segment the reconstructed image through an image segmenter to obtain a target segmentation mask, and generate a depth image based on the depth information and the reconstructed image through a depth estimator; finally, predict the simulated neural signal based on the first primitive information, the second primitive information, the third primitive information, the depth image, the reconstructed image, and the target segmentation mask through the electroencephalogram signal predictor.

[0101] In one embodiment, the electroencephalogram decoding feedback regulator generates a regulation signal based on the electroencephalogram signal, and can be specifically implemented in the following manner: Feature extraction is performed on the EEG signals through an EEG decoding feedback regulator to obtain an intensity matching feature and a frequency synchronization feature; when at least one of the following preset conditions is not satisfied, the regulation signal is generated, and the preset conditions include: the intensity matching feature indicates that the pulse intensity of the electrical stimulation signal reaches the nerve activation threshold, and the frequency synchronization feature indicates that the pulse frequency of the electrical stimulation signal resonates with the nerve oscillation.

[0102] In one embodiment, the method for partitioned electrical stimulation based on primitives further includes the following steps: Based on the correspondence relationship between the region, the electrode array identifier, and the type of primitive combination information, determine the identifier of the implantable electrode array and the type of target primitive combination information corresponding to the target region, and determine the information corresponding to the type of target primitive combination information from the first primitive information, the second primitive information, and the third primitive information to obtain the target primitive combination information; and control the implantable electrode array to generate the electrical stimulation signal based on the target primitive combination information.

[0103] It should be noted that for the specific implementation process of the method for partitioned electrical stimulation based on primitives, reference can be made to the specific description in the above-mentioned primitive-based brain-like visual prosthesis system, and the present invention will not elaborate here.

[0104] Next, a primitive-based partitioned electrical stimulation device provided by the present invention will be described. The primitive-based partitioned electrical stimulation device described below can be mutually corresponding and referred to the primitive-based partitioned electrical stimulation method described above.

[0105] The primitive-based partitioned electrical stimulation device includes a first receiving unit, a control unit, a second receiving unit, and an adjustment unit, where: The first receiving unit is configured to receive multiple primitive information in multi-bit form or multiple primitive information in single-bit form sent by a primitive-based brain-like visual sensor. The multiple primitive information in multi-bit form is obtained by the brain-like visual sensor performing primitive extraction on the collected spatio-temporal differential image and color intensity image. The multiple primitive information in single-bit form is obtained by the brain-like visual sensor encoding each type of primitive information using different encoding methods through a pulse encoder; The control unit is configured to control implantable electrode arrays implanted in different regions of the human visual processing pathway to generate electrical stimulation signals based on each primitive information in multi-bit form or each primitive information in single-bit form. The electrical stimulation signals are used to stimulate the corresponding regions through the implantable electrode arrays implanted in the corresponding regions; The second receiving unit is configured to receive the EEG signals of the corresponding regions collected by the implantable electrode arrays; An adjustment unit is configured to generate a regulation signal based on the EEG signal through an EEG decoding feedback regulator, and adjust the attribute parameters of the brain-like vision sensor based on the regulation signal, so as to re-collect the external environment based on the adjusted brain-like vision sensor, and stimulate the corresponding area based on the re-collected image.

[0106] Figure 12 is a schematic physical structure diagram of an electronic device provided by an embodiment of the present invention. As Figure 12 shown, the electronic device may include: a processor 1210, a communication interface 1220, a memory 1230, and a communication bus 1240. Among them, the processor 1210, the communication interface 1220, and the memory 1230 communicate with each other through the communication bus 1240. The processor 1210 may call logic instructions in the memory 1230 to execute a primitive-based partitioned electrical stimulation method, which includes: receiving multiple primitive information in multi-bit form or multiple primitive information in single-bit form sent by a primitive-based brain-like vision sensor, where the multiple primitive information in multi-bit form is obtained by the brain-like vision sensor extracting primitives from the collected spatio-temporal differential image and color intensity image, and the multiple primitive information in single-bit form is obtained by the brain-like vision sensor encoding each type of primitive information through a pulse encoder using different encoding methods; Based on each piece of primitive information in multi-bit form or each piece of primitive information in single-bit form, controlling an implantable electrode array implanted in different regions of the human visual processing pathway to generate an electrical stimulation signal, where the electrical stimulation signal is used to stimulate the corresponding region through the implantable electrode array implanted in the corresponding region; Receiving the EEG signal of the corresponding region collected by the implantable electrode array; Generating a regulation signal based on the EEG signal through an EEG decoding feedback regulator, and adjusting the attribute parameters of the brain-like vision sensor based on the regulation signal, so as to re-collect the external environment based on the adjusted brain-like vision sensor, and stimulate the corresponding area based on the re-collected image.

[0107] In addition, when the logical instructions in the above-mentioned memory 1230 are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes: various media such as USB flash drives, mobile hard disks, read-only memories (ROM, Read-Only Memory), random access memories (RAM, Random Access Memory), magnetic disks, or optical discs that can store program codes.

[0108] On the other hand, the present invention also provides a computer program product. The computer program product includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the primitives-based partitioned electrical stimulation method provided by the above-mentioned various methods. The method includes: receiving multiple primitives information in multi-bit form or multiple primitives information in single-bit form sent by a primitives-based brain-like vision sensor. The multiple primitives information in multi-bit form is obtained by the brain-like vision sensor through primitive extraction of the collected spatio-temporal difference image and color intensity image. The multiple primitives information in single-bit form is obtained by the brain-like vision sensor through a pulse encoder using different coding methods for each type of primitives information. Based on each primitives information in multi-bit form or each primitives information in single-bit form, controlling an implantable electrode array implanted in different regions of the human visual processing pathway to generate an electrical stimulation signal, where the electrical stimulation signal is used to stimulate the corresponding region through the implantable electrode array implanted in the corresponding region. Receiving the electroencephalogram signals of the corresponding region collected by the implantable electrode array. Through an electroencephalogram decoding feedback regulator, generating a regulation signal based on the electroencephalogram signals, and adjusting the attribute parameters of the brain-like vision sensor based on the regulation signal, so as to re-collect the external environment based on the adjusted brain-like vision sensor, and stimulating the corresponding region based on the re-collected image.

[0109] In another aspect, the present invention also provides a non-transitory computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, it implements a primitive-based partitioned electrical stimulation method provided by the above-mentioned various methods. The method includes: receiving a plurality of primitive information in multi-bit form or a plurality of primitive information in single-bit form sent by a primitive-based brain-inspired vision sensor, where the plurality of primitive information in multi-bit form is obtained by the brain-inspired vision sensor performing primitive extraction on the collected spatio-temporal difference image and color intensity image, and the plurality of primitive information in single-bit form is obtained by the brain-inspired vision sensor encoding each type of primitive information through a pulse encoder using different encoding methods; Based on each piece of primitive information in multi-bit form or each piece of primitive information in single-bit form, controlling an implantable electrode array implanted in different regions of the human visual processing pathway to generate an electrical stimulation signal, where the electrical stimulation signal is used to stimulate the corresponding region through the implantable electrode array implanted in the corresponding region; Receiving the electroencephalogram signal of the corresponding region collected by the implantable electrode array; Based on the electroencephalogram signal, generating a regulation signal through an electroencephalogram decoding feedback regulator, and adjusting the attribute parameters of the brain-inspired vision sensor based on the regulation signal, so as to re-collect the external environment based on the adjusted brain-inspired vision sensor, and stimulate the corresponding region based on the re-collected image.

[0110] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed to multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment. Those of ordinary skill in the art can understand and implement it without creative efforts.

[0111] Through the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. The computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disc, etc., and includes several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0112] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions described in the foregoing embodiments, or perform equivalent replacements for some of the technical features; and these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A primitive-based brain-like visual prosthesis system, characterized in that: It includes a primitive-based brain-like visual sensor, an implantable electrode array and a processor, wherein the primitive-based brain-like visual sensor and the implantable electrode array are both connected to the processor; The brain-like visual sensor is used to collect data from the external environment to obtain a spatiotemporal difference image and a color intensity image, perform primitive extraction on the spatiotemporal difference image and the color intensity image to obtain a plurality of primitive information in a multi-bit form, and use a pulse encoder to encode the primitive information in a different encoding manner according to the type of the primitive information to obtain a plurality of primitive information in a single-bit form, and send the plurality of primitive information in a multi-bit form or the plurality of primitive information in a single-bit form to the processor; The processor is used to control the implanted electrode arrays implanted in different areas of the human visual processing pathway to generate electrical stimulation signals based on the primitive information in multi-bit form or the primitive information in single-bit form, wherein the electrical stimulation signals are used to stimulate the corresponding areas through the implanted electrode arrays implanted in the corresponding areas; The implantable electrode array is used to collect EEG signals of the corresponding area and send the EEG signals to the processor; The processor is also used to generate a control signal based on the EEG signal through an EEG decoding feedback controller, and adjust the attribute parameters of the brain-like visual sensor based on the control signal, so as to re-collect the external environment based on the adjusted brain-like visual sensor, and stimulate the corresponding area based on the re-collected image.

2. The primitive-based brain-like visual prosthesis system according to claim 1, characterized in that: The plurality of primitive information in multi-bit form includes contour information, change information, color information and light intensity information; The brain-like visual sensor is specifically used to perform primitive extraction on the spatiotemporal difference image to obtain contour information and change information of the external environment, perform primitive extraction on the color intensity image to obtain color information and light intensity information of the external environment, and use the pulse encoder to binary encode the contour information to obtain first primitive information in a single-bit form, use time encoding on the change information to obtain second primitive information in a single-bit form, and use rate encoding on the color information and the light intensity information to obtain third primitive information in a single-bit form.

3. The primitive-based brain-like visual prosthesis system according to claim 2, characterized in that: The processor is specifically used to control a first implantable electrode array implanted in a retinal area to generate a first electrical stimulation signal encoded with the first primitive information and the second primitive information based on the first primitive information and the second primitive information, wherein the first electrical stimulation signal is used to stimulate the rod cells in the retinal area; The processor is specifically used to control a second implantable electrode array implanted in the retinal area to generate a second electrical stimulation signal encoded with the third primitive information based on the third primitive information, wherein the second electrical stimulation signal is used to stimulate cone cells in the retinal area.

4. The primitive-based brain-like visual prosthesis system according to claim 2, characterized in that: The processor is specifically used to control the third implantable electrode array implanted in the primary visual cortex V1 area to generate a third electrical stimulation signal encoded with the first primitive information, the second primitive information and the third primitive information based on the first primitive information, the second primitive information and the third primitive information, and the third electrical stimulation signal is used to stimulate the primary visual cortex V1 area.

5. The primitive-based brain-like visual prosthesis system according to claim 2, characterized in that: The processor is specifically used to extract features from a plurality of primitive information in multi-bit form to obtain semantic information; The processor is specifically configured to control a fourth implantable electrode array implanted in the primary visual cortex V1 region to generate a fourth electrical stimulation signal encoded with the first primitive information and the third primitive information based on the first primitive information and the third primitive information, wherein the fourth electrical stimulation signal is used to stimulate the primary visual cortex V1 region; The processor is specifically configured to control a fifth implantable electrode array implanted in the visual cortex V2 region to generate a fifth electrical stimulation signal encoded with the second primitive information based on the second primitive information, wherein the fifth electrical stimulation signal is used to stimulate the visual cortex V2 region; The processor is specifically used to control a sixth implantable electrode array implanted in the visual cortex V4 region to generate a sixth electrical stimulation signal encoding the semantic information based on the semantic information, and the sixth electrical stimulation signal is used to stimulate the visual cortex V4 region.

6. The primitive-based brain-like visual prosthesis system according to claim 2, characterized in that: The processor is specifically configured to determine the motion attention information based on the second primitive information through the EEG signal predictor, generate contrast enhancement information based on the third primitive information, and predict the simulated neural signal based on the motion attention information, the contrast enhancement information and the first primitive information; The processor is specifically used to determine stimulation parameters executable by the implantable electrode array based on the simulated neural signal, and control the implantable electrode array to generate the electrical stimulation signal based on the stimulation parameters.

7. The primitive-based brain-like visual prosthesis system according to claim 6, characterized in that: The brain-like visual sensor is used to collect the depth of the external environment, obtain depth information, and send the depth information to the processor; The processor is further configured to generate a reconstructed image based on the first primitive information, the second primitive information and the third primitive information through an image reconstructor, segment the reconstructed image through an image segmentor to obtain a target segmentation mask, and generate a depth image based on the depth information and the reconstructed image through a depth estimator; The processor is also used to predict the simulated neural signal based on the first primitive information, the second primitive information, the third primitive information, the depth image, the reconstructed image and the target segmentation mask through the EEG signal predictor.

8. The primitive-based brain-like visual prosthesis system according to claim 1, characterized in that: The processor is specifically used to extract features of the EEG signal through an EEG decoding feedback controller to obtain intensity matching features and frequency synchronization features; The processor is specifically used to generate the control signal when at least one of the following preset conditions is not met, and the preset conditions include: the intensity matching feature represents that the pulse intensity of the electrical stimulation signal reaches the neural activation threshold, and the frequency synchronization feature represents that the pulse frequency of the electrical stimulation signal resonates with the neural oscillation.

9. The primitive-based brain-like visual prosthesis system according to any one of claims 2 to 7, characterized in that: The processor is specifically configured to determine the target primitive combination information type corresponding to the identifier of the implantable electrode array and the target area based on the correspondence between the area, the electrode array identifier and the primitive combination information type, and determine information corresponding to the target primitive combination information type from the first primitive information, the second primitive information and the third primitive information to obtain the target primitive combination information; The processor is specifically used to control the implantable electrode array to generate the electrical stimulation signal based on the target primitive combination information.

10. A primitive-based partitioned electrical stimulation method, characterized in that: The method applied to the primitive-based brain-like visual prosthesis system according to any one of claims 1 to 9 comprises: Receiving a plurality of primitive information in a multi-bit form or a plurality of primitive information in a single-bit form sent by a primitive-based brain-like visual sensor, wherein the plurality of primitive information in a multi-bit form is obtained by the brain-like visual sensor performing primitive extraction on a collected spatiotemporal differential image and a color intensity image, and the plurality of primitive information in a single-bit form is obtained by the brain-like visual sensor encoding the primitive information in different encoding modes using a pulse encoder according to the type of each primitive information; Based on each primitive information in multi-bit form or each primitive information in single-bit form, control the implanted electrode arrays implanted in different areas of the human visual processing pathway to generate electrical stimulation signals, wherein the electrical stimulation signals are used to stimulate the corresponding areas through the implanted electrode arrays implanted in the corresponding areas; Receiving EEG signals of the corresponding area collected by the implanted electrode array; The EEG decoding feedback controller generates a control signal based on the EEG signal, and adjusts the property parameters of the brain-like visual sensor based on the control signal, so as to re-acquire the external environment based on the adjusted brain-like visual sensor, and stimulate the corresponding area based on the re-acquired image.

11. A primitive-based partitioned electrical stimulation device, characterized in that: include: A first receiving unit is used to receive a plurality of primitive information in a multi-bit form or a plurality of primitive information in a single-bit form sent by a primitive-based brain-like visual sensor, wherein the plurality of primitive information in a multi-bit form is obtained by the brain-like visual sensor performing primitive extraction on a collected spatiotemporal differential image and a color intensity image, and the plurality of primitive information in a single-bit form is obtained by the brain-like visual sensor encoding each type of primitive information by using a pulse encoder in a different encoding manner; A control unit, for controlling the implanted electrode arrays implanted in different areas of the human visual processing pathway to generate electrical stimulation signals based on the primitive information in multi-bit form or the primitive information in single-bit form, wherein the electrical stimulation signals are used to stimulate the corresponding areas through the implanted electrode arrays implanted in the corresponding areas; A second receiving unit is used to receive the EEG signal of the corresponding area collected by the implantable electrode array; An adjustment unit is used to generate a control signal based on the EEG signal through an EEG decoding feedback controller, and adjust the attribute parameters of the brain-like visual sensor based on the control signal, so as to re-collect the external environment based on the adjusted brain-like visual sensor, and stimulate the corresponding area based on the re-collected image.

12. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that: When the processor executes the computer program, the primitive-based partitioned electrical stimulation method according to claim 10 is implemented.

13. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the primitive-based partitioned electrical stimulation method according to claim 10 is implemented.

14. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the primitive-based partitioned electrical stimulation method according to claim 10 is implemented.

Images