Control system and method for visual prosthesis

Abstract

The invention discloses a control system and method for a visual prosthesis, and provides three schemes including a virtual patient model, bandwidth adaptive coding and a multi-modal closed loop for solving the three problems of insufficient training and evaluation channels, limited implantation link bandwidth and disjunction of simulation and real experience in visual prosthesis research and development. A repeatable, low-risk and high-efficiency control system is constructed, the virtual patient model is used for replacing scarce real implanters to perform large-scale tests, the dependence on the implanters is reduced, the dependence on frequently-used implanters is not needed any more, key information is preferentially transmitted in a limited bandwidth by using the virtual patient model, and the definition and efficiency of visual reconstruction are improved. Moreover, according to the feedback information of the experiencer, the system collects feedback and safety threshold values in real time, the next round of light spots are automatically adjusted, a closed loop of stimulation-perception-adjustment is formed, and the research and development efficiency and effect of the visual prosthesis can be effectively improved.

Description

Technical Field

[0001] This application relates to the field of visual prosthesis research and development technology, and in particular to a control system and method for visual prostheses. Background Technology

[0002] A visual prosthesis is an implantable device that provides artificial visual perception to certain types of blind people by stimulating the visual pathway with electricity or light. Instead of restoring natural vision, it generates patterns of light to help users perceive contours, motion, and light and shadow information in their environment, thereby improving independence and quality of life.

[0003] However, existing technologies have several drawbacks. First, the stimulation parameters are almost fixed. The camera image is usually scaled down to "square pixels" and then mapped to the electrodes. A small number of electrodes results in a loss of detail, and parameters such as brightness, frequency, and pulse width can often only be adjusted manually and slowly after surgery. Second, parameter tuning is highly dependent on the implant recipient. Due to the limited number of recipients, each trial and error requires a hospital visit, resulting in a long process and placing a significant burden on patients. Third, early simulations could only be demonstrated offline. Although attempts were made to load regular pixel patterns into VR headsets to allow sighted individuals to experience low-resolution vision, the approach was rudimentary, involving downscaling photos to specific grayscale levels for playback. Real-time modifications were not possible, and objective feedback was difficult to obtain.

[0004] These problems have led to insufficient training and evaluation channels, limited implantation bandwidth, and a disconnect between simulation and real-world experience during the development of visual prostheses, severely impacting the efficiency and effectiveness of visual prosthesis development. Summary of the Invention

[0005] In view of this, this application provides a system and method for the development of visual prostheses to solve the above-mentioned technical problems.

[0006] In a first aspect, this application provides a control system for a visual prosthesis, the system comprising a multi-source sensing unit, a computing processing unit, a virtual patient model, a bandwidth adaptive coding unit, and a display and feedback unit; wherein the multi-source sensing unit is connected to the computing processing unit; the virtual patient model is connected to the computing processing unit, the bandwidth adaptive coding unit, and the display and feedback unit respectively; and the bandwidth adaptive coding unit is connected to the display and feedback unit.

[0007] The multi-source sensing unit is used to collect environmental information through multiple sensors.

[0008] The computational processing unit is used to fuse the environmental information and output the corresponding saliency map;

[0009] The virtual patient model is used to score each pixel of the saliency map according to the scene in which the virtual patient is located, and to determine N key light points according to a preset upper limit for the number of safety pulses; the N key light points are the light points that match the projection of the saliency map onto the cortical topological optical illusion dot matrix, and N is an integer greater than 1;

[0010] The bandwidth adaptive coding unit is used to filter the N key light spots to obtain the filtered key light spots;

[0011] The display and feedback unit is used to project the selected key light points onto the lens and receive feedback information for the selected key light points.

[0012] The virtual patient model is also used to automatically adjust the brightness threshold and spot selection rules based on the feedback information.

[0013] Optionally, the virtual patient model includes:

[0014] A saliency calculation module is used to score each pixel of the saliency map according to the scene in which the virtual patient is located, so as to obtain a scoring result, which is used to indicate the importance of the pixel;

[0015] The bit width management module is used to read the number of pulses that the virtual brain electrodes can safely output to determine the upper limit of the number of safe pulses in the current frame;

[0016] The sparse selection module is used to sort multiple pixels of the saliency map from high to low according to the upper limit of the number of safety pulses and the scoring result, and project the sorted saliency map onto the cortical topological optical illusion matrix to determine the matching N key light points.

[0017] The coordinate remapping module is used to convert the coordinates of the N key light points into electrode numbers of the cerebral cortex by using a pre-set correspondence between light points and electrode numbers.

[0018] Optionally, the bandwidth adaptive coding unit is specifically used to perform bandpass filtering on the N key light spots according to the effective threshold and safety threshold of the visual cortex electrical stimulation to obtain the filtered key light spots, wherein the effective threshold of the visual cortex electrical stimulation is the minimum value of the stimulation intensity of the electrode that can satisfy the generation of photic hallucinations, and the safety threshold is the minimum value between the value of patient-perceived discomfort and the value of physiological tissue safety.

[0019] Optionally, the system further includes a stimulation interface unit connected to the display and feedback unit;

[0020] The stimulation interface unit is used to acquire the instantaneous current and cumulative charge of the electrode of the patient's implant in real time, and automatically perform safety operations when the instantaneous current exceeds a first threshold or the cumulative charge exceeds a second threshold.

[0021] Optionally, the stimulation interface unit includes:

[0022] The pulse limit module is used to set the pulse width limit and peak current limit of the electrode based on clinical safety data.

[0023] The charge counting module is used to continuously count the accumulated charge using a hardware counter.

[0024] The over-limit processing module is used to reduce the pulse frequency and decrease the brightness of the electrode when the instantaneous current exceeds a first threshold or the cumulative charge exceeds a second threshold. If the adjusted instantaneous current still exceeds the first threshold or the adjusted cumulative charge still exceeds the second threshold, the corresponding electrode is temporarily turned off.

[0025] Optionally, the virtual patient model further includes:

[0026] The cortical and visual field localization module is used to convert the planar coordinates of each electrode in the cerebral cortex into the corresponding visual field angle position based on the preset projection relationship between the retina and the cortex and the coordinates of the electrode array.

[0027] The current diffusion calculation module is used to calculate the radial attenuation range of the stimulation current in the cortical tissue based on the pulse amplitude and pulse width of each electrode, combined with the empirical constant of tissue conductivity; and to obtain the diameter parameter of the light spot at the visual field angle position based on the radial attenuation range.

[0028] The brightness and threshold conversion module is used to map the peak current of the electrode to the brightness of the center circle, and generate a two-dimensional Gaussian distribution of the brightness of the light spot using the brightness of the center circle as the amplitude and the diameter parameter as the size of the light spot. The brightness of the light spot decreases radially from the center to the periphery. When the number of times the target electrode is triggered exceeds the third threshold within a preset time window, the output brightness of the target electrode is automatically reduced.

[0029] Optionally, the model parameters of the virtual patient model are fine-tuned using model training data, which includes training clinical threshold parameters and psychophysical test output files of the patient implant. The psychophysical test output files include stimulation thresholds, implanted electrode sites, spatial visual field mapping areas, stimulation intensity, and photic parameters.

[0030] Optionally, the sensor includes at least one of a color camera, a depth camera, a near-infrared camera, an ultrasonic probe, or a millimeter-wave probe, and the sensor is synchronized via a synchronization bus or a common clock.

[0031] Secondly, this application provides a control method for a visual prosthesis, applied to the control system described in the first aspect above, the method comprising:

[0032] Environmental information is collected using multiple sensors.

[0033] After fusing the environmental information, the corresponding saliency map is output.

[0034] Each pixel of the saliency map is scored according to the scene in which the virtual patient is located, and N key light points are determined according to the preset upper limit of the number of safety pulses; the N key light points are the light points that match the projection of the saliency map onto the cortical topological optical illusion dot matrix, and N is an integer greater than 1;

[0035] The N key light spots are filtered to obtain the filtered key light spots;

[0036] The selected key light spots are projected onto the lens, and feedback information on the selected key light spots is received.

[0037] The brightness threshold and selection rules are automatically adjusted based on the feedback information.

[0038] Optionally, the method further includes:

[0039] The system acquires the instantaneous current and cumulative charge of the patient's implanted electrode in real time, and automatically performs safety operations when the instantaneous current exceeds a first threshold or the cumulative charge exceeds a second threshold.

[0040] Thirdly, embodiments of this application provide an electronic device, the electronic device comprising:

[0041] Memory, used to store one or more programs;

[0042] A processor; when the processor executes the one or more programs, it implements the control method for visual prostheses as described in any of the first aspects above.

[0043] Fourthly, embodiments of this application provide a computer storage medium storing a program that, when executed by a processor, implements the control method for visual prostheses described in any of the first aspects.

[0044] The above technical solution has the following beneficial effects:

[0045] This application provides a control system and method for visual prostheses. The system includes a multi-source sensing unit, a computational processing unit, a virtual patient model, a bandwidth adaptive coding unit, and a display and feedback unit. The multi-source sensing unit is connected to the computational processing unit. The virtual patient model is connected to the computational processing unit, the bandwidth adaptive coding unit, and the display and feedback unit. The bandwidth adaptive coding unit is connected to the display and feedback unit. The multi-source sensing unit is used to collect environmental information through multiple sensors. The computational processing unit is used to fuse the environmental information and output a corresponding saliency map. The virtual patient model... The model is used to score each pixel of the saliency map based on the scene in which the virtual patient is located, and to determine N key light points according to a preset upper limit for the number of safety pulses. The N key light points are the light points that match the projection of the saliency map onto the cortical topological optical illusion dot matrix, where N is an integer greater than 1. The bandwidth adaptive coding unit is used to filter the N key light points to obtain the filtered key light points. The display and feedback unit is used to project the filtered key light points onto the lens and receive feedback information regarding the filtered key light points. The virtual patient model is also used to automatically adjust the brightness threshold and the selection rules based on the feedback information. Through the above technical solution, a virtual patient model is used to replace scarce real implant subjects for large-scale trials, reducing reliance on implant subjects and eliminating the need for frequent implant use. The virtual patient model prioritizes the transmission of key information within limited bandwidth, improving the clarity and efficiency of visual reconstruction. Moreover, based on feedback from users, the system collects feedback and safety thresholds in real time and automatically adjusts the light spots for the next round, forming a closed loop of "stimulation-perception-adjustment", which can effectively improve the efficiency and effectiveness of visual prostheses. Attached Figure Description

[0046] To more clearly illustrate the technical solutions in this embodiment or the prior art, the drawings used in the description of the embodiment or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0047] Figure 1 A schematic diagram of a control system for a visual prosthesis provided in an embodiment of this application;

[0048] Figure 2 Another structural schematic diagram of the control system for a visual prosthesis provided in an embodiment of this application;

[0049] Figure 3 Another structural schematic diagram of the control system for a visual prosthesis provided in an embodiment of this application;

[0050] Figure 4 A schematic diagram illustrating the data flow provided in the embodiments of this application;

[0051] Figure 5 A schematic diagram of a user interface for a control system for a visual prosthesis provided in an embodiment of this application;

[0052] Figure 6 A schematic diagram of another user interface for a control system for a visual prosthesis provided in an embodiment of this application;

[0053] Figure 7 A schematic diagram of another user interface for a control system for a visual prosthesis provided in an embodiment of this application;

[0054] Figure 8 A schematic diagram of another user interface for a control system for a visual prosthesis provided in an embodiment of this application;

[0055] Figure 9 This is a schematic flowchart of a control method for a visual prosthesis provided in an embodiment of this application. Detailed Implementation

[0056] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of this application.

[0057] The technical terms used in this application are described below:

[0058] Photopic visual points: bright spots or flashes of light that appear in the patient's field of vision after electrical stimulation, used to convey visual information.

[0059] Virtual Patient Model (VPM): A simulation model constructed using bioelectrical conduction equations and visual field-cortical projection rules, which can predict the "electrode stimulation -> subjective visual hallucination" link and dynamically correct it with clinical data.

[0060] Multi-source sensing unit: A data acquisition module that combines sensors such as color camera, depth camera, near-infrared, ultrasound, and millimeter wave sensors.

[0061] Polar-logarithmic mapping: The visual field coordinates are converted into primary visual cortex topological coordinates using a polar-logarithmic function, making the central area points dense and the periphery sparse.

[0062] LUT lookup table: The correspondence between "input coordinates and electrode number" is stored in advance, and the mapping is accelerated by directly looking up the table at runtime.

[0063] SoC: A computing platform that integrates processors, memory, interfaces, etc. on a single chip.

[0064] Event camera: Outputs events only when pixel brightness changes, suitable for low-latency capture in high-speed scenes.

[0065] Frequency modulation: A stimulation method that encodes brightness or grayscale information by adjusting the pulse frequency rather than the amplitude.

[0066] Edge-cloud collaboration: The head-mounted device completes basic processing, while a large amount of computation or database matching is handled by the mobile phone or cloud, and then a simplified result is returned.

[0067] Finite element method (FE): A numerical method that divides a continuous medium into small elements and solves for electric fields or stresses, used to quickly estimate the distribution of stimuli.

[0068] Electric field threshold formula: an empirical curve describing the perceptible response of neural tissue under a specific electric field.

[0069] Millimeter-wave radar: emits electromagnetic waves with a wavelength of millimeters to measure distance and speed and penetrate smoke or fog.

[0070] Vibration belt: A wearable device that transmits spatial or alarm information to the wearer through localized vibration.

[0071] Interest Window: A small area in the image containing key targets. Only this area is processed with high resolution to save computing power and bandwidth.

[0072] Sparse selection: Low-value pixels are removed from the saliency map by weight, leaving only a small number of key points.

[0073] Safety current limiting: Sets upper limits on the peak value of the output pulse and the cumulative charge to prevent tissue overheating or electrolytic reactions.

[0074] Matrix: A two-dimensional distribution composed of several optically illuminating points, used to present low-resolution visual scenes within the patient's field of vision.

[0075] To facilitate a further understanding of the technical solutions provided in this application, the background technology involved in this application will be explained below.

[0076] Visual prostheses have been researched for over two decades, with early products mostly following two paths: one is retinal stimulation (e.g., Argus II, which can only illuminate 60 electrodes), and the other is cortical stimulation (commonly seen in the Utah array experiment). Their common architecture can be summarized in four steps:

[0077] Head-mounted camera -> External processing unit -> Implanted electrodes -> Patient sees light spots

[0078] Along this chain, the industry generally faces several types of problems:

[0079] 1. Stimulus parameters are almost fixed.

[0080] Camera images are typically scaled down proportionally into a bunch of "square pixels," and then arranged by row and column to correspond to electrodes. Fewer electrodes mean less detail; and parameters such as brightness, frequency, and pulse width can often only be adjusted manually after surgery.

[0081] 2. Parameter tuning is highly dependent on the implantee.

[0082] The number of recipients is limited, and each trial and error requires a trip to the hospital, resulting in a long process and a heavy burden.

[0083] 3. Early simulations could only be demonstrated offline.

[0084] If regular pixel patterns are loaded into a VR headset, it allows sighted users to experience low-resolution vision. However, the approach is very primitive: the photo is shrunk to 32×32 or 40×40 grayscale and then played back. It cannot be modified in real time, nor can objective feedback be obtained.

[0085] 4. Poor scene adaptation when using only a color camera.

[0086] When the light is dim, backlit, or on a glass floor, the camera noise spikes, and the edges of stairs and obstacles are completely obscured; some teams have added depth or ultrasonic sensors to the white cane, but it only beeps or vibrates, which is completely different from the light spot vision.

[0087] In existing technology, a color camera is hung in front of the head-mounted display device, and a small processing box is placed on the side to convert the camera output into a dot matrix. The dot matrix image is then projected onto the head-mounted display screen in a white dot and black background format.

[0088] The steps are as follows: 1. Convert the scene to grayscale; 2. Shrink it to a uniform 32×32 grid; 3. The brightness of each grid is directly mapped to the "electrode current"; 4. The participants look at these white dots in the corridor and grab objects; 5. The participants verbally say "brighter" or "darker"; 6. The researchers manually change the scaling ratio or threshold and repeat the process.

[0089] The problems with this existing technology include:

[0090] The points are too evenly distributed—the electrode resources on door frames, stair edges, and walls are the same, and key lines are not highlighted.

[0091] Single-sensor photography—relying solely on a color camera—will fail in low light or due to glass reflections.

[0092] There is no closed loop—the user's reaction cannot be entered into the algorithm, and the changes rely entirely on human guesswork.

[0093] Fixed bandwidth—replacing with a prosthesis that has fewer electrodes requires manually adjusting the grid size, which can easily lead to overclocking or distortion.

[0094] The problems with existing technologies include:

[0095] 1. Insufficient training and assessment channels

[0096] Very few people actually wear visual prostheses, and each parameter adjustment requires surgical time or increases physical burden. The research team lacks a low-cost, non-invasive system that can be repeatedly tested to verify whether the brightness, position, and frequency of the light spot are appropriate, and it is even more difficult to systematically compare the advantages and disadvantages of different algorithms for navigation, literacy, and obstacle avoidance.

[0097] 2. Limited bandwidth of implanted link

[0098] Limited by the number of electrodes, the cross-sectional area of ​​the leads, tissue heating, and the energy transmission capacity of the external machine, existing cortical, optic nerve, or retinal prostheses can only safely output a limited number of pulse bits per second. If the high-resolution images captured by the camera are directly compressed into a regular grid and then stimulated point by point, it will not only exceed the bandwidth limit but also waste the most valuable electrode resources, resulting in blurred visual information and insufficient detail.

[0099] 3. The simulation is disconnected from the real experience.

[0100] Most existing methods involve displaying a uniform matrix of dots on a head-mounted display, with users subjectively describing their experience, and researchers manually adjusting the parameters. This "see first, then modify" process cannot automatically incorporate feedback into stimulus generation. Repeated manual trial and error is time-consuming and difficult to quantify, and the algorithm, device, and user remain disconnected.

[0101] To overcome some of the aforementioned technical problems, this application provides a control system and method for visual prostheses. It utilizes a virtual patient model, replacing the scarce implant recipients with a calculable and fine-tunable model. Large-scale trials are first conducted on a computer, followed by final calibration using a small amount of real-person data. Only the most useful pixels are selected from multi-sensor data, and light spots are distributed to electrodes according to the dense-sparse distribution of the primary visual cortex, ensuring bandwidth is not exceeded and information is not excessively diluted. A multimodal closed loop is established: the user in the glasses or head-mounted display sees the compressed light spots (consistent with cortical projection), completes tasks such as navigation and literacy, and receives feedback via buttons or voice. The system collects these feedbacks and safety thresholds in real time, automatically adjusting the light spots for the next round, forming a closed loop of "stimulus-perception-adjustment."

[0102] By using these three steps, people with normal vision can first experience the optical illusion viewpoint while wearing the device, and then the feedback can be immediately used to improve the algorithm, preparing for future implantation. The research and development team can quickly refine a clearer and more efficient visual reconstruction strategy within a safe range without frequently using implant recipients.

[0103] See Figure 1 , Figure 1 This is a schematic diagram of a control system for a visual prosthesis provided in an embodiment of this application.

[0104] The system includes a multi-source sensing unit 101, a computing processing unit 102, a virtual patient model 103, a bandwidth adaptive coding unit 104, and a display and feedback unit 105.

[0105] The multi-source sensing unit 101 is connected to the computing processing unit 102; the virtual patient model 103 is connected to the computing processing unit 102, the bandwidth adaptive coding unit 104, and the display and feedback unit 105 respectively; the bandwidth adaptive coding unit 104 is connected to the display and feedback unit 105.

[0106] In practical applications, such as Figure 2 and Figure 3 As shown, Figure 2 Another structural schematic diagram of the control system for a visual prosthesis provided in an embodiment of this application; Figure 3 This is another schematic diagram of the control system for visual prostheses provided in an embodiment of this application.

[0107] in, Figure 2 The attached figures are labeled as follows: 200: Head-mounted display or AR glasses; 201: Computer display screen; 202: Electrode array; 203: AR glasses lens; 204: Ultrasonic probe; 205a: Binocular camera lens (left); 205b: Binocular camera lens (right); 206: Near-infrared camera; 207: RGB camera; 208: Data transmission line; 209: Computing processing unit; 210: Low-power processor / micro graphics card; 211: Microprocessor (implementing safety logic).

[0108] Figure 3 The attached figures are labeled as follows: 300: Headset or AR glasses; 301: Visual prosthesis system; 302: External signal processing module; 303: AR glasses lens; 304: Ultrasonic probe; 305a: Binocular camera lens (left); 305b: Binocular camera lens (right); 306: Near-infrared camera; 307: RGB camera; 308: Brain; 309: Electrode array; 310: Primary visual cortex area; 311: Data transmission line; 312: IPG: Implantable pulse generator; 313: Data processing unit; 314: Waist belt; 315: Wireless charging unit.

[0109] The multi-source sensing unit is used to collect environmental information through multiple sensors.

[0110] In one possible implementation, the sensor includes at least one of a color camera, a depth camera, a near-infrared camera, an ultrasonic probe, or a millimeter-wave probe, and the sensor is synchronized via a synchronization bus or a common clock.

[0111] Understandably, monocular / binocular color cameras, depth cameras, and near-infrared cameras can be embedded within the head-mounted display or glasses; external ultrasonic or millimeter-wave probes can also be attached. All sensors share a single synchronization line or common clock. Color, depth, near-infrared, and ultrasonic sensors are freely pluggable; the internal compression module first captures the most useful information such as edges, text, and obstacles, and then compresses it to the number of pulses that the electrodes can withstand, thus avoiding both overload and waste.

[0112] It should be noted that the multi-source sensing unit supports any combination of color-depth cameras, near-infrared sensors, ultrasonic sensors, millimeter-wave radar, inertial measurement units, etc. Furthermore, it adopts a unified synchronization bus and interface specification, so replacing or adding sensors does not require changes to the backend processing flow.

[0113] Optionally, for multi-source sensing units, depending on the needs of different scenarios, such as pursuing a minimalist configuration, only one color camera can be retained and an infrared fill light can be added. If dealing with high-speed scenarios, an event camera with millimeter-wave radar can be used instead. Moreover, depth information can also be obtained by single-line LiDAR, replacing binocular or structured light cameras. This means that the sensor type can be adjusted according to the needs of the scenario to ensure that key environmental information can still be effectively captured under different lighting and speed conditions.

[0114] The computational processing unit is used to fuse the environmental information and output the corresponding saliency map.

[0115] It should be noted that the computing unit can be equipped with a low-power processor or a micro graphics card to perform timestamp alignment of the sensors and output the corresponding saliency map after multi-modal information fusion of environmental information.

[0116] Optionally, when the computing power of the computing unit is limited, classic algorithms such as Sobel edge detection can be used to complete the contour extraction; when conditions are sufficient, template matching can be introduced to improve the accuracy of text or step recognition, ensuring that feature extraction can be achieved under different computing capabilities.

[0117] Optionally, for the hardware configuration of the computing processing unit, a low-power DSP plus SRAM, a dedicated ASIC / RISC-V accelerator, and access to a laptop / industrial PC in desktop emulation mode can be used to "achieve low-latency data processing". The hardware architecture can be adjusted according to the scenario and performance requirements to ensure that the total data processing latency meets the requirements of tasks such as walking and reading.

[0118] The virtual patient model is used to score each pixel of the saliency map according to the scene in which the virtual patient is located, and to determine N key light points according to a preset upper limit for the number of safety pulses; the N key light points are the light points that match the projection of the saliency map onto the cortical topological optical illusion matrix, and N is an integer greater than 1.

[0119] The virtual patient model incorporates visual field-cortical projection and neural electrical conduction equations, and can import clinical thresholds or subjective scores. Based on the neural electrical conduction equations and visual field-cortical projection rules, it can automatically correct itself after importing implantation thresholds, evoked potentials, and subjective scores, and can be used to replace real implantees to predict the "electrode stimulation -> cortical activity -> photic hallucinations" pathway in large quantities.

[0120] Optionally, embodiments of this application may also employ wavelet layered compression with threshold clipping, or divide the image into interest windows, with fixed grid sampling performed within each window; for text reading, the center line of the line frame can be directly extracted, and for navigation, the ground center trajectory can be output, all of which can significantly reduce the number of points. Redundant data is reduced through different compression logics.

[0121] In one possible implementation, the virtual patient model includes:

[0122] A saliency calculation module is used to score each pixel of the saliency map according to the scene in which the virtual patient is located, so as to obtain a scoring result, which is used to indicate the importance of the pixel;

[0123] The bit width management module is used to read the number of pulses that the virtual brain electrodes can safely output to determine the upper limit of the number of safe pulses in the current frame;

[0124] The sparse selection module is used to sort multiple pixels of the saliency map from high to low according to the upper limit of the number of safety pulses and the scoring result, and project the sorted saliency map onto the cortical topological optical illusion matrix to determine the matching N key light points.

[0125] The coordinate remapping module is used to convert the coordinates of the N key light points into electrode numbers of the cerebral cortex by using a pre-set correspondence between light points and electrode numbers.

[0126] Specifically, in this embodiment, the saliency calculation module scores each pixel of the saliency map based on the scene in which the virtual patient is located, to obtain a scoring result. The scoring result is used to indicate the importance of the pixel. The higher the score, the more important the corresponding pixel.

[0127] The scenarios in which virtual patients are located can include various applications such as indoor navigation, staircase recognition, and text reading.

[0128] In practical applications, brightness and size can be dynamically allocated to each light point based on task labels (navigation, stairs, reading, faces, etc.) and target distance, highlighting key targets within a fixed number of points and improving recognition and obstacle avoidance efficiency.

[0129] The bit-width management module determines the upper limit of the safe pulse count for the current frame by reading the number of pulses that the virtual brain electrodes can currently safely output. For example, the current safe output pulse count is 800 pulses per second.

[0130] The sparse selection module sorts multiple pixels in the saliency map from highest to lowest score based on the upper limit of the number of safety pulses and the scoring results. The sorted saliency map is then projected onto a cortical topological optical illusion matrix to determine the N matching key light points. In essence, by queuing the scored pixels from highest to lowest and selecting only the top N, the background and low-scoring regions are discarded. This saves bandwidth and avoids wasting electrodes.

[0131] The coordinate remapping module is used to convert the coordinates of the N key light points into electrode numbers of the cerebral cortex by using a pre-set correspondence between light points and electrode numbers.

[0132] Understandably, the virtual patient model automatically selects the most valuable sparse feature points within the limits of the implanted electrode channel; and outputs a pulse sequence that directly corresponds to the electrode number, and detects current and charge in real time, automatically reducing peaks when the total amount exceeds the limit.

[0133] The bandwidth adaptive coding unit is used to filter the N key light spots to obtain the filtered key light spots.

[0134] like Figure 4 The diagram illustrates a data flow provided in an embodiment of this application. The virtual patient model achieves "electrode stimulation -> photohallucination" prediction through dual mapping: First, the pulse frequency, pulse width, and amplitude of each electrode are converted into the probability of light spot appearance, area, and brightness, respectively; then, based on the retinal-cortical topology, the stimulation position of each electrode on the cortex is mapped to the corresponding visual field coordinates, generating a photohallucination intensity distribution map for each electrode within the model; finally, all electrode layers are superimposed pixel by pixel to obtain a comprehensive photohallucination dot matrix. This dot matrix can be directly used as input to the bandwidth adaptive coding unit, or compared with the subjective scores collected by the display and feedback units and the weights can be written back to achieve closed-loop correction of stimulation parameters and perceptual effects.

[0135] In one possible implementation, the bandwidth adaptive coding unit is specifically used to perform bandpass filtering on the N key light spots according to the effective threshold and safety threshold of the visual cortex electrical stimulation to obtain the filtered key light spots, wherein the effective threshold of the visual cortex electrical stimulation is the minimum value of the stimulation intensity of the electrode that can satisfy the generation of photic hallucinations, and the safety threshold is the minimum value between the value of patient-perceived discomfort and the physiological tissue safety value.

[0136] Specifically, based on the effective threshold and safety threshold of visual cortical electrical stimulation, bandpass filtering is performed on the N key light spots: light spots with brightness higher than the lower limit of perception and lower than the upper limit of safety are retained to obtain the filtered key light spots.

[0137] It should be noted that by screening N key light points, it is ensured that the stimulation intensity of the selected key light points is not lower than the effective threshold, thus guaranteeing the stable generation of photopsychic visions and avoiding information transmission failure due to insufficient intensity. Secondly, strictly controlling the intensity within the safety threshold can prevent excessive stimulation intensity from causing patient discomfort or damage to physiological tissues, ensuring the safety of use. Thirdly, precise screening within the reasonable range formed by the effective threshold and the safety threshold can prioritize the retention of the most critical visual information when electrode bandwidth is limited, avoiding bandwidth waste, improving the efficiency of visual information transmission under limited resources, and making the photopsychic vision clearer and more prominent.

[0138] The display and feedback unit is used to project the selected key light points onto the lens and receive feedback information on the selected key light points.

[0139] Specifically, the display and feedback unit projects the selected key light points onto the lens. The user sees the optical illusion through the display and feedback unit and receives feedback information from the user regarding the selected key light points. For example, feedback can be submitted via a button or voice.

[0140] Understandably, the display and feedback unit projects selected key light points onto the lens, allowing users to directly see a light illusion consistent with the projection onto the cortex, thus enabling them to successfully complete tasks such as navigation and literacy, providing the R&D team with a testing foundation that closely resembles real-world usage scenarios. Simultaneously, this unit receives feedback information via buttons or voice, transmitting the user's feelings and task performance back to the system in real time, driving the formation of a closed loop of "stimulus -> perception -> adjustment," replacing manual trial and error, and significantly improving parameter optimization efficiency.

[0141] Alternatively, in addition to VR / AR headsets, embodiments of this application may use curved micro-LED masks, reflective LCOS projection, or display the dot matrix directly on a regular display for users with normal vision to experience.

[0142] The virtual patient model is also used to automatically adjust the brightness threshold and spot selection rules based on the feedback information.

[0143] Optionally, in extremely low power consumption scenarios, real-time learning can be canceled, and only the "calibration wizard" can be entered periodically to manually collect feedback for a few seconds and then update the threshold offline; if the device has a microphone, the brightness threshold can also be quickly adjusted via simple voice commands.

[0144] Understandably, the virtual patient model automatically adjusts the brightness threshold and spot selection rules based on feedback information, which can reduce reliance on rare visual prosthesis implantees. A large number of tests can be completed on a computer, requiring only a small amount of real-person data for calibration, reducing the physical burden and time cost for implantees. By receiving feedback in real time and dynamically adjusting parameters, it replaces the traditional manual trial-and-error process, greatly improving the efficiency of parameter optimization. The adjusted brightness threshold and spot selection rules can better fit the laws of real perception, making the generated optical illusions closer to the actual human experience, helping the R&D team to quickly develop a clearer and more efficient visual reconstruction strategy.

[0145] In one possible implementation, the virtual patient model further includes:

[0146] The cortical and visual field localization module is used to convert the planar coordinates of each electrode in the cerebral cortex into the corresponding visual field angle position based on the preset projection relationship between the retina and the cortex and the coordinates of the electrode array.

[0147] The current diffusion calculation module is used to calculate the radial attenuation range of the stimulation current in the cortical tissue based on the pulse amplitude and pulse width of each electrode, combined with the empirical constant of tissue conductivity; and to obtain the diameter parameter of the light spot at the visual field angle position based on the radial attenuation range.

[0148] The brightness and threshold conversion module is used to map the peak current of the electrode to the brightness of the center circle, and generate a two-dimensional Gaussian distribution of the brightness of the light spot using the brightness of the center circle as the amplitude and the diameter parameter as the size of the light spot. The brightness of the light spot decreases radially from the center to the periphery. When the number of times the target electrode is triggered exceeds the third threshold within a preset time window, the output brightness of the target electrode is automatically reduced.

[0149] The virtual patient model receives a "list" for each frame, which shows which electrodes are lit, their respective current amplitudes, and pulse widths.

[0150] The virtual patient model outputs a predicted holographic view, which combines all the light points using a Gaussian blur to create a low-resolution image with a human-friendly grayscale. This image is then sent directly to the head-mounted display, allowing the user to experience "pixelated vision."

[0151] Understandably, the virtual patient model first translates the visuals into "electrode pulses," then translates them into a pattern of light spots in front of the user based on neurophysiological perception. It also continuously improves the formula based on real-time feedback and can use real clinical patient perceptions as training data for the model, allowing for continuous iteration and a more biomimetic visual experience for future implantation in real people.

[0152] In practical applications, the cortical and visual field localization module of the virtual patient model: based on the preset projection relationship between the retina and the cortex and the coordinates of the electrode array, it uses a reference table similar to a "brain map" (such as latitude and longitude mapping) and a pole-logarithmic function to convert the cortical coordinates into visual field coordinates, clarifying the position of the visual field corresponding to each electrode, and realizing the spatial matching between the stimulus position and visual perception.

[0153] Optionally, the pole-logarithmic mapping formula can be replaced with piecewise linear polar coordinates or a LUT lookup table can be pre-generated for each electrode, and the conversion can be completed directly by looking up the table at runtime, thus optimizing processing efficiency.

[0154] Current diffusion calculation module: Combining the pulse amplitude, pulse width and empirical constant of tissue conductance of each electrode, the radial attenuation range of the stimulation current in the cortical tissue is calculated by simplified equation (referencing clinical results literature) (exhibiting square root attenuation characteristics), and this range is converted into the diameter parameter of the light spot in the visual field angle position to simulate the range of photopic vision caused by real current diffusion.

[0155] Brightness and threshold conversion module: Maps the peak current of the electrode to the brightness of the center of the light spot. Using the brightness of the center as the amplitude and the diameter parameter as the size, it generates a two-dimensional Gaussian distribution brightness map with a bright center and a radially decreasing periphery, which fits the brightness gradient perceived in reality. At the same time, when the number of triggers of the target electrode exceeds the third threshold within the preset time window, it automatically reduces the output brightness to simulate the fatigue decay characteristics of nerve tissue and adapt to the real neural adaptation law.

[0156] It should be noted that the cortical and visual field localization module achieves a precise spatial correspondence between electrode stimulation and visual field location, the current diffusion calculation module determines the size of the light spot based on physiological measurement data, and the brightness and threshold conversion module considers individual differences in perception threshold. Together, these three components make the spatial distribution, size, and brightness of photoillusion more closely resemble the real physiological response of the human body, greatly improving the realism of virtual simulation.

[0157] The brightness and threshold conversion module automatically reduces brightness for high-frequency stimulation, realistically simulating the fatigue decay characteristics of nerve tissue. This avoids simulation distortion caused by overstimulation, enabling the virtual patient model to more accurately predict human perception under long-term or high-frequency stimulation and reducing reliance on real implant recipients.

[0158] In one possible implementation, the model parameters of the virtual patient model are fine-tuned using model training data, which includes training clinical threshold parameters and psychophysical test output files of the patient implant. The psychophysical test output files include stimulation thresholds, implanted electrode sites, spatial visual field mapping areas, stimulation intensity, and photic parameters.

[0159] Understandably, the clinical threshold parameters and psychophysical test output files are derived from real patients' clinical data, which can provide the model with a "calibration benchmark" that closely reflects the actual physiological response. This allows the fine-tuned model parameters to more accurately reflect the real "stimulus-perception" mapping pattern, greatly improving the realism and reliability of virtual simulation.

[0160] Secondly, the individual-specific information contained in the psychophysical test output files (such as the correspondence between stimulus intensity and photopsychic parameters for different patients) allows the model to specifically simulate individual differences, avoid the prediction bias of the "general model" for special cases, and enhance the model's adaptability to different implantees.

[0161] Finally, by fine-tuning the model using real clinical data, we can reduce our reliance on scarce implant recipients and eliminate the need for frequent trial-and-error involving real patients. The model can then simulate photopsia responses in various clinical scenarios on a computer, which reduces the burden of research and development on patients' bodies and accelerates the iteration efficiency of visual reconstruction strategies.

[0162] It should be noted that this technical solution achieves reliable and trainable optical illusion vision for blind users within a limited stimulus channel through a four-level link of "multi-source sensing - bandwidth adaptation - safe output - feedback calibration". At the same time, it supports rapid replacement of sensors or upgrading of computing modules to adapt to various application scenarios such as indoor navigation, stair recognition, and text reading.

[0163] In one possible implementation, the system further includes a stimulation interface unit connected to the display and feedback unit;

[0164] The stimulation interface unit is used to acquire the instantaneous current and cumulative charge of the electrode of the patient's implant in real time, and automatically perform safety operations when the instantaneous current exceeds a first threshold or the cumulative charge exceeds a second threshold.

[0165] Optionally, if amplitude modulation is limited, pure frequency modulation can be used instead; if channel switching takes a long time, a fixed subset of electrodes can be lit sequentially in a loop.

[0166] In one possible implementation, the stimulation interface unit includes:

[0167] The pulse limit module is used to set the pulse width limit and peak current limit of the electrode based on clinical safety data.

[0168] The charge counting module is used to continuously count the accumulated charge using a hardware counter.

[0169] The over-limit processing module is used to reduce the pulse frequency and decrease the brightness of the electrode when the instantaneous current exceeds a first threshold or the cumulative charge exceeds a second threshold. If the adjusted instantaneous current still exceeds the first threshold or the adjusted cumulative charge still exceeds the second threshold, the corresponding electrode is temporarily turned off.

[0170] Specifically, the pulse limit module pre-sets the pulse width and peak current upper limit of the electrode based on clinical safety data to ensure that the basic stimulation parameters meet the human safety threshold; the charge counting module continuously counts the cumulative charge output by the electrode through a hardware counter to avoid tissue heating or electrolysis caused by long-term stimulation.

[0171] The over-limit handling module adopts a hierarchical protection strategy: when the instantaneous current exceeds the first threshold or the cumulative charge exceeds the second threshold, the stimulation intensity is reduced by decreasing the pulse frequency and reducing the electrode brightness; if the limit is still exceeded after adjustment, the corresponding electrode is temporarily turned off and automatically turned on again after the parameters return to the safe range, forming a multi-level safety protection mechanism.

[0172] Understandably, the pulse limit module sets basic parameters based on clinical safety data, the charge counting module monitors the accumulated charge in real time, and the over-limit handling module responds quickly to abnormalities through graded operations. Together, they ensure that the stimulation intensity is always within the physiological safety range, avoiding the risk of tissue damage or overheating, and complying with medical device safety standards.

[0173] It should be noted that the embodiments of this application, through the interface stimulation module, can monitor the single pulse peak, charge accumulation and temperature rise in real time, which complies with relevant medical device registration standards and human safety standards; if any indicator exceeds the threshold, it will immediately take protective measures such as amplitude reduction, frame skipping or shutdown, and store and report relevant error files and contents.

[0174] like Figures 5 to 8 The diagram shown is a user interface schematic of the control system for visual prostheses provided in an embodiment of this application.

[0175] like Figure 5 The image shown is a schematic diagram of the "Display and Feedback Unit" interface of this system: the central black window uses white dots to simulate the visual reconstruction effect of a blind user in a virtual corridor (projected onto the lens), and the text below indicates the current task "Walk along the corridor". At the bottom of the interface are four gray buttons – "Clear", "Blurred", "Too Bright", and "Too Dark" – allowing subjects to quickly evaluate the dot matrix quality; at the very bottom are a detailed feedback input box and a voice input button, used to submit more specific subjective experience information, facilitating the system backend to adjust stimulus parameters accordingly.

[0176] like Figure 6The diagram illustrates the collaborative operation of the multimodal data acquisition interface and module links: Researchers can select any combination of modalities, such as color video, depth map, near-infrared, and ultrasound images, and trigger data recording. The selected modalities are synchronously acquired by the multi-source sensing unit 101 and then sent in real-time to the computational processing unit 102 for denoising, registration, and temporal alignment, subsequently outputting a unified frame rate and quality assessment results. By comparing the dot matrix restoration effects and frame rate performance of different combinations, the system can quantify the marginal contribution of each modality to the visual reconstruction quality. This provides experimental evidence for the subsequent parameter initialization of the virtual patient model 103, the selection of compression strategies for the bandwidth adaptive coding unit 104, and the optimization of light spot mapping in the display and feedback unit 105, thereby accelerating the parameter tuning iteration of the multimodal fusion algorithm and shortening the overall development cycle.

[0177] like Figure 7 The image shown is the real-time debugging page of the "Visual Prosthesis R&D Platform," which intuitively connects the entire processing chain of multi-modal fusion, saliency analysis, and bandwidth adaptive coding. The upper half displays the image fused by the computational processing unit 102, highlighting key areas calculated by the virtual patient model 103 with semi-transparent hot zones; the lower half renders the optical illusion dot matrix output by the bandwidth adaptive coding unit 104 in real time, providing instantaneous rates such as "320 / 400" pulses / frames. The three sets of sliders at the bottom of the interface allow developers to dynamically adjust the dot matrix bandwidth limit, brightness attenuation coefficient, and saliency threshold during operation. Coupled with "Reset Parameters" and "Instant Preview" buttons, it allows for quick verification of the impact of different settings on the visual reconstruction effect, providing visual and quantitative evidence for the iteration of multi-modal fusion algorithms and coding strategies.

[0178] like Figure 8 The diagram shows the interface of the "Model Configuration and Safety Monitoring Interface" provided in this application: The parameter adjustment function of the virtual patient model 103 and the stimulation safety monitoring function are integrated into the same operation page. The upper part of the interface has a continuous slider, which can adjust key physiological parameters such as "visual field-cortical mapping coefficient" and "neural conduction delay" within a quantization range of 0-10. After each parameter change, the system instantly refreshes the threshold response curve below, intuitively presenting the subjective comfort range corresponding to different stimulation amplitudes. The lower part of the interface displays two safety monitoring curves in real time: one is the peak value change curve of the stimulation current (current example 0.7mA), and the other is the percentage curve of the accumulated charge over 60 seconds relative to the safety upper limit (current example 85%). When either indicator exceeds the safety threshold, the "Safety Event Log" at the bottom automatically records the trigger time and system downgrade information, realizing visualized and closed-loop management of parameter adjustment and power safety.

[0179] In one possible implementation, the system uses detachable clips, allowing sensors and display components to be replaced independently; moreover, the computing unit has a high-speed interface that can be plugged into a single-board computer, a micro graphics accelerator card, or a low-power processor, allowing for a flexible balance between performance and power consumption depending on the scenario.

[0180] The following sections introduce different technical approaches to achieve the invention's objectives. Alternative methods to the complete technical solution described in this section all revolve around the core technical problem that this invention aims to solve, and are explained in detail below:

[0181] Offline dictionary / retrieval solution: This solution collects samples in common scenarios in advance and establishes an offline dictionary with a "scene-light illusion viewpoint" correspondence. This allows the system to directly match the closest template and make fine adjustments during actual use, greatly reducing the need for real-time calculations on site.

[0182] Vision-Touch-Audio Multi-Channel Solution: This solution reduces the amount of data transmitted through the visual link by allocating key information such as stair edges and obstacles to the vibration belt / patches or auditory channels, while retaining only core information such as directional indications in the visual channel.

[0183] Ultralight Waveguide AR Solution: This solution uses waveguide lenses to directly overlay optical illusions onto the real field of vision, replacing traditional head-mounted displays. While optimizing the portability of the display device, it maintains the core processes such as multi-source perception, data compression, and stimulus output.

[0184] Edge-cloud collaborative solution: This solution has the head-mounted device complete the basic processing such as initial downsampling and security rate limiting, while the complex tasks such as large-block text recognition and database matching are handled by the mobile phone or cloud. After processing in the cloud, the compressed optical illusion viewpoint is sent back.

[0185] Analytical Virtual Patient Model Solution: This solution uses finite element analysis combined with electric field threshold formula to quickly estimate the distribution of light illusion viewpoints without the need for large-scale data fitting, significantly reducing the demand for on-site computing power.

[0186] The aforementioned alternatives cover multiple dimensions, including data processing modes, information transmission channels, device form factors, and computing architectures. They are suitable for both lightweight laboratory validation scenarios and can be seamlessly adapted to clinical implantation needs.

[0187] The above are some embodiments of a control system for visual prostheses provided in this application. Based on this, this application also provides corresponding methods. See also Figure 9 This is a flowchart illustrating a control method for a visual prosthesis provided in an embodiment of this application. The method includes:

[0188] Step S101: Collect environmental information using multiple sensors.

[0189] Step S102: After fusing the environmental information, output the corresponding saliency map.

[0190] Step S103: Score each pixel of the saliency map according to the scene where the virtual patient is located, and determine N key light points according to the preset upper limit of the number of safety pulses; the N key light points are the light points that match the projection of the saliency map onto the cortical topological optical illusion dot matrix, and N is an integer greater than 1.

[0191] Step S104: Filter the N key light spots to obtain the filtered key light spots.

[0192] Step S105: Project the selected key light spots onto the lens and receive feedback information on the selected key light spots.

[0193] Step S106: Automatically adjust the brightness threshold and selection rules based on the feedback information.

[0194] It should be noted that the specific implementation of the control method for visual prostheses provided in this application embodiment can refer to the foregoing embodiments, and will not be repeated here.

[0195] As can be seen from the above technical solutions, the embodiments of this application utilize virtual patient models to replace scarce real implant subjects for large-scale trials, reducing reliance on implant subjects and eliminating the need for frequent use of implant subjects. The virtual patient model prioritizes the transmission of key information within limited bandwidth, improving the clarity and efficiency of visual reconstruction. Furthermore, based on feedback from users, the system collects feedback and safety thresholds in real time, automatically adjusting the light spot for the next round, forming a closed loop of "stimulus-perception-adjustment," which can effectively improve the efficiency and effectiveness of visual prostheses in research and development.

[0196] Optionally, the step of scoring each pixel of the saliency map based on the scene in which the virtual patient is located, and determining N key light points according to a preset upper limit for the number of safety pulses, includes:

[0197] Each pixel of the saliency map is scored according to the scene in which the virtual patient is located, to obtain a scoring result, which is used to indicate the importance of the pixel;

[0198] Read the number of pulses that the virtual brain electrodes can safely output to determine the upper limit of the safe number of pulses in the current frame;

[0199] Based on the upper limit of the number of safety pulses and the scoring results, the multiple pixels of the saliency map are sorted from high to low scores. The saliency map after pixel sorting is projected onto the cortical topological optical illusion matrix to determine the N matching key light points.

[0200] By using a pre-defined correspondence between light spots and electrode numbers, the coordinates of the N key light spots are converted into electrode numbers for the cerebral cortex.

[0201] Optionally, the step of filtering the N key light spots to obtain the filtered key light spots includes:

[0202] Based on the effective threshold and safety threshold of visual cortical electrical stimulation, the N key light spots are bandpass filtered to obtain the filtered key light spots. The effective threshold of visual cortical electrical stimulation is the minimum value of the electrode stimulation intensity that can produce photic hallucinations, and the safety threshold is the minimum value between the value of patient-perceived discomfort and the value of physiological tissue safety.

[0203] Optionally, the method further includes:

[0204] The system acquires the instantaneous current and cumulative charge of the patient's implanted electrode in real time, and automatically performs safety operations when the instantaneous current exceeds a first threshold or the cumulative charge exceeds a second threshold.

[0205] Optionally, the method further includes:

[0206] Based on clinical safety data, set pulse width limits and peak current limits for the electrodes;

[0207] The accumulated charge is continuously counted using a hardware counter;

[0208] When the instantaneous current exceeds a first threshold or the accumulated charge exceeds a second threshold, the pulse frequency is reduced and the brightness of the electrode is decreased. If the adjusted instantaneous current still exceeds the first threshold or the adjusted accumulated charge still exceeds the second threshold, the corresponding electrode is temporarily turned off.

[0209] Optionally, the method further includes:

[0210] Based on the preset projection relationship between the retina and the cortex and the coordinates of the electrode array, the planar coordinates of each electrode in the cerebral cortex are converted into the corresponding visual field angle position.

[0211] Based on the pulse amplitude and pulse width of each electrode, combined with the empirical constant of tissue conductivity, the radial attenuation range of the stimulation current in the cortical tissue is calculated; and based on the radial attenuation range, the diameter parameter of the light spot at the visual field angle position is obtained.

[0212] The peak current of the electrode is mapped to the brightness of the center circle, and a two-dimensional Gaussian distribution of the brightness of the spot is generated using the brightness of the center circle as the amplitude and the diameter parameter as the size of the spot. The brightness of the spot decreases radially from the center to the periphery. When the number of times the target electrode is triggered exceeds the third threshold within a preset time window, the output brightness of the target electrode is automatically reduced.

[0213] Optionally, the method further includes:

[0214] The model parameters of the virtual patient model are fine-tuned using model training data, which includes training clinical threshold parameters and psychophysical test output files of the patient implant. The psychophysical test output files include stimulation threshold, implanted electrode site, spatial visual field mapping area, stimulation intensity, and photic parameters.

[0215] Optionally, the method further includes:

[0216] The sensor includes at least one of a color camera, a depth camera, a near-infrared camera, an ultrasonic probe, or a millimeter-wave probe, and the sensor is synchronized via a synchronization bus or a common clock.

[0217] Next, this application will be described in detail using a specific scenario example.

[0218] S1. Simultaneously acquire environmental images and distance information using multiple sensors;

[0219] S2. Align the images captured by all sensors at the same time.

[0220] S3. Integrate multi-modal information to identify key areas and synthesize a significant heatmap;

[0221] S4. Determine the number of currently usable and safe electrodes and the maximum number of light spots that can be lit in this frame.

[0222] S5. Select pixels from the heatmap according to their brightness until the number allowed by the effective bandwidth is reached;

[0223] S6. Convert the selected pixels into cortical coordinates using the polar-logarithmic formula and determine the corresponding electrodes;

[0224] S7. Estimate the total charge and peak current of the pulse. If they exceed the limit, reduce the brightness of the peripheral light spot.

[0225] S8. Project the light spot onto the display device or send it to the electrode array;

[0226] S9. Record the user's behavioral data and evaluation information;

[0227] S10. Input the feedback information into the virtual patient model, automatically adjust the brightness threshold and spot selection rules for processing in the next frame.

[0228] It should be noted that by linking electrode stimulation, cortical conduction, and photopic sensations into a calculable chain, researchers first experimented with thousands of parameters on a computer, quickly converging to a reasonable pulse intensity and location. Then, they fine-tuned the process using a small amount of clinical data, taking up almost no time for the implant patient, making it both safe and time-efficient.

[0229] Secondly, the external stimulator is equipped with bandwidth adaptive coding. The system first calculates the total amount of pulses that the implanted electrodes can withstand, then selects the most critical edges, steps, or text outlines, and distributes the light spots according to the cortical pattern of "dense in the center and sparse at the periphery." Under the same bit limit, users can still distinguish the key points, fundamentally solving the bandwidth bottleneck.

[0230] Furthermore, the device supports pluggable sensors such as color, depth, near-infrared, and ultrasound, along with a low-power processing pipeline: it can still stably identify targets in low light, glass surfaces, or strong backlight environments, while the headband remains lightweight and has a long battery life.

[0231] The external stimulator can be switched to a simulation display mode—projecting the same array of light dots onto a VR or lightweight display screen, allowing people with normal vision to experience "dot matrix vision" safely and without implants. Researchers can then collect a large amount of behavioral data (walking, reading, obstacle avoidance, etc.) to further refine navigation, reading, and facial recognition algorithms. Once the results are mature, the technology can be transferred to real implant recipients, greatly shortening the research and development trial-and-error time.

[0232] In summary, this application takes into account security, information efficiency, environmental adaptability, and scalable testing channels, providing a complete solution for visual reconstruction in the blind, from rapid validation in healthy volunteers to clinical application.

[0233] This application also provides an electronic device, including: a memory for storing one or more programs;

[0234] Processor; when the processor executes the one or more programs, it implements the control method for visual prostheses in the above embodiments.

[0235] This application also provides a computer storage medium storing a program that, when executed by a processor, implements the control method for visual prostheses described in the above embodiments.

[0236] In the embodiments of this application, the terms "first" and "second" (if they exist) are used only as name identifiers and do not represent the order of first and second.

[0237] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0238] Those skilled in the art will understand that the flowchart shown is merely an example in which the embodiments of this application can be implemented, and the scope of application of the embodiments of this application is not limited by any aspect of the flowchart.

[0239] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A control system for a visual prosthesis, characterized in that, The system includes a multi-source sensing unit, a computing processing unit, a virtual patient model, a bandwidth adaptive coding unit, and a display and feedback unit; wherein, the multi-source sensing unit is connected to the computing processing unit; the virtual patient model is connected to the computing processing unit, the bandwidth adaptive coding unit, and the display and feedback unit respectively; the bandwidth adaptive coding unit is connected to the display and feedback unit. The multi-source sensing unit is used to collect environmental information through multiple sensors. The computational processing unit is used to fuse the environmental information and output the corresponding saliency map; The virtual patient model is used to score each pixel of the saliency map according to the scene in which the virtual patient is located, and to determine N key light points according to a preset upper limit for the number of safety pulses; the N key light points are the light points that match the projection of the saliency map onto the cortical topological optical illusion dot matrix, and N is an integer greater than 1; The bandwidth adaptive coding unit is used to filter the N key light spots to obtain the filtered key light spots; The display and feedback unit is used to project the selected key light points onto the lens and receive feedback information for the selected key light points. The virtual patient model is also used to automatically adjust the brightness threshold and spot selection rules based on the feedback information.

2. The system according to claim 1, characterized in that, The virtual patient model includes: A saliency calculation module is used to score each pixel of the saliency map according to the scene in which the virtual patient is located, so as to obtain a scoring result, which is used to indicate the importance of the pixel; The bit width management module is used to read the number of pulses that the virtual brain electrodes can safely output to determine the upper limit of the number of safe pulses in the current frame; The sparse selection module is used to sort multiple pixels of the saliency map from high to low according to the upper limit of the number of safety pulses and the scoring result, and project the sorted saliency map onto the cortical topological optical illusion matrix to determine the matching N key light points. The coordinate remapping module is used to convert the coordinates of the N key light points into electrode numbers of the cerebral cortex by using a pre-set correspondence between light points and electrode numbers.

3. The system according to claim 1, characterized in that, The bandwidth adaptive coding unit is specifically used to perform bandpass filtering on the N key light spots according to the effective threshold and safety threshold of the visual cortex electrical stimulation to obtain the filtered key light spots. The effective threshold of the visual cortex electrical stimulation is the minimum value of the stimulation intensity of the electrode that can satisfy the generation of photic hallucinations, and the safety threshold is the minimum value between the value of patient-perceived discomfort and the value of physiological tissue safety.

4. The system according to claim 1, characterized in that, The system also includes a stimulation interface unit, which is connected to the display and feedback unit; The stimulation interface unit is used to acquire the instantaneous current and cumulative charge of the electrode of the patient's implant in real time, and automatically perform safety operations when the instantaneous current exceeds a first threshold or the cumulative charge exceeds a second threshold.

5. The system according to claim 4, characterized in that, The stimulation interface unit includes: The pulse limit module is used to set the pulse width limit and peak current limit of the electrode based on clinical safety data. The charge counting module is used to continuously count the accumulated charge using a hardware counter. The over-limit processing module is used to reduce the pulse frequency and decrease the brightness of the electrode when the instantaneous current exceeds a first threshold or the cumulative charge exceeds a second threshold. If the adjusted instantaneous current still exceeds the first threshold or the adjusted cumulative charge still exceeds the second threshold, the corresponding electrode is temporarily turned off.

6. The system according to claim 1, characterized in that, The virtual patient model also includes: The cortical and visual field localization module is used to convert the planar coordinates of each electrode in the cerebral cortex into the corresponding visual field angle position based on the preset projection relationship between the retina and the cortex and the coordinates of the electrode array. The current diffusion calculation module is used to calculate the radial attenuation range of the stimulation current in the cortical tissue based on the pulse amplitude and pulse width of each electrode, combined with the empirical constant of tissue conductivity; and to obtain the diameter parameter of the light spot at the visual field angle position based on the radial attenuation range. The brightness and threshold conversion module is used to map the peak current of the electrode to the brightness of the center circle, and generate a two-dimensional Gaussian distribution of the brightness of the light spot using the brightness of the center circle as the amplitude and the diameter parameter as the size of the light spot. The brightness of the light spot decreases radially from the center to the periphery. When the number of times the target electrode is triggered exceeds the third threshold within a preset time window, the output brightness of the target electrode is automatically reduced.

7. The system according to claim 1, characterized in that, The model parameters of the virtual patient model are fine-tuned using model training data, which includes training clinical threshold parameters and psychophysical test output files for the patient implant. The psychophysical test output files include stimulation thresholds, implanted electrode sites, spatial visual field mapping areas, stimulation intensity, and photopsychotropia parameters.

8. The system according to claim 1, characterized in that, The sensor includes at least one of a color camera, a depth camera, a near-infrared camera, an ultrasonic probe, or a millimeter-wave probe, and the sensor is synchronized via a synchronization bus or a common clock.

9. A control method for visual prostheses, characterized in that, Applied to the control system as described in claims 1-8, the method comprises: Environmental information is collected using multiple sensors. After fusing the environmental information, the corresponding saliency map is output. Each pixel of the saliency map is scored according to the scene in which the virtual patient is located, and N key light points are determined according to the preset upper limit of the number of safety pulses; the N key light points are the light points that match the projection of the saliency map onto the cortical topological optical illusion dot matrix, and N is an integer greater than 1; The N key light spots are filtered to obtain the filtered key light spots; The selected key light spots are projected onto the lens, and feedback information on the selected key light spots is received. The brightness threshold and selection rules are automatically adjusted based on the feedback information.

10. The method according to claim 9, characterized in that, The method further includes: The system acquires the instantaneous current and cumulative charge of the patient's implanted electrode in real time, and automatically performs safety operations when the instantaneous current exceeds a first threshold or the cumulative charge exceeds a second threshold.

11. An electronic device, characterized in that, The electronic device includes: Memory, used to store one or more programs; A processor; when the processor executes the one or more programs, it implements the control method for a visual prosthesis as described in any one of claims 9-10.

12. A computer storage medium, characterized in that, The computer storage medium stores a program that, when executed by a processor, implements the control method for visual prostheses as described in any one of claims 9-10.

Images