Virtual interaction training method and device of visual reconstruction brain-computer interface

Abstract

The application discloses a kind of virtual interaction training method and device of visual reconstruction brain-computer interface, the method comprises: generating virtual reality scene, and extracting the importance representation relevant to current task in virtual reality scene;Extract the key visual information therein, and be mapped to the spatial position of the cortical electrode array by the preset visual field-cortical topological mapping model;Based on the individualized perception function constraint obtained in advance, the key visual information after mapping is converted into the space-time stimulation parameter of cortical electrode;Collect the behavior data of user under space-time stimulation parameter execution task, to calculate behavior error comprehensive score;According to behavior error comprehensive score, the difficulty parameter of task is dynamically adjusted, to complete closed-loop adaptive training.The problems of single training scene, subjective parameter adjustment, system closed-loop loss and insufficient evaluation standard are solved, and personalized, quantitative evaluation and safe and efficient of visual reconstruction brain-computer interface postoperative rehabilitation training are realized.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of visual reconstruction training, in particular to a virtual interaction training method and device of a visual reconstruction brain-computer interface. BACKGROUND

[0002] As a frontier in the intersection of artificial vision and rehabilitation medicine, the visual reconstruction brain-computer interface technology aims to help patients with visual impairment recover part of the visual function by generating light hallucination points in the blind field through cortical electrical stimulation. At present, although the visual prosthesis device (including retinal, optic nerve or cortical stimulation) can generate light hallucination points, patients still need a long time of training after surgery to learn to read the visual information represented by these light points. In the prior art, the postoperative rehabilitation training of visual reconstruction mainly relies on static images or limited physical environments, and the training scene is single and lacks diversity, which is difficult to cover various perception task types required in daily life such as navigation, reading and object recognition. More importantly, most of the existing training systems adopt an open-loop operation mode, and the adjustment of stimulation parameters mainly depends on the subjective experience and offline adjustment of doctors, which cannot be dynamically optimized according to the real-time behavior performance and subjective experience of patients, resulting in long parameter adjustment period, low efficiency, and difficulty in matching the individual needs of each patient due to differences in electrode implantation position, perception sensitivity and cognitive ability. In addition, the existing technology lacks a scientific and quantifiable evaluation system, and the rehabilitation effect mainly relies on empirical judgment, which cannot provide accurate task scores and phased indicators, making it difficult to form an individualized rehabilitation path and long-term tracking effect. In terms of training safety, traditional methods often need to conduct complex scene training in real environments, which not only has safety hazards, but also has high cost and is difficult to reproduce various task types in daily life. The technical solution closest to the present application is to map a preset pattern template to a cortical electrode array, and then perform offline parameter adjustment based on the feedback of the subject, but this method has obvious deficiencies: on the one hand, it lacks real-time closed-loop feedback mechanism, and the adjustment of stimulation parameters cannot quickly and accurately correspond to the behavior results of the patient; on the other hand, it is difficult to complete effective training on complex scenes under safe and low-cost conditions. Therefore, there is an urgent need for a real-time closed-loop training method that can organically combine visual tasks in a virtual environment with cortical electrical stimulation output, support the whole process of information selection, individual adaptation, behavior feedback collection and parameter self-adjustment. SUMMARY

[0003] Therefore, the present application provides a virtual interaction training method and device of a visual reconstruction brain-computer interface, which can realize individualized parameter adaptation, quantitative effect evaluation and efficient training of the visual reconstruction brain-computer interface. The present application provides the following technical solutions: a virtual interaction training method of a visual reconstruction brain-computer interface, the method comprising:

[0004] generating a virtual reality scene and extracting an importance representation related to a current task in the virtual reality scene;

[0005] extracting key visual information in the importance representation and mapping the key visual information to spatial locations of the cortical electrode array;

[0006] converting the mapped key visual information into spatiotemporal stimulation parameters of the cortical electrode based on pre-acquired individualized perceptual function constraints;

[0007] collecting behavioral data of the user performing the task under the spatiotemporal stimulation parameters and comparing the behavioral data with preset baseline data to calculate a comprehensive behavioral error score;

[0008] adjusting difficulty parameters of the task dynamically according to the comprehensive behavioral error score to complete a closed-loop adaptive training.

[0009] Optionally, the dynamically generating a virtual reality scene and extracting an importance representation related to a current task in the virtual reality scene comprises:

[0010] selecting a target scene type based on a preset scene label set;

[0011] randomly and dynamically combining model components according to the target scene type to generate virtual reality scenes with different layouts and task elements;

[0012] determining a current task type according to the target scene type;

[0013] extracting visual features related to the task type in the virtual reality scene based on the task type using an image processing algorithm to generate an importance representation.

[0014] Optionally, the extracting key visual information in the importance representation and mapping the key visual information to spatial locations of the cortical electrode array comprises:

[0015] determining a position of a visual topology center in the importance representation;

[0016] loading a preset standard visual field-cortical topology relationship matrix for representing a correspondence between image coordinates and cortical coordinates, and performing individualized correction on the standard visual field-cortical topology relationship matrix according to electrode implantation position information of the patient to generate an individualized visual field-cortical topology mapping model;

[0017] mapping the key visual information in the importance representation to spatial locations of the cortical electrode array based on the position of the visual topology center and the individualized visual field-cortical topology mapping model through coordinate transformation, wherein the coordinate transformation takes the visual topology center as a reference origin.

[0018] Optionally, the converting the mapped key visual information into the spatio-temporal stimulation parameters of the cortical electrodes based on the pre-acquired individualized perceptual function constraint comprises:

[0019] acquiring an individualized perceptual function for representing a correspondence between the electrode stimulation parameters and the phosphene perceptual parameters of the patient; and converting the visual feature parameters of the mapped key visual information into the spatio-temporal stimulation parameters of the cortical electrodes according to the individualized perceptual function;

[0020] applying a safety constraint to the converted spatio-temporal stimulation parameters.

[0021] Optionally, the acquiring the behavior data of the user performing the task under the spatio-temporal stimulation parameters and comparing the behavior data with the preset reference data to calculate a behavior error comprehensive score comprises:

[0022] acquiring the behavior data of the user performing the task under the spatio-temporal stimulation parameters, the behavior data comprising: a task result indicator, a completion degree indicator, a time consumption ratio indicator, an operation precision indicator, and a subjective comfort indicator;

[0023] calculating the mean and the standard deviation of each indicator in the behavior data corresponding to the data set based on a pre-established normal population behavior data set to form the reference data;

[0024] performing standardization on the task result indicator, the completion degree indicator, the time consumption ratio indicator, and the operation precision indicator in the behavior data of the user to generate a first set of standardized error components;

[0025] performing reverse standardization processing on the subjective comfort indicator to generate a second standardized error component;

[0026] combining the first set of standardized error components and the second standardized error component into a behavior error vector;

[0027] performing weighted calculation on the behavior error vector according to a preset weight coefficient to obtain a behavior error comprehensive score of the difference between the current performance and the target performance of the user.

[0028] Optionally, the dynamically adjusting the difficulty parameter of the task according to the behavior error comprehensive score to form a closed-loop adaptive training comprises:

[0029] setting an initial threshold range according to the behavior scores of the normal population;

[0030] performing comparison judgment of the behavior error comprehensive score and the subjective comfort indicator with the initial threshold range, and adjusting the training task difficulty parameter according to the comparison judgment result;

[0031] performing smooth updating on the adjusted complexity parameter to complete the closed-loop adaptive training.

[0032] The application further discloses a virtual interaction training device of a visual reconstruction brain-computer interface, which comprises:

[0033] a virtual scene generation and processing module, which is used for dynamically generating a virtual reality scene and extracting an importance representation related to a current task in the virtual reality scene;

[0034] a visual information mapping module, which is used for extracting key visual information in the importance representation and mapping the key visual information to spatial positions of a cortical electrode array;

[0035] a stimulation parameter conversion module, which is used for converting the mapped key visual information into spatiotemporal stimulation parameters of the cortical electrode based on a pre-acquired individualized perceptual function constraint;

[0036] a behavior data acquisition and analysis module, which is used for acquiring behavior data of a user performing a task under the spatiotemporal stimulation parameters and comparing the behavior data with preset reference data to calculate a behavior error comprehensive score;

[0037] a closed-loop adjustment module, which is used for dynamically adjusting a difficulty parameter of the task according to the behavior error comprehensive score to complete a closed-loop adaptive training.

[0038] Optionally, the virtual scene generation and processing module is further used for:

[0039] selecting a target scene type based on a preset scene label set;

[0040] randomly and dynamically combining model components according to the target scene type to generate virtual reality scenes with different layouts and task elements;

[0041] determining a current task type according to the target scene type;

[0042] based on the task type, extracting visual features related to the task type in the virtual reality scene by using an image processing algorithm to generate an importance representation.

[0043] The application further discloses a computer readable storage medium, wherein the storage medium stores a computer program, and the computer program is executed by a processor to realize the method.

[0044] The application further discloses an electronic device, which comprises a memory, a processor and a computer program stored in the memory and capable of running on the processor, and the processor realizes the method when executing the program.

[0045] According to the technical scheme of the present application, by dynamically generating a virtual reality scene and extracting an importance representation related to the current task, high simulation of multiple scenes is realized, overcoming the limitations of traditional training relying only on static images or physical environments. Further, key visual information in the importance representation is extracted and mapped to the spatial position of the cortical electrode array, and the conversion is combined with the individualized perception function constraint to convert into space-time stimulation parameters, so that the stimulation parameter setting is free from the limitations of the subjective experience of doctors, and can accurately match the electrode implantation position, perception sensitivity and cognitive ability differences of each patient. Finally, by collecting user behavior data and comparing with the benchmark data to calculate the behavior error comprehensive score, and dynamically adjusting the task difficulty parameters according to the score to form a closed-loop adaptive training, a complete feedback loop of behavior data collection, error vector calculation, parameter dynamic adjustment and effect verification is constructed, so that the system can automatically optimize the stimulation strategy according to the real-time performance of the patient, solving the problem of long parameter adjustment cycle and low efficiency caused by open-loop operation in the prior art. BRIEF DESCRIPTION OF DRAWINGS

[0046] For the purpose of illustration and not limitation, the present application will now be described in conjunction with embodiments thereof and the accompanying drawings, in which:

[0047] Figure 1 is a flowchart of a virtual interactive training method of a visual reconstruction brain-computer interface in an embodiment of the present application;

[0048] Figure 2 is a structural schematic diagram of a virtual interactive training system of a visual reconstruction brain-computer interface in an embodiment of the present application;

[0049] Figure 3 is a structural schematic diagram of an electronic device in an embodiment of the present application;

[0050] Figure 4 is a schematic diagram of image light phasor array conversion in an embodiment of the present application;

[0051] Figure 5 is a schematic diagram of virtual scene generation selection and key visual feature extraction in an embodiment of the present application;

[0052] Figure 6 is another schematic diagram of virtual scene generation selection and key visual feature extraction in an embodiment of the present application. DETAILED DESCRIPTION

[0053] In order for those skilled in the art to better understand the technical scheme of the present application, the technical scheme in the embodiments of the present application will be described clearly and completely in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor should be within the scope of protection of the present application.

[0054] It should be noted that the features in the embodiments and the embodiments of the present application can be combined with each other without conflict. The embodiments of the present application will be described in detail below in conjunction with the drawings.

[0055] Reference Figure 1 The present embodiment discloses a virtual interaction training method of visual reconstruction brain-computer interface, which comprises the following steps:

[0056] S100: generating a virtual reality scene and extracting an importance representation related to the current task in the virtual reality scene.

[0057] First, a set of predefined scene labels is obtained, which exemplarily includes bedroom scene, kitchen scene, large shopping mall and traffic intersection and other daily life scenes. Before starting the training, the user is selected to select the target scene type, for example, selecting a large shopping mall as the target scene. The current task type is automatically determined according to the selected scene label, for example, in the large shopping mall scene, the task type includes navigation task, reading task and object recognition task. For the selection of scene label, the combination selection of scene label is supported, for example, the traffic intersection and reading label are selected at the same time, so as to construct a composite scene containing red light recognition and road sign reading. In some embodiments, suitable scene combination can be intelligently recommended according to the user historical training data and the current rehabilitation stage, exemplarily, for the user in the primary training stage, the scene combination with less obstacle and simple task element is preferentially recommended.

[0058] After selecting the scene label, the dynamic scene generation function of Unity engine is called, referring to Figure 5 and Figure 6 , respectively showing an exemplary virtual scene generation selection and key visual feature extraction schematic diagram. Randomly dynamically combining model components generates virtual reality scenes with different layouts and task elements. Specifically, the corresponding scene structure template is called according to the scene label. For example, for the large shopping mall scene, the system will randomly generate shopping mall structures with different numbers of floors, different layouts and different stair combinations. The scene structure generation algorithm adopts a random layout algorithm based on graph theory, which ensures that the generated scene not only conforms to the reality logic but also has diversity. This randomization ensures that the scene layout is different each time the training is performed, avoiding the user completing the task through memory rather than visual reconstruction ability.

[0059] After generating the virtual reality scene, the importance representation related to the current task is extracted. Specifically, first, the current frame image of the Unity real-time rendering is obtained, and the key visual features in the image are extracted according to the corresponding image processing algorithm called according to the current task type to form a task-oriented importance representation. For example, in the navigation task, an edge detection algorithm and transformation are used to extract the boundary contour of the walkable area and obstacles, and the saliency of the obstacles is determined by analyzing the area contrast. For example, in the traffic intersection scene, the importance representation is the boundary of the sidewalk, the position of the traffic light, and the vehicle contour. In the reading task, the importance representation is the identification and text information efficiently extracted by the MSER algorithm and the deep learning-based text detection model, and different text areas are distinguished by the semantic segmentation algorithm. In the object recognition task, the importance representation is the target object accurately identified by calling the deep learning-based target detection algorithm, and the key area of the object is determined by the saliency detection algorithm.

[0060] The extracted key visual features are converted into an importance point array to obtain the importance representation, where the value of each pixel point represents the importance weight of the position in the current task. For example, in the navigation task, a higher weight is given to the edge of the walkable area, a medium weight is given to the obstacle contour, and a lower weight is given to the background area.

[0061] S200: Extract the key visual information in the importance representation and map the key visual information to the spatial position of the cortical electrode array.

[0062] First, the position of the visual topology center in the importance representation obtained in step S100 is determined. For users with detectable eye movement data, including normal vision users and visually impaired users with residual eye movement, real-time head orientation and gaze point information is obtained through a head-mounted eye tracking system, and the accurate coordinate position of the visual topology center in the current frame image is calculated. For completely blind users who cannot provide reliable eye movement data, the default position of the visual topology center is intelligently determined based on the current task type and scene content, for example, in the navigation task, the visual topology center is set to 15% below the center of the scene by default, simulating the viewing habit when a human naturally walks; in the reading task, it is set to the center position of the text area by default.

[0063] After determining the visual topology center, a preset standard visual field-cortical topology model is used to realize the mapping of the key visual information to the cortical electrode coordinates. Specifically, when there is a lack of personalized data of the user, the preset standard visual field-cortical topology model is used for mapping. For the construction of the model, based on the fMRI data of healthy subjects, a hyperbolic mapping relationship between the flattened cortex and the visual field is established:

[0064] x elec= c1 ln(r) cos(0), y elec = c2 ln(r) sin(0), where r is the visual field radius, 0 is the horizontal visual angle, c1 and c2 are normalization coefficients, (x elec , y elec ) are the physical coordinates of the electrode on the flattened cortex. Then the standard model is corrected according to the CT / MRI data of the user's electrode implantation position. Specifically, the standard visual field-cortex topological relationship matrix is loaded, which represents the correspondence between the image coordinates and the cortex coordinates. The standard visual field-cortex topological relationship matrix is personalized according to the individualized electrode implantation position information of the patient. Specifically, the preoperative CT or MRI image data of the patient is imported, and the physical position of the electrode array is aligned with the standard cortical anatomical structure through medical image registration technology. In the registration process, the iterative closest point algorithm is used to accurately match the actual position of the implanted electrode with the standard cortical model, and the three-dimensional coordinates of each electrode in the cortical coordinate system are calculated. Based on these coordinate information, the standard visual field-cortex topological relationship matrix is subjected to affine transformation and local deformation to generate an individualized visual field-cortex topological mapping model. This model can accurately reflect the correspondence between the patient's specific electrode implantation position and the visual field area, and solve the problem of perceptual misplacement caused by differences in electrode implantation position.

[0065] For the mapping of key visual information to cortical electrode coordinates, in one or more embodiments, a lookup table method based on psychophysical tests can also be used. Specifically, after the user implants the electrode, an individualized perception mapping table is established through psychophysical tests. First, different parameter combinations of electrical pulses are applied to each electrode channel, including current amplitude I e , frequency f e , pulse width w e , the user's reported photism perception characteristics are recorded, including position (r, 0), brightness L e and size B e , and the above test results are arranged into a three-dimensional lookup table in the format: LUT(x elec , y elec , I e ) = (r, 0, L e , B e ), where (x elec , y elec ) are the physical coordinates of the electrode on the flattened cortex.

[0066] After generating the individualized visual field-cortex topological mapping model, the mapping process of key visual information is started. First, N most task-relevant pixel points are extracted from the importance representation, where the value of N is determined according to the number of channels of the electrode array. The selection of these pixel points is based on the weight distribution in the importance representation, and the pixel points with a weight value higher than a threshold value are preferentially selected. In the mapping process, the relative coordinates of each key pixel point are converted to cortex coordinates through the individualized visual field-cortex topological mapping model with the visual topological center as the reference origin. Specifically, for the key pixel point (u, v) in the importance representation, the offset Δu = u - u0 and Δv = v - v0 of the key pixel point relative to the visual topological center (u0, v0) are calculated. Subsequently, the image offset is converted to visual field polar coordinates (r, θ) using the visual field-cortex topological mapping model: r = |z|. Wherein, is the image space distance, g, h, j, q are preset model constants.

[0067] An implementation example is given in this embodiment. When the user is in a large shopping mall navigation task, 100 key pixel points in the importance representation are identified, of which 70% are concentrated on the edge of the navigation path, 20% are on the floor indicator, and 10% are on the target store logo. With the visual topological center as the origin, these pixel points are mapped to the cortex electrode array through the individualized visual field-cortex topological mapping model to generate the final electrode stimulation position allocation.

[0068] For the steps of this embodiment, accurate mapping from key visual information in a virtual scene to spatial positions of the cortex electrode is achieved, effectively solving the problem of visual reconstruction accuracy caused by differences in electrode implantation positions and perception misalignment in the prior art. This implementation method based on individualized visual field-cortex topological mapping makes the visual reconstruction more consistent with the actual perception characteristics of the patient, significantly improves the readability of the phosphenes information and the task completion efficiency, and provides support for subsequent stimulation parameter conversion and closed-loop adaptive adjustment.

[0069] S300: Based on the individualized perception function constraint obtained in advance, the mapped key visual information is converted into spatiotemporal stimulation parameters of the cortex electrode.

[0070] First, the pre-acquired individualized perception function is loaded, which is a patient-specific mapping relationship established through preoperative psychophysical testing. In this embodiment, the patient undergoes a series of standardized tests before surgery, and different parameter combinations of electrical stimulation are applied to each electrode channel in turn, and the patient's response to photic hallucination perception is recorded. Specifically, the perception threshold is determined by single electrode stimulation, and then the interference effect between electrodes is tested by multi-electrode combination stimulation, and finally the perception function curve of each electrode channel is formed. The test data is smoothed by spline interpolation method to generate a continuous perception function model, which represents the correspondence between electrode stimulation parameters (current amplitude, stimulation frequency, pulse width) and photic hallucination perception parameters (position, size, brightness). The perception function is stored in the form of parameter threshold table, and each electrode channel corresponds to a stimulation parameter lookup table, which records the brightness and size of the photic hallucination perceived by the patient under different stimulation parameter combinations.

[0071] Reference Figure 4 , an exemplary image phosphen array conversion schematic diagram is shown, after the individualized perception function constraint is obtained, the conversion of key visual information into spatio-temporal stimulation parameters is performed. Specifically, it includes image feature extraction and phosphen array image generation. Among them, for image feature extraction, the key pixel points in the importance representation output in step S200 are first processed by graphics, edge detection and contrast enhancement are performed on the current frame image, and a binary image containing brightness intensity gradient is generated. This image forms a simplified picture similar to a line by retaining high gradient areas and suppressing low gradient backgrounds, where the point brightness intensity at different positions reflects the weight distribution of task relevance. For the generation of the phosphen array image, based on the patient-specific mapping relationship function established by the above psychophysical test, the processed image is matched with the cortical electrode site. Specifically, according to the perception function, the stimulation parameter combination required to achieve the target brightness is determined, and the brightness intensity L image of each point light position in the image is converted into the corresponding electrode stimulation parameters (current amplitude I e , pulse width w e ). Taking current amplitude as an example, query the stimulation parameter lookup table of this electrode channel to find the minimum current amplitude I that can produce the target brightness L. Using the visual cortex topological mapping model, the point light position in the image coordinate system is mapped to the cortical electrode coordinate. For the area where there is a many-to-one mapping, i.e. multiple image points correspond to the same electrode, the weighted average method is used to distribute the stimulation intensity; for many-to-many mapping, the contribution degree of each electrode is balanced through matrix transformation. Further, multiple safety constraints are applied to ensure that the stimulation parameters are within the safe range, and the following safety boundaries are given in this embodiment:

[0072] Single point brightness check: ensure that the stimulation parameters of each electrode do not exceed the maximum safe brightness L max; activation point number check: dynamically adjust the maximum number of activated points according to the current task type, ensure that the number of activated points does not exceed the total number of electrodes N elec ;

[0073] charge accumulation check: track the 60-second cumulative charge in real time, and when the cumulative charge reaches the preset maximum charge C max , trigger the low-power protection mode.

[0074] The present embodiment gives an exemplary training process:

[0075] When the user identifies the traffic light in the traffic intersection scene, 20 key pixel points mapped to the cortical electrode array are identified, of which 12 correspond to the traffic light outline and 8 correspond to the sidewalk boundary. According to the individualized perception function, the stimulation current of the traffic light outline point is set to 85μA, close to 75% of the maximum safe brightness L max of the electrode; the sidewalk boundary point is set to 65μA, while ensuring that the total number of activated points does not exceed 30, and the cumulative charge within 60 seconds is controlled within 70% of the maximum charge C max . When it is detected that the user has failed to correctly identify the traffic light state twice in a row, the stimulation current of the traffic light outline point is automatically increased to 95μA, and a brief highlight prompt is added in the virtual scene to help the user establish the correct perception association.

[0076] Through the implementation of this step, the precise conversion from key visual information to cortical electrode spatiotemporal stimulation parameters is realized, effectively solving the problems of subjectivity, lack of individualization and insufficient safety constraints in the prior art. The stimulation parameter conversion method based on individualized perception function makes the optical phosphenes output more in line with the perception characteristics of the patient, significantly improving the readability of the visual reconstruction information and the task completion efficiency, while ensuring the safety and comfort of the training process, providing a high-quality stimulation basis for subsequent behavior feedback collection and closed-loop adaptive adjustment.

[0077] S400: Collect behavior data of the user performing the task under the spatiotemporal stimulation parameters, and compare the behavior data with the preset reference data to calculate a behavior error comprehensive score.

[0078] For the reference data, first, a plurality of normal vision personnel complete all training tasks in a unified virtual scene library. For each volunteer, five key indicators are recorded in each training scene: task result b (0 for success, 1 for failure), completion degree d (number of completed subtasks ÷ total number of subtasks), time consumption ratio t (actual time consumption ÷ reference time consumption), operation accuracy a (number of correct operations ÷ total number of operations), and subjective comfort c (0-10 points, the higher the score, the more comfortable). Based on the above key indicator data, statistical analysis is performed to calculate the mean μ i and standard deviation σi , forming a normal person baseline library, i.e., reference data.

[0079] After the reference data is established, the behavior data of the user under the spatiotemporal stimulation parameters determined in step S300 is collected. For example, when the blind patient is performing the traffic intersection training task, the following behavior data is monitored and recorded in real time: whether the patient successfully identifies the traffic light state (task result indicator b), whether the patient completes all subtasks of crossing the road (completion degree indicator d), the ratio of the time taken to cross the road to the reference time (time consumption ratio indicator t), whether the patient correctly judges the walkable area (operation accuracy indicator a), and the patient's subjective comfort score for the current stimulation parameters (subjective comfort indicator c).

[0080] After the behavior data is collected, it is compared with the preset reference data to calculate the behavior error vector. First, the task result, completion degree, time consumption ratio, and operation accuracy indicators are subjected to Z-score standardization processing: wherein x i is the behavior data of the above four indicators of the user, μ i is the mean of the reference data of the above four indicators, and σ i is the standard deviation of the reference data of the above four indicators. For the subjective comfort indicator, reverse standardization processing is performed: wherein x c is the subjective comfort indicator data of the user, μ c is the mean of the reference data of the above four indicators, and σ c is the standard deviation of the reference data of the above four indicators. The above five standardized error components are combined to form the behavior error vector: E[|z b |, |z d |, |z t |, |z a |, |z c |], which reflects the degree of deviation of the user from the normal person in each indicator.

[0081] After obtaining the behavior error vector, a weighted calculation is performed on the behavior error vector according to the preset weight coefficients to obtain the behavior error comprehensive score: S = ∑ n ω n |z n |, n ∈ {b, d, t, a, c}, wherein ω n is the preset weight value corresponding to each indicator. The lower the value of the behavior error comprehensive score, the closer the task execution performance of the current user to the normal person level, and the better the subjective experience.

[0082] This implementation method enables objective and quantitative assessment of user behavior, effectively addressing the problem that existing technologies rely primarily on experience-based evaluations and lack quantifiable indicators for rehabilitation effectiveness. The behavior error calculation method based on five-dimensional indicators not only comprehensively reflects the user's training effect but also provides precise feedback signals for closed-loop adaptive adjustment. This allows for automatic optimization of training parameters based on the user's real-time performance, significantly improving the scientific rigor and effectiveness of post-visual reconstruction brain-computer interface rehabilitation training.

[0083] S500: Based on the comprehensive score of the behavioral error, dynamically adjust the difficulty parameters of the task to complete closed-loop adaptive training.

[0084] First, based on the baseline data of normal personnel in step S400, the comprehensive behavioral error score sequence S corresponds to... norm The statistical distribution is defined with its 25th percentile as the lower threshold T. low The 75th percentile is used as the upper threshold T. high In this implementation, starting from the 14th day after the user's surgery, the threshold is automatically updated to the same quantile of the user's comprehensive behavioral error score distribution over the past 7 days, achieving a smooth transition from the group baseline to the individual baseline.

[0085] After determining the threshold corresponding to the user baseline, the training adjustment strategy is executed based on the comparison result between the comprehensive behavioral error score S calculated in step S400 and the threshold. In this embodiment, a continuous judgment mechanism is adopted. Parameter adjustment is triggered only when the comprehensive behavioral error score calculated three consecutive times meets a specific condition, avoiding erroneous adjustments caused by random fluctuations. Specifically, the following judgment rules are adopted:

[0086] When the comprehensive behavioral error score S ≤ T for three consecutive rounds of training low When the comfort index is not lower than 8, the user training for the current task is judged to be excellent, and a strategy to increase the task difficulty parameter is implemented; when the comprehensive behavioral error score S≥T for two consecutive rounds of training. high If the comfort index is not higher than 4, the user training for the current task is determined to be difficult, and a strategy to lower the task difficulty parameter is implemented. For other cases, the user training for the current task is determined to be neutral, and only a small, smooth adjustment is performed on the task difficulty parameter. All parameter adjustments employ a smooth update strategy to avoid perceptual discomfort caused by sudden parameter changes. Specifically: P k+1 =(1-α)P k +αP new , where P k+1 For the updated parameter value, P k P is the current parameter value. new The adjusted target parameter value is α, where α is the parameter smoothing coefficient.

[0087] The embodiment further exemplarily gives the above-mentioned task difficulty parameter, specifically including virtual reality scene complexity, importance representation extraction strategy and space-time collection parameter conversion rule.

[0088] Among them, the virtual reality scene complexity adjustment includes:

[0089] Increase the scene element density: in the navigation task, increase the number of obstacles in the scene;

[0090] Increase the dynamic element motion speed: in the traffic intersection scene, increase the motion speed of pedestrians and vehicles;

[0091] Reduce the size of the text identification: in the reading task, reduce the size of the text identification;

[0092] Reduce the environmental lighting: in the object recognition task, reduce the scene light intensity.

[0093] Importance representation extraction strategy adjustment:

[0094] Reduce the number of key visual information: reduce the number of extracted importance points;

[0095] Increase the feature extraction threshold: increase the threshold of edge detection and saliency detection, and only keep the most significant features;

[0096] Reduce the task-related area: reduce the range of walkable area in the navigation task.

[0097] Adjust the space-time stimulation parameter conversion rule:

[0098] Reduce the stimulation density: reduce the number of electrodes activated at the same time;

[0099] Reduce the stimulation intensity: reduce the stimulation current amplitude;

[0100] Shorten the stimulation duration: reduce the stimulation pulse width;

[0101] Increase the stimulation interval: increase the stimulation interval time.

[0102] Through the specific implementation of the present step, a complete closed-loop adaptive training mechanism is realized, effectively solving the problems of "subjective parameter adjustment" and "lack of system closed loop" in the prior art. The dynamic adjustment method of the behavior error comprehensive score can not only automatically optimize the training parameters according to the real-time performance of the user, but also improve the user's training ability limit under the premise of ensuring safety, significantly improving the efficiency and effect of visual reconstruction brain-computer interface postoperative rehabilitation training. At the same time, the parameter smoothing update and safety boundary mechanism ensures the comfort and sustainability of the training process, enabling users to gradually improve their visual reconstruction ability in a safe and comfortable environment, and ultimately achieving precise training of the user's visual reconstruction.

[0103] S600: generating a comprehensive performance report based on the behavior error comprehensive score of the current task stage.

[0104] According to the scene type and characteristics of the training task, the training task is divided into multiple categories, including, for example, indoor navigation tasks, reading function tasks, and interpersonal interaction tasks. For each type of task, a special evaluation dimension and scoring standard are established to ensure that the report content is highly relevant to the task characteristics. Specifically, during the training process, the current task is automatically identified according to the scene category, and the behavior data is classified according to the category. For example, when the user completes a large shopping mall navigation task, it is classified as an indoor navigation task; when the user completes a traffic intersection traffic light recognition task, it is classified as a reading function task. After completing each task, the behavior error comprehensive score of the task is calculated based on step S500, and the score is stored in the database corresponding to the scene category together with the historical data. For each scene category, a targeted comprehensive performance report is generated.

[0105] In summary, the present embodiment effectively solves the key problems of single training scene, subjective parameter adjustment, lack of system closed loop, and insufficient evaluation standards in the prior art by constructing a real-time closed-loop virtual interactive training system based on behavior error feedback, achieving personalized parameter adaptation, quantitative effect evaluation, and safe and efficient training for visual reconstruction brain-computer interface postoperative rehabilitation training; by dynamically generating diversified virtual reality scenes and extracting important representations related to the task, the limitations of traditional training relying only on static images or physical environments are overcome; by individualized visual field-cortical topological mapping and perception function constrained stimulus parameter conversion, the stimulus parameter setting accurately matches the electrode implantation position, perception sensitivity, and cognitive ability differences of each patient; by multi-dimensional behavior error vector calculation and closed-loop adaptive adjustment, a complete feedback is constructed, enabling the system to automatically optimize the stimulation strategy according to the real-time performance of the patient; at the same time, the application of virtual reality technology enables complex scene training to be completed in a safe and low-cost environment, avoiding the safety hazards and high costs of training in real environments, significantly improving the efficiency and effect of visual reconstruction brain-computer interface postoperative rehabilitation training.

[0106] Reference Figure 2 The embodiment further discloses a virtual interaction training device of a visual reconstruction brain-computer interface, comprising a virtual scene generation and processing module 21, a visual information mapping module 22, a stimulation parameter conversion module 23, a behavior data acquisition and analysis module 24 and a closed-loop adjustment module 25, which are described in detail as follows:

[0107] The virtual scene generation and processing module 21 is used for dynamically generating a virtual reality scene and extracting an importance representation related to a current task in the virtual reality scene, and comprises the following steps: selecting a target scene type based on a preset scene label set; dynamically combining model components according to the target scene type to generate a virtual reality scene with different layouts and task elements; determining a current task type according to the target scene type; and extracting visual features related to the task type in the virtual reality scene based on the task type by using an image processing algorithm to generate an importance representation.

[0108] The visual information mapping module 22 is used for extracting key visual information in the importance representation and mapping the key visual information to spatial positions of a cortical electrode array through a preset visual field-cortex topological mapping model, and comprises the following steps: determining a position of a visual topological center in the importance representation; loading a standard visual field-cortex topological relationship matrix used for representing a corresponding relationship between image coordinates and cortex coordinates; generating an individualized visual field-cortex topological mapping model by individualizing the standard visual field-cortex topological relationship matrix according to electrode implantation position information of a patient; and mapping the key visual information in the importance representation to spatial positions of the cortical electrode array through coordinate transformation based on the position of the visual topological center and the individualized visual field-cortex topological mapping model, wherein the coordinate transformation takes the visual topological center as a reference origin.

[0109] The stimulation parameter conversion module 23 is configured to convert the mapped key visual information into the spatiotemporal stimulation parameters of the cortical electrodes based on the pre-acquired individualized perceptual function constraint, including: acquiring an individualized perceptual function for characterizing the correspondence between the electrode stimulation parameters and the visual perceptual parameters of the patient; converting the visual feature parameters of the mapped key visual information into the spatiotemporal stimulation parameters of the cortical electrodes according to the individualized perceptual function; applying safety constraints to the converted spatiotemporal stimulation parameters; the behavior data acquisition and analysis module 24 is configured to acquire the behavior data of the user performing the task under the spatiotemporal stimulation parameters, and compare the behavior data with the preset reference data to calculate a behavior error comprehensive score, including: acquiring the behavior data of the user performing the task under the spatiotemporal stimulation parameters, the behavior data including: a task result indicator, a completion degree indicator, a time consumption ratio indicator, an operation accuracy indicator, and a subjective comfort indicator; calculating the mean and standard deviation of each indicator in the behavior data corresponding to the data set based on the pre-established normal population behavior data set to form the reference data; performing standardization on the task result indicator, the completion degree indicator, the time consumption ratio indicator, and the operation accuracy indicator in the behavior data of the user to generate a first standardized error component set; performing reverse standardization processing on the subjective comfort indicator to generate a second standardized error component; combining the first standardized error component set and the second standardized error component into a behavior error vector; performing weighted calculation on the behavior error vector according to the preset weight coefficient to obtain a behavior error comprehensive score of the difference between the current performance and the target performance of the user; the closed-loop adjustment module 25 is configured to dynamically adjust the difficulty parameters of the task according to the behavior error comprehensive score to complete the closed-loop adaptive training, including: setting an initial threshold range according to the behavior score of the normal population; performing comparison judgment with the initial threshold range based on the behavior error comprehensive score and the subjective comfort indicator, and adjusting the training task difficulty parameters according to the comparison judgment result; performing smooth updating on the adjusted complexity parameters to complete the closed-loop adaptive training.

[0110] Figure 3 An electronic device entity structure schematic diagram provided for the embodiment of the present application is shown in Figure 3 The electronic device 50 includes a processor 501, a memory 502, and a bus 503.

[0111] The processor 501 and the memory 502 communicate with each other through the bus 503; the processor 501 is configured to invoke the program instructions in the memory 502 to execute the methods provided in the above method embodiments.

[0112] The embodiment provides a non-transitory computer-readable storage medium storing computer instructions, and the computer instructions cause a computer to execute the methods provided in the above method embodiments.

[0113] Those skilled in the art can understand that all or part of the steps of the above-mentioned method embodiments can be completed by program instruction related hardware, and the foregoing program can be stored in a computer readable storage medium, and the program performs the steps of the above-mentioned method embodiments when executed; and the foregoing storage medium includes ROM, RAM, magnetic disc or optical disc and various storage media that can store program codes.

[0114] The device embodiments described above are only schematic, wherein the units shown as separate components can or can not be physically separate, and the components shown as units can or can not be physical units, i.e., can be located in one place, or can be distributed on multiple network units. Part or all of the modules can be selected to achieve the purpose of the embodiments according to actual needs. Those skilled in the art can understand and implement without creative labor.

[0115] From the above description of the embodiments, those skilled in the art can clearly understand that the embodiments can be realized by means of software plus necessary general hardware platforms, and of course can also be realized by hardware. Based on such understanding, the above technical solutions, essentially or in other words, the part that contributes to the prior art, can be embodied in the form of a software product, which can be stored in a computer readable storage medium, such as ROM / RAM, magnetic disc, optical disc and the like, and includes a number of instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) execute the methods of the embodiments or some parts of the embodiments.

[0116] The above specific embodiments do not constitute a limitation on the protection scope of the present application. Those skilled in the art should understand that various modifications, combinations, sub-combinations and substitutions can occur depending on design requirements and other factors. Any modification, equivalent replacement and improvement made within the spirit and principles of the present application shall fall within the scope of the present application.

Claims

1. A virtual interactive training method for visual reconstruction brain-computer interface, characterized in that, The method includes: Generate a virtual reality scene and extract the importance representation of the virtual reality scene that is relevant to the current task; Extract key visual information from the importance representation and map the key visual information to the spatial location of the cortical electrode array; Based on pre-acquired individualized perception function constraints, the mapped key visual information is converted into spatiotemporal stimulation parameters of cortical electrodes, including: acquiring an individualized perception function to characterize the correspondence between the patient's electrode stimulation parameters and photophobia perception parameters; converting the visual feature parameters of the mapped key visual information into spatiotemporal stimulation parameters of cortical electrodes according to the individualized perception function; and applying safety constraints to the converted spatiotemporal stimulation parameters. Collect behavioral data of users performing tasks under the spatiotemporal stimulus parameters, and compare the behavioral data with preset benchmark data to calculate a comprehensive score of behavioral error; Based on the comprehensive score of the behavioral errors, the difficulty parameters of the task are dynamically adjusted to complete closed-loop adaptive training.

2. The virtual interactive training method for visual reconstruction brain-computer interface according to claim 1, characterized in that, The process of generating a virtual reality scene and extracting importance representations from the virtual reality scene that are relevant to the current task includes: Select the target scene type based on a preset set of scene tags; Based on the target scene type, the model components are randomly and dynamically combined to generate virtual reality scenes with different layouts and task elements; Determine the current task type based on the target scenario type; Based on the task type, an image processing algorithm is used to extract visual features related to the task type in the virtual reality scene and generate an importance representation.

3. The virtual interactive training method for visual reconstruction brain-computer interface according to claim 1, characterized in that, The step of extracting key visual information from the importance representation and mapping the key visual information to the spatial location of the cortical electrode array includes: Determine the position of the visual topological center in the importance representation; Load a pre-defined standard visual field-cortical topological relation matrix that characterizes the correspondence between image coordinates and cortical coordinates; The standard visual field-cortical topology matrix is ​​individually modified based on the patient's electrode implantation location information to generate an individualized visual field-cortical topology mapping model. Based on the location of the visual topology center and the individualized visual field-cortical topology mapping model, key visual information in the importance representation is mapped to the spatial location of the cortical electrode array through coordinate transformation, wherein the coordinate transformation takes the visual topology center as the reference origin.

4. The virtual interactive training method for visual reconstruction brain-computer interface according to claim 1, characterized in that, The process of collecting user behavior data under the spatiotemporal stimulus parameters and comparing the behavior data with preset benchmark data to calculate a comprehensive behavior error score includes: Collect behavioral data of users performing tasks under spatiotemporal stimulus parameters. The behavioral data includes: task result indicators, completion indicators, time consumption ratio indicators, operation accuracy indicators, and subjective comfort indicators. Based on a pre-established dataset of normal population behavior, the mean and standard deviation of each indicator in the corresponding behavioral data of the dataset are calculated to form benchmark data. Standardize the task result indicators, completion indicators, time consumption ratio indicators and operation accuracy indicators in the user's behavior data to generate a first standardized error component set; The subjective comfort index is subjected to inverse standardization to generate a second standardized error component; The first set of standardized error components and the second set of standardized error components are combined into a behavior error vector; The behavior error vector is weighted according to preset weighting coefficients to obtain a comprehensive score of behavior error that reflects the difference between the user's current performance and the target performance.

5. The virtual interactive training method for visual reconstruction brain-computer interface according to claim 4, characterized in that, The step of dynamically adjusting the task difficulty parameters based on the comprehensive score of the behavioral errors to form closed-loop adaptive training includes: The initial threshold range is set based on the behavioral scores of the normal population; Based on the comprehensive score of the behavioral error and the subjective comfort index, a comparison is made with the initial threshold range, and the difficulty parameters of the training task are adjusted according to the comparison results. Perform a smooth update on the adjusted complexity parameters to complete closed-loop adaptive training.

6. A virtual interactive training device for visual reconstruction brain-computer interface, characterized in that, include: The virtual scene generation and processing module is used to dynamically generate virtual reality scenes and extract importance representations related to the current task from the virtual reality scenes; A visual information mapping module is used to extract key visual information from the importance representation and map the key visual information to the spatial location of the cortical electrode array. The stimulation parameter conversion module is used to convert mapped key visual information into spatiotemporal stimulation parameters of cortical electrodes based on pre-acquired individualized perceptual function constraints. This includes: acquiring an individualized perceptual function that characterizes the correspondence between the patient's electrode stimulation parameters and photophobia perception parameters; converting the visual feature parameters of the mapped key visual information into spatiotemporal stimulation parameters of the cortical electrodes according to the individualized perceptual function; and applying safety constraints to the converted spatiotemporal stimulation parameters. The behavior data acquisition and analysis module is used to collect the behavior data of the user performing the task under the spatiotemporal stimulus parameters, and compare the behavior data with the preset benchmark data to calculate the comprehensive score of behavior error. The closed-loop adjustment module is used to dynamically adjust the difficulty parameters of the task based on the comprehensive score of the behavior error, so as to complete the closed-loop adaptive training.

7. The virtual interactive training device for visual reconstruction brain-computer interface according to claim 6, characterized in that, The virtual scene generation and processing module is also used for: Select the target scene type based on a preset set of scene tags; Based on the target scene type, the model components are randomly and dynamically combined to generate virtual reality scenes with different layouts and task elements; Determine the current task type based on the target scenario type; Based on the task type, an image processing algorithm is used to extract visual features related to the task type in the virtual reality scene and generate an importance representation.

8. A computer-readable storage medium, characterized in that, The storage medium stores a computer program, which, when executed by a processor, implements the method described in any one of claims 1-5.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the method of any one of claims 1-5.

Images