Virtual interaction training method and device for visual reconstruction brain-computer interface

Abstract

The invention discloses a virtual interaction training method and device for a visual reconstruction brain-computer interface, and the method comprises the steps: generating a virtual reality scene, and extracting an importance representation related to a current task in the virtual reality scene; key visual information is extracted, and the key visual information is mapped to the spatial position of the cortex electrode array through a preset visual field-cortex topological mapping model; converting the mapped key visual information into space-time stimulation parameters of the cortical electrode based on a pre-acquired individualized perception function constraint; collecting behavior data of task execution of the user under the spatio-temporal stimulation parameters to calculate a behavior error comprehensive score; and dynamically adjusting difficulty parameters of the task according to the behavior error comprehensive score so as to complete closed-loop adaptive training. The problems of single training scene, subjective parameter adjustment, system closed-loop deficiency and insufficient evaluation standards are solved, and personalized and quantitative evaluation, safety and high efficiency of visual reconstruction brain-computer interface postoperative rehabilitation training are realized.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of visual reconstruction training, in particular to a virtual interaction training method and device of a visual reconstruction brain-computer interface. BACKGROUND

[0002] As a frontier in the intersection of artificial vision and rehabilitation medicine, the visual reconstruction brain-computer interface technology aims to help patients with visual impairment recover part of the visual function by generating light hallucination points in the blind field through cortical electrical stimulation. At present, although the visual prosthesis device (including retinal, optic nerve or cortical stimulation) can generate light hallucination points, patients still need a long time of training after surgery to learn to read the visual information represented by these light points. In the prior art, the postoperative rehabilitation training of visual reconstruction mainly relies on static images or limited physical environments, and the training scene is single and lacks diversity, which is difficult to cover various perception task types required in daily life such as navigation, reading and object recognition. More importantly, most of the existing training systems adopt an open-loop operation mode, and the adjustment of stimulation parameters mainly depends on the subjective experience and offline adjustment of doctors, which cannot be dynamically optimized according to the real-time behavior performance and subjective experience of patients, resulting in long parameter adjustment period, low efficiency, and difficulty in matching the individual needs of each patient due to differences in electrode implantation position, perception sensitivity and cognitive ability. In addition, the existing technology lacks a scientific and quantifiable evaluation system, and the rehabilitation effect mainly relies on empirical judgment, which cannot provide accurate task scores and phased indicators, making it difficult to form an individualized rehabilitation path and long-term tracking effect. In terms of training safety, traditional methods often need to conduct complex scene training in real environments, which not only has safety hazards, but also has high cost and is difficult to reproduce various task types in daily life. The technical solution closest to the present application is to map a preset pattern template to a cortical electrode array, and then perform offline parameter adjustment based on the feedback of the subject, but this method has obvious deficiencies: on the one hand, it lacks real-time closed-loop feedback mechanism, and the adjustment of stimulation parameters cannot quickly and accurately correspond to the behavior results of the patient; on the other hand, it is difficult to complete effective training on complex scenes under safe and low-cost conditions. Therefore, there is an urgent need for a real-time closed-loop training method that can organically combine visual tasks in a virtual environment with cortical electrical stimulation output, support the whole process of information selection, individual adaptation, behavior feedback collection and parameter self-adjustment. SUMMARY

[0003] Therefore, the present application provides a virtual interaction training method and device of a visual reconstruction brain-computer interface, which can realize individualized parameter adaptation, quantitative effect evaluation and efficient training of the visual reconstruction brain-computer interface. The present application provides the following technical solutions: a virtual interaction training method of a visual reconstruction brain-computer interface, the method comprising: generating a virtual reality scene and extracting an importance representation related to a current task in the virtual reality scene; extracting key visual information in the importance representation and mapping the key visual information to spatial locations of the cortical electrode array; converting the mapped key visual information into spatiotemporal stimulation parameters of the cortical electrode based on the pre-acquired individualized perceptual function constraint; collecting behavior data of the user performing the task under the spatiotemporal stimulation parameters and comparing the behavior data with preset reference data to calculate a behavior error comprehensive score; dynamically adjusting a difficulty parameter of the task according to the behavior error comprehensive score to complete a closed-loop adaptive training.

[0004] Optionally, the dynamically generating a virtual reality scene and extracting an importance representation related to a current task in the virtual reality scene comprises: selecting a target scene type based on a preset scene label set; randomly and dynamically combining model components according to the target scene type to generate a virtual reality scene with different layouts and task elements; determining a current task type according to the target scene type; based on the task type, extracting visual features related to the task type in the virtual reality scene using an image processing algorithm to generate an importance representation.

[0005] Optionally, the extracting key visual information in the importance representation and mapping the key visual information to spatial locations of the cortical electrode array comprises: determining a position of a visual topology center in the importance representation; loading a preset standard visual field-cortical topology relationship matrix for representing a corresponding relationship between image coordinates and cortical coordinates, and performing individualized correction on the standard visual field-cortical topology relationship matrix according to electrode implantation position information of the patient to generate an individualized visual field-cortical topology mapping model; based on the position of the visual topology center and the individualized visual field-cortical topology mapping model, mapping key visual information in the importance representation to spatial locations of the cortical electrode array through coordinate transformation, wherein the coordinate transformation takes the visual topology center as a reference origin.

[0006] Optionally, the converting the mapped key visual information into spatiotemporal stimulation parameters of the cortical electrode based on the pre-acquired individualized perceptual function constraint comprises: acquiring an individualized perceptual function for representing a corresponding relationship between electrode stimulation parameters and phosphene perceptual parameters of the patient; and converting visual feature parameters of the mapped key visual information into spatiotemporal stimulation parameters of the cortical electrode according to the individualized perceptual function; applying a safety constraint to the converted spatiotemporal stimulation parameters.

[0007] Optionally, the collecting the behavior data of the user performing the task under the spatio-temporal stimulation parameters, and comparing the behavior data with the preset reference data to calculate a behavior error comprehensive score comprises: collecting the behavior data of the user performing the task under the spatio-temporal stimulation parameters, the behavior data comprising: a task result index, a completion degree index, a time consumption ratio index, an operation precision index and a subjective comfort index; based on a pre-established normal population behavior data set, calculating the mean and standard deviation of each index in the behavior data corresponding to the data set to form reference data; performing standardization on the task result index, the completion degree index, the time consumption ratio index and the operation precision index in the behavior data of the user to generate a first set of standardized error components; performing reverse standardization processing on the subjective comfort index to generate a second standardized error component; combining the first set of standardized error components and the second standardized error component into a behavior error vector; performing weighted calculation on the behavior error vector according to a preset weight coefficient to obtain a behavior error comprehensive score of the difference between the current performance of the user and the target performance.

[0008] Optionally, the dynamically adjusting the difficulty parameter of the task according to the behavior error comprehensive score to form a closed-loop adaptive training comprises: setting an initial threshold range according to the behavior score of the normal population; comparing the behavior error comprehensive score and the subjective comfort index with the initial threshold range, and adjusting the training task difficulty parameter according to the comparison result; performing smooth updating on the adjusted complexity parameter to complete the closed-loop adaptive training.

[0009] The application further discloses a virtual interaction training device of a visual reconstruction brain-computer interface, comprising: a virtual scene generation and processing module, used for dynamically generating a virtual reality scene and extracting an importance representation related to a current task in the virtual reality scene; a visual information mapping module, used for extracting key visual information in the importance representation and mapping the key visual information to a spatial position of a cortical electrode array; a stimulation parameter conversion module, used for converting the mapped key visual information into spatio-temporal stimulation parameters of the cortical electrode based on a pre-acquired individualized perception function constraint; a behavior data collection and analysis module configured to collect behavior data of the user performing the task under the spatiotemporal stimulation parameters, and compare the behavior data with preset reference data to calculate a behavior error comprehensive score; a closed-loop adjustment module configured to dynamically adjust a difficulty parameter of the task according to the behavior error comprehensive score to complete closed-loop adaptive training.

[0010] Optionally, the virtual scene generation and processing module is further configured to: select a target scene type based on a preset scene label set; randomly and dynamically combine model components according to the target scene type to generate virtual reality scenes with different layouts and task elements; determine a current task type according to the target scene type; based on the task type, extract visual features related to the task type in the virtual reality scene using an image processing algorithm to generate an importance representation.

[0011] The application further discloses a computer readable storage medium, wherein the storage medium stores a computer program, and the computer program is executed by a processor to implement the method.

[0012] The application further discloses an electronic device, which comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor implements the method when executing the program.

[0013] According to the technical scheme of the application, the virtual reality scene is dynamically generated, and the importance representation related to the current task is extracted, so that high simulation of multiple scenes is realized, the limitation of traditional training which only relies on static images or entity environment is overcome, the key visual information in the importance representation is further extracted and mapped to the spatial position of the cortical electrode array, the conversion is combined with the individualized perception function constraint to convert into the spatiotemporal stimulation parameters, the stimulation parameter setting is freed from the limitation of the subjective experience of doctors, can accurately match the electrode implantation position, perception sensitivity and cognitive ability difference of each patient, finally, the behavior data of the user is collected, compared with the reference data to calculate the behavior error comprehensive score, and the difficulty parameter of the task is dynamically adjusted according to the score to form the closed-loop adaptive training, so that a complete feedback closed loop of behavior data collection, error vector calculation, parameter dynamic adjustment and effect verification is constructed, the system can automatically optimize the stimulation strategy according to the real-time performance of the patient, and the problem of long parameter adjustment period and low efficiency caused by open-loop operation in the prior art is solved. BRIEF DESCRIPTION OF DRAWINGS

[0014] For the purpose of illustration and not limitation, the application will now be described in conjunction with embodiments thereof and the accompanying drawings, in which: Figure 1is a flowchart of a virtual interaction training method of a visual reconstruction brain-computer interface in an embodiment of the present application; Figure 2 is a structural diagram of a virtual interaction training system of a visual reconstruction brain-computer interface in an embodiment of the present application; Figure 3 is a structural diagram of an electronic device in an embodiment of the present application; Figure 4 is a schematic diagram of image light phantoscope array conversion in an embodiment of the present application; Figure 5 is a schematic diagram of virtual scene generation selection and key visual feature extraction in an embodiment of the present application; Figure 6 is another schematic diagram of virtual scene generation selection and key visual feature extraction in an embodiment of the present application. DETAILED DESCRIPTION

[0015] In order to enable persons skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by persons skilled in the art without creative labor should fall within the scope of protection of the present application.

[0016] It should be noted that the features in the embodiments of the present application and the embodiments can be combined with each other without conflict. The embodiments of the present application will be described in detail below with reference to the accompanying drawings.

[0017] Reference Figure 1 The present embodiment discloses a virtual interaction training method of a visual reconstruction brain-computer interface, which comprises the following steps: S100: generating a virtual reality scene and extracting an importance representation related to a current task in the virtual reality scene.

[0018] First, a predefined set of scene labels is obtained, including, for example, bedroom scene, kitchen scene, large shopping mall, and traffic intersection, etc. Before starting the training, the user selects a target scene type, for example, selects a large shopping mall as the target scene. According to the selected scene label, the current task type is automatically determined, for example, in the large shopping mall scene, the task types include navigation task, reading task, and object recognition task. For the selection of scene labels, the combination selection of scene labels is supported, for example, the traffic intersection and reading labels are selected at the same time, so as to construct a composite scene including red light recognition and road sign reading. In some embodiments, according to the user historical training data and the current rehabilitation stage, the suitable scene combination can be intelligently recommended, for example, for the user in the primary training stage, the scene combination with less obstacles and simple task elements is preferentially recommended.

[0019] After selecting the scene label, the dynamic scene generation function of the Unity engine is called, referring to Figure 5 and Figure 6 , which respectively show the virtual scene generation selection and key visual feature extraction schematic diagram. The random dynamic combination model component generates virtual reality scenes with different layouts and task elements. Specifically, according to the scene label, the corresponding scene structure template is called. For example, for the large shopping mall scene, the system will randomly generate a shopping mall structure with different numbers of floors, different layouts, and different stair combinations. The scene structure generation algorithm adopts a random layout algorithm based on graph theory, which ensures that the generated scene not only conforms to the reality logic but also has diversity. This randomization ensures that the scene layout is different each time the training is performed, avoiding the user completing the task through memory rather than visual reconstruction ability.

[0020] After generating the virtual reality scene, the importance representation related to the current task is extracted. Specifically, first, the current frame image of the Unity real-time rendering is obtained, and according to the current task type, the corresponding image processing algorithm is called to extract the key visual features in the image, forming the task-oriented importance representation. For example, in the navigation task, the edge detection algorithm and transformation are used to extract the boundary contour of the walkable area and the obstacle, and the saliency of the obstacle is determined through the area contrast analysis, for example, in the traffic intersection scene, the importance representation is the highlighted sidewalk boundary, the position of the traffic light, and the vehicle contour; in the reading task, the importance representation is the identification and text information efficiently extracted by the MSER algorithm and the deep learning-based text detection model, and different text areas are distinguished by the semantic segmentation algorithm; and in the object recognition task, the importance representation is the target object accurately recognized by calling the deep learning-based target detection algorithm, and the key area of the object is determined by the saliency detection algorithm.

[0021] The extracted key visual features are converted into an importance map to obtain an importance representation, where the value of each pixel represents the importance weight of the position in the current task. For example, in a navigation task, a higher weight is given to the edge of the walkable area, a medium weight is given to the obstacle contour, and a lower weight is given to the background area.

[0022] S200: Extract key visual information in the importance representation and map the key visual information to the spatial position of the cortical electrode array.

[0023] First, the position of the visual top center in the importance representation obtained in step S100 is determined. For users with detectable eye movement data, including normal vision users and visually impaired users with residual eye movement, real-time head orientation and gaze point information is obtained through a head-mounted eye tracking system, and the precise coordinate position of the visual top center in the current frame image is calculated. For completely blind users who cannot provide reliable eye movement data, the default position of the visual top center is intelligently determined based on the current task type and scene content, for example, in a navigation task, the visual top center is set to a position 15% below the center of the scene, simulating the viewing habit when a human naturally walks; in a reading task, it is set to the center position of the text area by default.

[0024] After determining the visual top center, a preset standard visual field-cortical topological model is used to realize the mapping of key visual information to cortical electrode coordinates. Specifically, when there is a lack of personalized data of the user, a preset standard visual field-cortical topological model is used for mapping. For the construction of the model, based on the fMRI data of healthy subjects, a hyperbolic mapping relationship between the flattened cortex and the visual field is established: x elec =c1 ln(r)cos(θ), y elec =c2 ln(r)sin(θ), where r is the visual field radius, θ is the horizontal viewing angle, c1 and c2 are normalization coefficients, (x elec , y elec) is the physical coordinate of the electrode in the flattened cortex. Then the standard model is corrected according to the CT / MRI data of the user's electrode implantation position. Specifically, the standard visual field-cortex topological relationship matrix is loaded, which represents the corresponding relationship between the image coordinates and the cortical coordinates. The standard visual field-cortex topological relationship matrix is personalized according to the individualized electrode implantation position information of the patient. Specifically, the preoperative CT or MRI image data of the patient is imported, and the physical position of the electrode array is aligned with the standard cortical anatomical structure through medical image registration technology. In the registration process, the iterative closest point algorithm is used to accurately match the actual position of the implanted electrode with the standard cortical model, and the three-dimensional coordinates of each electrode in the cortical coordinate system are calculated. Based on these coordinate information, the standard visual field-cortex topological relationship matrix is subjected to affine transformation and local deformation to generate an individualized visual field-cortex topological mapping model. This model can accurately reflect the correspondence between the patient's specific electrode implantation position and the visual field area, and solve the problem of perceptual misplacement caused by differences in electrode implantation position.

[0025] For the mapping of key visual information to cortical electrode coordinates, in one or more embodiments, a lookup table method based on psychophysical tests can also be used. Specifically, after the user implants the electrode, an individualized perception mapping table is established through psychophysical tests. First, different parameter combinations of electrical pulses are applied to each electrode channel, including current amplitude I e , frequency f e , pulse width w e , and the user's reported photism perception characteristics, including position (r, θ), brightness L e and size B e , are recorded. The above test results are arranged into a three-dimensional lookup table in the format: LUT(x elec , y elec , I e ) = (r, θ, L e , B e ), where (x elec , y elec ) is the physical coordinate of the electrode in the flattened cortex.

[0026] After generating the individualized visual field-cortex topological mapping model, the mapping process of key visual information begins. First, N most task-relevant pixel points are extracted from the importance representation, where the value of N is determined according to the number of channels of the electrode array. The selection of these pixel points is based on the weight distribution in the importance representation, and the pixel points with a weight value higher than a threshold value are preferentially selected. In the mapping process, the relative coordinates of each key pixel point are converted to cortex coordinates through the individualized visual field-cortex topological mapping model with the visual topological center as the reference origin. Specifically, for the key pixel point (u, v) in the importance representation, the offset Δu = u - u0 and Δv = v - v0 of the key pixel point relative to the visual topological center (u0, v0) are calculated. Subsequently, the image offset is converted to visual field polar coordinates (r, θ) using the visual field-cortex topological mapping model: r = |z|. Wherein, is the image space distance, and g, h, j, and q are preset model constants.

[0027] This embodiment gives an implementation example. When the user is in a large shopping mall navigation task, 100 key pixel points in the importance representation are identified, of which 70% are concentrated on the edge of the navigation path, 20% are on the floor indicator, and 10% are on the target store logo. With the visual topological center as the origin, these pixel points are mapped to the cortex electrode array through the individualized visual field-cortex topological mapping model to generate the final electrode stimulation position allocation.

[0028] For this embodiment step, the precise mapping from key visual information in the virtual scene to the spatial position of the cortex electrode is achieved, effectively solving the problem of visual reconstruction accuracy caused by differences in electrode implantation position and perception misalignment in the prior art. This implementation method based on individualized visual field-cortex topological mapping makes the visual reconstruction more consistent with the actual perception characteristics of the patient, significantly improving the readability of the phosphenes information and the task completion efficiency, and providing support for subsequent stimulation parameter conversion and closed-loop adaptive adjustment.

[0029] S300: Based on the pre-acquired individualized perception function constraint, the mapped key visual information is converted into the spatiotemporal stimulation parameters of the cortex electrode.

[0030] First, the pre-acquired individualized perception function is loaded, which is a patient-specific mapping relationship established through preoperative psychophysical testing. In this embodiment, the patient undergoes a series of standardized tests before surgery, and different parameter combinations of electrical stimulation are applied to each electrode channel in turn, and the patient's response to photic hallucination perception is recorded. Specifically, the perception threshold is determined by single electrode stimulation, and then the interference effect between electrodes is tested by multi-electrode combination stimulation, and finally the perception function curve of each electrode channel is formed. The test data is smoothed by spline interpolation method to generate a continuous perception function model, which represents the correspondence between electrode stimulation parameters (current amplitude, stimulation frequency, pulse width) and photic hallucination perception parameters (position, size, brightness). The perception function is stored in the form of parameter threshold table, and each electrode channel corresponds to a stimulation parameter lookup table, which records the brightness and size of the photic hallucination perceived by the patient under different stimulation parameter combinations.

[0031] Reference Figure 4 , an exemplary image phosphen array conversion schematic diagram is shown, after the individualized perception function constraint is obtained, the conversion of key visual information into spatio-temporal stimulation parameters is performed. Specifically, it includes image feature extraction and phosphen array image generation. Among them, for image feature extraction, the key pixel points in the importance representation output in step S200 are first processed by graphics, edge detection and contrast enhancement are performed on the current frame image, and a binary image containing brightness intensity gradient is generated. This image forms a simplified picture similar to lines by retaining high gradient areas and suppressing low gradient backgrounds, where the point brightness intensity at different positions reflects the weight distribution of task relevance. For the generation of the phosphen array image, based on the patient-specific mapping relationship function established by the above psychophysical test, the processed image is matched with the cortical electrode site. Specifically, according to the perception function, the stimulation parameter combination required to achieve the target brightness is determined, and the brightness intensity L image of each point light position in the image is e converted into the corresponding electrode stimulation parameters (current amplitude I e , pulse width w ). Taking current amplitude as an example, query the stimulation parameter lookup table of this electrode channel to find the minimum current amplitude I that can produce the target brightness L. Using the visual cortex topological mapping model, the point light position in the image coordinate system is mapped to the cortical electrode coordinate. For the area where there is a many-to-one mapping, i.e. multiple image points correspond to the same electrode, the weighted average method is used to distribute the stimulation intensity; for many-to-many mapping, the contribution degree of each electrode is balanced through matrix transformation. Further, multiple safety constraints are applied to ensure that the stimulation parameters are within the safe range, and the following safety boundaries are given in this embodiment: Single point brightness check: ensure that the stimulation parameters of each electrode do not exceed the maximum safe brightness L max; activation point number check: dynamically adjust the maximum number of activated points according to the current task type, ensure that the number of activated points does not exceed the total number of electrodes N elec ; ; charge accumulation check: track the 60-second cumulative charge in real time, and trigger the low-power protection mode when the cumulative charge reaches the preset maximum charge C max .

[0032] The present embodiment gives an exemplary training process: When the user identifies the traffic light in the traffic intersection scene, 20 key pixel points mapped to the cortical electrode array are identified, of which 12 correspond to the traffic light outline and 8 correspond to the sidewalk boundary. According to the individualized perception function, the stimulation current of the traffic light outline point is set to 85μA, close to 75% of the maximum safe brightness L max of the electrode; the sidewalk boundary point is set to 65μA, while ensuring that the total number of activated points does not exceed 30, and the cumulative charge within 60 seconds is controlled within 70% of the maximum charge C max . When it is detected that the user has failed to correctly identify the traffic light state twice in a row, the stimulation current of the traffic light outline point is automatically increased to 95μA, and a brief highlight prompt is added in the virtual scene to help the user establish the correct perception association.

[0033] Through the implementation of this step, the precise conversion from key visual information to cortical electrode spatiotemporal stimulation parameters is realized, effectively solving the problems of subjectivity, lack of individualization, and insufficient safety constraints in the prior art. The stimulation parameter conversion method based on individualized perception function makes the optical phosphenes output more consistent with the patient's perception characteristics, significantly improving the interpretability of the visual reconstruction information and the task completion efficiency, while ensuring the safety and comfort of the training process, providing a high-quality stimulation basis for subsequent behavior feedback collection and closed-loop adaptive adjustment.

[0034] S400: Collect behavior data of the user performing tasks under the spatiotemporal stimulation parameters, and compare the behavior data with the preset reference data to calculate a behavior error comprehensive score.

[0035] For the reference data, first, a plurality of normal vision personnel complete all training tasks in a unified virtual scene library. For each volunteer, five key indicators are recorded in each training scene: task result b (0 for success, 1 for failure), completion degree d (number of completed sub-tasks ÷ total number of sub-tasks), time consumption ratio t (actual time consumption ÷ reference time consumption), operation accuracy a (number of correct operations ÷ total number of operations), and subjective comfort c (0-10 points, the higher the score, the more comfortable). Based on the above key indicator data, statistical analysis is performed to calculate the mean μ i and standard deviation σ i, to form a normal person baseline library, i.e., reference data.

[0036] After the reference data is established, the behavior data of the user under the spatio-temporal stimulation parameters determined in step S300 is collected. For example, when the blind patient is performing the traffic intersection training task, the following behavior data is monitored and recorded in real time: whether the patient successfully identifies the traffic light state (task result indicator b), whether the patient completes all sub-tasks of crossing the road (completion degree indicator d), the ratio of the time taken to cross the road to the reference time (time consumption ratio indicator t), whether the patient correctly judges the walkable area (operation accuracy indicator a), and the patient's subjective comfort score for the current stimulation parameters (subjective comfort indicator c).

[0037] After the behavior data is collected, it is compared with the preset reference data to calculate the behavior error vector. First, the task result, completion degree, time consumption ratio, and operation accuracy indicators are subjected to Z-score standardization processing: wherein x i is the behavior data of the user for the above four indicators, μ i is the mean of the reference data for the above four indicators, and σ i is the standard deviation of the reference data for the above four indicators. For the subjective comfort indicator, reverse standardization processing is performed: wherein x c is the subjective comfort indicator data of the user, μ c is the mean of the reference data for the above subjective comfort indicator, and σ c is the standard deviation of the reference data for the above subjective comfort indicator. The five standardized error components after the above standardization are combined to form the behavior error vector: E[|z b |, |z d |, |z t |, |z a |, |z c |], which reflects the degree of deviation of the user from the normal person in each indicator.

[0038] After obtaining the behavior error vector, a weighted calculation is performed on the behavior error vector according to the preset weight coefficients to obtain the behavior error comprehensive score: S = ∑ n ω n |z n |, n ∈ {b, d, t, a, c}, wherein ω n is the preset weight value corresponding to each indicator. Through the behavior error comprehensive score, it can be determined that the lower the value, the closer the user's task execution performance to the normal person level, and the better the subjective experience.

[0039] This implementation method enables objective and quantitative assessment of user behavior, effectively addressing the problem that existing technologies rely primarily on experience-based evaluations and lack quantifiable indicators for rehabilitation effectiveness. The behavior error calculation method based on five-dimensional indicators not only comprehensively reflects the user's training effect but also provides precise feedback signals for closed-loop adaptive adjustment. This allows for automatic optimization of training parameters based on the user's real-time performance, significantly improving the scientific rigor and effectiveness of post-visual reconstruction brain-computer interface rehabilitation training.

[0040] S500: Based on the comprehensive score of the behavioral error, dynamically adjust the difficulty parameters of the task to complete closed-loop adaptive training.

[0041] First, based on the baseline data of normal personnel in step S400, the comprehensive behavioral error score sequence S corresponds to... norm The statistical distribution is defined with its 25th percentile as the lower threshold T. low The 75th percentile is used as the upper threshold T. high In this implementation, starting from the 14th day after the user's surgery, the threshold is automatically updated to the same quantile of the user's comprehensive behavioral error score distribution over the past 7 days, achieving a smooth transition from the group baseline to the individual baseline.

[0042] After determining the threshold corresponding to the user baseline, the training adjustment strategy is executed based on the comparison result between the comprehensive behavioral error score S calculated in step S400 and the threshold. In this embodiment, a continuous judgment mechanism is adopted. Parameter adjustment is triggered only when the comprehensive behavioral error score calculated three consecutive times meets a specific condition, avoiding erroneous adjustments caused by random fluctuations. Specifically, the following judgment rules are adopted: When the comprehensive behavioral error score S ≤ T for three consecutive rounds of training low When the comfort index is not lower than 8, the user training for the current task is judged to be excellent, and a strategy to increase the task difficulty parameter is implemented; when the comprehensive behavioral error score S≥T for two consecutive rounds of training. high If the comfort index is not higher than 4, the user training for the current task is determined to be difficult, and a strategy to lower the task difficulty parameter is implemented. For other cases, the user training for the current task is determined to be neutral, and only a small, smooth adjustment is performed on the task difficulty parameter. All parameter adjustments employ a smooth update strategy to avoid perceptual discomfort caused by sudden parameter changes. Specifically: P k+1 =(1-α)P k +αP new , where P k+1 For the updated parameter value, P k P is the current parameter value. new The adjusted target parameter value is α, where α is the parameter smoothing coefficient.

[0043] The embodiment further exemplarily gives the above-mentioned task difficulty parameter, specifically including virtual reality scene complexity, importance representation extraction strategy and space-time collection parameter conversion rule.

[0044] Among them, the virtual reality scene complexity adjustment includes: Increase scene element density: increase the number of obstacles in the scene in the navigation task; Increase the speed of dynamic elements: increase the movement speed of pedestrians and vehicles in the traffic intersection scene; Reduce the size of the text identification: reduce the size of the text identification in the reading task; Reduce environmental lighting: reduce scene lighting intensity in object recognition tasks.

[0045] Importance representation extraction strategy adjustment includes: Reduce the number of key visual information: reduce the number of extracted importance points; Increase feature extraction threshold: increase the threshold of edge detection and saliency detection, and only keep the most significant features; Reduce task-related areas: reduce the range of walkable areas in navigation tasks.

[0046] Space-time stimulus parameter conversion rule adjustment includes: Reduce stimulation density: reduce the number of simultaneously activated electrodes; Reduce stimulation intensity: reduce stimulation current amplitude; Shorten stimulation duration: reduce stimulation pulse width; Increase stimulation interval: increase stimulation interval time.

[0047] Through the specific implementation of this step, a complete closed-loop adaptive training mechanism is realized, effectively solving the problems of "subjective parameter adjustment" and "system closed loop missing" in the prior art. The dynamic adjustment method of the behavior error comprehensive score not only can automatically optimize the training parameters according to the real-time performance of the user, but also can improve the user's training ability limit under the premise of ensuring safety, significantly improving the efficiency and effect of visual reconstruction brain-computer interface postoperative rehabilitation training. At the same time, the parameter smoothing update and safety boundary mechanism ensures the comfort and sustainability of the training process, so that the user can gradually improve the visual reconstruction ability in a safe and comfortable environment, and finally realize the precise training of the user's visual reconstruction.

[0048] S600: Generate a comprehensive performance report based on the behavior error comprehensive score of the current task stage.

[0049] According to the scene type and characteristics of the training task, the training task is divided into multiple categories, including, for example, indoor navigation tasks, reading function tasks, and interpersonal interaction tasks. For each type of task, a special evaluation dimension and scoring standard are established to ensure that the report content is highly relevant to the task characteristics. Specifically, during the training process, the scene category to which the current task belongs is automatically identified, and the behavior data is classified according to the category. For example, when the user completes a large shopping mall navigation task, it is classified as an indoor navigation task; when the user completes a traffic intersection traffic light recognition task, it is classified as a reading function task. After completing each task, the behavior error comprehensive score of the task is calculated based on step S500, and it is stored in the database of the corresponding scene category together with the historical data. For each scene category, a targeted comprehensive performance report is generated.

[0050] In summary, the present embodiment effectively solves the key problems of single training scene, subjective parameter adjustment, lack of system closed loop, and insufficient evaluation standards in the prior art by constructing a real-time closed-loop virtual interaction training system based on behavior error feedback, realizes personalized parameter adaptation, quantitative effect evaluation, and safe and efficient training for visual reconstruction brain-computer interface postoperative rehabilitation training; by dynamically generating diversified virtual reality scenes and extracting important representations related to the task, the limitations of traditional training relying only on static images or physical environments are overcome; through individualized visual field-cortical topological mapping and perception function constrained stimulation parameter conversion, the stimulation parameter setting accurately matches the electrode implantation position, perception sensitivity, and cognitive ability differences of each patient; through multi-dimensional index behavior error vector calculation and closed-loop adaptive adjustment, a complete feedback is constructed, enabling the system to automatically optimize the stimulation strategy according to the real-time performance of the patient; at the same time, the application of virtual reality technology enables complex scene training to be completed in a safe and low-cost environment, avoiding the safety hazards and high costs of training in real environments, significantly improving the efficiency and effectiveness of visual reconstruction brain-computer interface postoperative rehabilitation training.

[0051] Reference Figure 2 The present embodiment further discloses a virtual interaction training device for a visual reconstruction brain-computer interface, which includes a virtual scene generation and processing module 21, a visual information mapping module 22, a stimulation parameter conversion module 23, a behavior data acquisition and analysis module 24, and a closed-loop adjustment module 25, which are described in detail as follows: The virtual scene generation and processing module 21 is configured to dynamically generate a virtual reality scene and extract an importance representation related to a current task in the virtual reality scene, and includes: selecting a target scene type based on a preset scene label set; dynamically combining model components according to the target scene type to generate a virtual reality scene with different layouts and task elements; determining a current task type according to the target scene type; and extracting visual features related to the task type in the virtual reality scene based on the task type by using an image processing algorithm to generate an importance representation; The visual information mapping module 22 is configured to extract key visual information in the importance representation and map the key visual information to spatial positions of a cortical electrode array through a preset visual field-cortex topological mapping model, and includes: determining a position of a visual topological center in the importance representation; loading a standard visual field-cortex topological relationship matrix used to represent a corresponding relationship between image coordinates and cortex coordinates; performing individualized correction on the standard visual field-cortex topological relationship matrix according to electrode implantation position information of a patient to generate an individualized visual field-cortex topological mapping model; and mapping the key visual information in the importance representation to spatial positions of the cortical electrode array through coordinate transformation based on the position of the visual topological center and the individualized visual field-cortex topological mapping model, wherein the coordinate transformation takes the visual topological center as a reference origin; The stimulation parameter conversion module 23 is configured to convert the mapped key visual information into the spatiotemporal stimulation parameters of the cortical electrodes based on the pre-acquired individualized perceptual function constraint, including: acquiring an individualized perceptual function for characterizing the correspondence between the electrode stimulation parameters and the visual perceptual parameters of the patient; converting the visual feature parameters of the mapped key visual information into the spatiotemporal stimulation parameters of the cortical electrodes according to the individualized perceptual function; applying safety constraints to the converted spatiotemporal stimulation parameters; the behavior data acquisition and analysis module 24 is configured to acquire the behavior data of the user performing the task under the spatiotemporal stimulation parameters, and compare the behavior data with the preset reference data to calculate a behavior error comprehensive score, including: acquiring the behavior data of the user performing the task under the spatiotemporal stimulation parameters, the behavior data including: a task result indicator, a completion degree indicator, a time consumption ratio indicator, an operation accuracy indicator, and a subjective comfort indicator; calculating the mean and standard deviation of each indicator in the behavior data corresponding to the data set based on the pre-established normal population behavior data set to form the reference data; performing standardization on the task result indicator, the completion degree indicator, the time consumption ratio indicator, and the operation accuracy indicator in the behavior data of the user to generate a first standardized error component set; performing reverse standardization processing on the subjective comfort indicator to generate a second standardized error component; combining the first standardized error component set and the second standardized error component into a behavior error vector; performing weighted calculation on the behavior error vector according to the preset weight coefficient to obtain a behavior error comprehensive score of the difference between the current performance and the target performance of the user; the closed-loop adjustment module 25 is configured to dynamically adjust the difficulty parameters of the task according to the behavior error comprehensive score to complete the closed-loop adaptive training, including: setting an initial threshold range according to the behavior score of the normal population; performing comparison judgment with the initial threshold range based on the behavior error comprehensive score and the subjective comfort indicator, and adjusting the training task difficulty parameters according to the comparison judgment result; performing smooth updating on the adjusted complexity parameters to complete the closed-loop adaptive training.

[0052] Figure 3 An electronic device entity structure schematic diagram provided for the embodiment of the present application is shown in Figure 3 The electronic device 50 includes a processor 501, a memory 502, and a bus 503. The processor 501, the memory 502, and the bus 503 are in communication with each other; the processor 501 is configured to invoke program instructions in the memory 502 to execute the methods provided in the above method embodiments.

[0053] The embodiment provides a non-transitory computer-readable storage medium storing computer instructions, and the computer instructions cause a computer to execute the methods provided in the above method embodiments.

[0054] Those skilled in the art can understand that all or part of the steps of the above-mentioned method embodiments can be completed by program instruction related hardware, and the foregoing program can be stored in a computer readable storage medium, and the program performs the steps of the above-mentioned method embodiments when executed; and the foregoing storage medium includes ROM, RAM, magnetic disc or optical disc and various storage media that can store program codes.

[0055] The device embodiments described above are merely illustrative, wherein the units shown as separate components can or can not be physically separated, and the components shown as units can or can not be physical units, i.e., can be located in one place, or can be distributed to multiple network units. Part or all of the modules can be selected to achieve the purpose of the embodiments according to actual needs. Those skilled in the art can understand and implement without creative labor.

[0056] Through the description of the above embodiments, those skilled in the art can clearly understand that the embodiments can be realized by means of software and necessary universal hardware platforms, and of course can also be realized by hardware. Based on such understanding, the above technical solutions can be embodied in the form of software products, and the computer software products can be stored in a computer readable storage medium, such as ROM / RAM, magnetic disc, optical disc and the like, and include a plurality of instructions to make a computer device (which can be a personal computer, a server or a network device, etc.) execute the method of the embodiments or some parts of the embodiments.

[0057] The above specific embodiments do not constitute a limitation on the protection scope of the present application. Those skilled in the art should understand that various modifications, combinations, sub-combinations and substitutions can occur depending on design requirements and other factors. Any modification, equivalent replacement and improvement made within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. A virtual interactive training method for visual reconstruction brain-computer interface, characterized in that, The method includes: Generate a virtual reality scene and extract the importance representation of the virtual reality scene that is relevant to the current task; Extract key visual information from the importance representation and map the key visual information to the spatial location of the cortical electrode array; Based on the pre-acquired individualized perceptual function constraints, the mapped key visual information is converted into spatiotemporal stimulation parameters of cortical electrodes; Collect behavioral data of users performing tasks under the spatiotemporal stimulus parameters, and compare the behavioral data with preset benchmark data to calculate a comprehensive score of behavioral error; Based on the comprehensive score of the behavioral errors, the difficulty parameters of the task are dynamically adjusted to complete closed-loop adaptive training.

2. The virtual interactive training method for visual reconstruction brain-computer interface according to claim 1, characterized in that, The process of generating a virtual reality scene and extracting importance representations from the virtual reality scene that are relevant to the current task includes: Select the target scene type based on a preset set of scene tags; Based on the target scene type, the model components are randomly and dynamically combined to generate virtual reality scenes with different layouts and task elements; Determine the current task type based on the target scenario type; Based on the task type, an image processing algorithm is used to extract visual features related to the task type in the virtual reality scene and generate an importance representation.

3. The virtual interactive training method for visual reconstruction brain-computer interface according to claim 1, characterized in that, The step of extracting key visual information from the importance representation and mapping the key visual information to the spatial location of the cortical electrode array includes: Determine the position of the visual topological center in the importance representation; Load a pre-defined standard visual field-cortical topological relation matrix that characterizes the correspondence between image coordinates and cortical coordinates; The standard visual field-cortical topology matrix is ​​individually modified based on the patient's electrode implantation location information to generate an individualized visual field-cortical topology mapping model. Based on the location of the visual topology center and the individualized visual field-cortical topology mapping model, key visual information in the importance representation is mapped to the spatial location of the cortical electrode array through coordinate transformation, wherein the coordinate transformation takes the visual topology center as the reference origin.

4. The virtual interactive training method for visual reconstruction brain-computer interface according to claim 1, characterized in that, The process of converting the mapped key visual information into spatiotemporal stimulation parameters for cortical electrodes based on pre-acquired individualized perceptual function constraints includes: Obtain an individualized perception function to characterize the correspondence between the patient's electrode stimulation parameters and photophobia perception parameters; based on the individualized perception function, convert the visual feature parameters of the mapped key visual information into spatiotemporal stimulation parameters of the cortical electrodes. Safety constraints are applied to the spatiotemporal stimulus parameters obtained from the transformation.

5. The virtual interactive training method for visual reconstruction brain-computer interface according to claim 1, characterized in that, The process of collecting user behavior data under the spatiotemporal stimulus parameters and comparing the behavior data with preset benchmark data to calculate a comprehensive behavior error score includes: Collect behavioral data of users performing tasks under spatiotemporal stimulus parameters. The behavioral data includes: task result indicators, completion indicators, time consumption ratio indicators, operation accuracy indicators, and subjective comfort indicators. Based on a pre-established dataset of normal population behavior, the mean and standard deviation of each indicator in the corresponding behavioral data of the dataset are calculated to form benchmark data. Standardize the task result indicators, completion indicators, time consumption ratio indicators and operation accuracy indicators in the user's behavior data to generate a first standardized error component set; The subjective comfort index is subjected to inverse standardization to generate a second standardized error component; The first set of standardized error components and the second set of standardized error components are combined into a behavior error vector; The behavior error vector is weighted according to preset weighting coefficients to obtain a comprehensive score of behavior error that reflects the difference between the user's current performance and the target performance.

6. The virtual interactive training method for visual reconstruction brain-computer interface according to claim 5, characterized in that, The step of dynamically adjusting the task difficulty parameters based on the comprehensive score of the behavioral errors to form closed-loop adaptive training includes: The initial threshold range is set based on the behavioral scores of the normal population; Based on the comprehensive score of the behavioral error and the subjective comfort index, a comparison is made with the initial threshold range, and the difficulty parameters of the training task are adjusted according to the comparison results. Perform a smooth update on the adjusted complexity parameters to complete closed-loop adaptive training.

7. A virtual interactive training device for visual reconstruction brain-computer interface, characterized in that, include: The virtual scene generation and processing module is used to dynamically generate virtual reality scenes and extract importance representations related to the current task from the virtual reality scenes; A visual information mapping module is used to extract key visual information from the importance representation and map the key visual information to the spatial location of the cortical electrode array. The stimulation parameter conversion module is used to convert the mapped key visual information into spatiotemporal stimulation parameters of cortical electrodes based on pre-acquired individualized perceptual function constraints. The behavior data acquisition and analysis module is used to collect the behavior data of the user performing the task under the spatiotemporal stimulus parameters, and compare the behavior data with the preset benchmark data to calculate the comprehensive score of behavior error. The closed-loop adjustment module is used to dynamically adjust the difficulty parameters of the task based on the comprehensive score of the behavior error, so as to complete the closed-loop adaptive training.

8. The virtual interactive training device for visual reconstruction brain-computer interface according to claim 7, characterized in that, The virtual scene generation and processing module is also used for: Select the target scene type based on a preset set of scene tags; Based on the target scene type, the model components are randomly and dynamically combined to generate virtual reality scenes with different layouts and task elements; Determine the current task type based on the target scenario type; Based on the task type, an image processing algorithm is used to extract visual features related to the task type in the virtual reality scene and generate an importance representation.

9. A computer-readable storage medium, characterized in that, The storage medium stores a computer program, which, when executed by a processor, implements the method described in any one of claims 1-6.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the method of any one of claims 1-6.

Images