Closed-loop neural feedback system

Abstract

The invention discloses a closed-loop neural feedback system, and relates to the technical field of neural feedback, and the system comprises a near infrared spectrum signal collection module, a signal processing module and an interaction module. The interaction module is used for executing a hand movement task by a user; the near-infrared spectrum signal acquisition module is used for acquiring functional near-infrared spectrum signals of cerebral cortex in real time in the process that a user executes a hand movement task of a current test time, and sending the functional near-infrared spectrum signals acquired by the current test time to the signal processing module; and the signal processing module is used for extracting a target neural feature in the functional near infrared spectrum signal of the current test and generating a task difficulty parameter of the hand movement task of the next test according to the target neural feature of the current test. The neural feedback training effect is improved.

Description

Technical Field

[0001] The present application relates to the technical field of neural feedback, and in particular to a closed-loop neural feedback system. Background Art

[0002] A variety of neurological diseases, such as stroke, Parkinson's disease, Alzheimer's disease and mild cognitive impairment, may all show varying degrees of hand motor dysfunction in clinical practice. Compared with the motor rehabilitation of large limbs, the recovery of hand motor function is particularly difficult. At present, the fine motor rehabilitation methods used in clinical practice mainly focus on the patient's peripheral limb rehabilitation, such as cold therapy, electrical stimulation, exercise therapy and occupational therapy, etc., with a long rehabilitation cycle and limited effect. Taking stroke as an example, after clinical motor rehabilitation training, only 15% of stroke patients can recover their hand motor function to 50% of the original, and less than 3% of patients can recover to 70%. The remaining motor disorders such as abnormal hand muscle tension, difficulty in grasping, and inability to coordinate hands and eyes seriously affect the patient's ability and quality of life, and bring great burdens to family and social medical resources. In recent years, rehabilitation training methods based on neurofeedback technology are believed to induce neuroplasticity in patients with neurological diseases and help promote the recovery of motor function from the level of neural activity.

[0003] Neurofeedback usually converts the neural activity of the trainee's target brain area into easy-to-understand visual, auditory or force-tactile information and feeds it back to the trainee. The trainee autonomously adopts a certain psychological strategy to adjust the feedback signal and achieve the goal of regulating specific neural activity. Existing neurofeedback training methods mostly use the motor imagery paradigm, requiring the trainee to psychologically feel an action process or imagine seeing an action process in the mind but without actual movement; or do not specify a specific psychological strategy, and let the trainee autonomously find a psychological strategy to increase or decrease the feedback signal obtained. Trainees learn to control these neural characteristic indicators through psychological strategies such as motor imagery, and then achieve changes in specific brain functions and behaviors through long-term practice.

[0004] Although the neurofeedback training method based on motor imagery has been widely studied and has brought good news to specific populations, such as spinal cord injury patients with severe limb motor dysfunction, it still has great limitations for some populations. First of all, this method has certain requirements for the trainees' motor imagery and cognitive ability, and often requires a certain amount of training process to master. Many patients with neurological diseases find it difficult to autonomously control the changes in feedback signals by imagining limb movements. In addition, when using the neurofeedback training system, most subjects are required to perform psychological activities uninterruptedly, which can easily cause psychological fatigue and cognitive overload. For those patients who retain the gross motor ability of large limbs but have rehabilitation needs for fine motor functions such as hand flexibility, stability and muscle tension, such as stroke patients who have recovered upper limb and partial hand functions after acute rehabilitation treatment, or children with attention deficit hyperactivity disorder with fine motor disorders, the psychological strategy of motor imagery requires the inhibition of actual movement, which brings an unnatural experience and reduces the patient's willingness to take the initiative in rehabilitation training. Summary of the invention

[0005] The purpose of this application is to provide a closed-loop neurofeedback system that adjusts neural characteristics without the user having to perform additional psychological activities, so that the user can complete rehabilitation training more naturally and improve the training effect of neurofeedback.

[0006] To achieve the above objectives, this application provides the following solutions:

[0007] In a first aspect, the present application provides a closed-loop neural feedback system, comprising: a near-infrared spectral signal acquisition module, a signal processing module and an interaction module; the interaction module is used for a user to perform a hand movement task; the near-infrared spectral signal acquisition module is used to collect functional near-infrared spectral signals of the cerebral cortex in real time during the user's performance of the hand movement task of the current trial, and send the functional near-infrared spectral signals collected in the current trial to the signal processing module; the signal processing module is used to extract target neural features in the functional near-infrared spectral signals of the current trial, and generate task difficulty parameters for the hand movement task of the next trial based on the target neural features of the current trial.

[0008] Optionally, the interaction module uses a visual-tactile fusion interaction software algorithm to build an operating environment for the user to perform hand movement tasks.

[0009] Optionally, the interaction module includes a force feedback device and a visual display device, the visual display device is used to present a three-dimensional virtual scene, a handle at the end of the force feedback device is mapped to a virtual brush of the three-dimensional virtual scene, and when the user holds the handle at the end of the force feedback device and moves it, the virtual brush moves synchronously with the handle;

[0010] The three-dimensional virtual scene includes a virtual wall, the virtual wall includes a circular groove, the circular groove includes an inner circular wall, an outer circular wall and an annular bottom surface connecting the inner circular wall and the outer circular wall, and a circular trajectory is drawn on the annular bottom surface of the circular groove by the virtual brush, which is recorded as one trial;

[0011] When the user touches the annular groove with the virtual paintbrush, the virtual paintbrush and the annular groove in the three-dimensional virtual scene are equivalent to a spring torsion spring connection; when the virtual paintbrush touches the annular bottom surface of the annular groove, the user applies a contact force perpendicular to the virtual wall between the virtual paintbrush and the virtual wall;

[0012] The task difficulty parameter is determined according to the allowable force error range of the contact force, the width of the annular groove or the completion time of a trial.

[0013] Optionally, when the task difficulty parameter is determined according to the allowable force error range of the contact force, each task difficulty parameter corresponds to a task difficulty level, and each task difficulty level corresponds to an allowable force target range. When the user performs the hand movement task of the current trial, if the contact force is within the allowable force target range corresponding to the task difficulty parameter of the current trial, the trajectory of the virtual paintbrush is a first color; if the contact force exceeds the maximum value of the allowable force target range corresponding to the task difficulty parameter of the current trial, the trajectory of the virtual paintbrush is a second color; if the contact force is lower than the minimum value of the allowable force target range corresponding to the task difficulty parameter of the current trial, the trajectory of the virtual paintbrush is a third color, and the first color, the second color and the third color are different colors.

[0014] Optionally, when the task difficulty parameter is determined according to the allowable force error range of the contact force, the task difficulty level is divided into 8 levels, and the 8-level task difficulty levels are F1, F2, F3, F4, F5, F6, F7 and F8 from low to high. When the task difficulty level is F1, the allowable force target range is ±0.4N, when the task difficulty level is F2, the allowable force target range is ±0.35N, when the task difficulty level is F3, the allowable force target range is ±0.3N, when the task difficulty level is F4, the allowable force target range is ±0.25N, when the task difficulty level is F5, the allowable force target range is ±0.2N, when the task difficulty level is F6, the allowable force target range is ±0.15N, when the task difficulty level is F7, the allowable force target range is ±0.1N, and when the task difficulty level is F8, the allowable force target range is ±0.05N.

[0015] Optionally, in terms of generating a task difficulty parameter of a hand movement task for a next trial according to the target neural feature of the current trial, the signal processing module is specifically configured to:

[0016] If the target neural feature of the current trial is higher than the target neural feature of the previous trial, the task difficulty level of the next trial is increased; if the target neural feature of the current trial is lower than the target neural feature of the previous trial, the task difficulty level of the next trial is reduced. Each task difficulty level corresponds to a task difficulty parameter.

[0017] Optionally, the near-infrared spectral signal acquisition module includes a head cap, on which the transmitting probe position of the near-infrared light source and the absorbing probe position of the receiving light source are arranged according to the international 10-20 system. The transmitting probe position of the near-infrared light source is used to arrange the light source transmitter, and the absorbing probe position of the receiving light source is used to arrange the detector, and a measurement channel is formed between adjacent light source transmitters and detectors.

[0018] Optionally, the target neural characteristic is the average peak value of the oxygenated hemoglobin concentration on the target measurement channel in the sensorimotor area of ​​the cerebral cortex; the target measurement channel is a measurement channel with a hemodynamic response characteristic among multiple measurement channels formed between adjacent light source emitters and detectors; the measurement channel with a hemodynamic response characteristic is a measurement channel in which the oxygenated hemoglobin concentration shows a trend of first increasing and then decreasing over time during a hand movement task test.

[0019] Optionally, the task difficulty parameter is expressed as: δF i+1 =δF i +0.05fix(k i ·e i ), e i =p i-1 HbO -p i HbO ;

[0020] Among them, δF i+1 represents the task difficulty parameter of the i+1th trial, δF i represents the task difficulty parameter of the ith trial, fix() represents the rounding function in MATLAB, k i is the proportionality coefficient, p i-1 HbO is the average peak value of oxygenated hemoglobin concentration on the target channel of the i-1th trial, p i HbO is the average peak value of oxygenated hemoglobin concentration on the target channel of the i-th trial, e i For p i-1 HbO With p i HbO The difference.

[0021] Optionally, the proportionality coefficient is expressed as: Among them, a is the set percentage.

[0022] According to the specific embodiments provided in this application, this application discloses the following technical effects:

[0023] The present application provides a closed-loop neural feedback system, wherein a signal processing module extracts target neural features in the functional Near-infrared Spectroscopy (fNIRS) signal of the current trial, and generates task difficulty parameters for the hand movement task of the next trial based on the target neural features of the current trial. The training process does not require the user to adjust the neural features through psychological activities, and the task difficulty of the next trial is adaptively adjusted based on the target neural features extracted from the current trial, thereby reducing the interaction burden, improving the effect of neural feedback training, and further enhancing the degree of neural activation in the sensory motor area of ​​the brain. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0025] Figure 1 A schematic diagram of the structure of a closed-loop neural feedback system provided in one embodiment of the present application;

[0026] Figure 2 A schematic diagram of a hand movement task process provided by an embodiment of the present application;

[0027] Figure 3 A schematic diagram of a circular groove structure provided in an embodiment of the present application;

[0028] Figure 4 A schematic diagram of a virtual brush trajectory on a circular groove provided in an embodiment of the present application;

[0029] Figure 5 A flowchart of a visual-touch fusion interaction software algorithm provided in one embodiment of the present application;

[0030] Figure 6 A diagram showing the arrangement of probes for a near-infrared spectral imaging device provided in one embodiment of the present application;

[0031] Figure 7 A schematic diagram of an adaptive control strategy for task difficulty provided in an embodiment of the present application;

[0032] Figure 8A schematic diagram of tool graphics posture optimization provided in one embodiment of the present application.

[0033] Reference numerals:

[0034] 1-head cap, 2-handle, 3-force feedback device, 4-virtual brush, 5-virtual wall position, 61-annular bottom surface, 62-outer circular wall, 63-inner circular wall. DETAILED DESCRIPTION

[0035] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0036] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, the present application is further described in detail below with reference to the accompanying drawings and specific implementation methods.

[0037] The present application provides a closed-loop neural feedback system, such as Figure 1 As shown, the closed-loop neural feedback system includes: a near-infrared spectral signal acquisition module, a signal processing module and an interaction module; the interaction module is used for the user to perform a hand movement task; the near-infrared spectral signal acquisition module is used to collect the functional near-infrared spectral signal of the cerebral cortex in real time during the user's execution of the hand movement task of the current trial, and send the functional near-infrared spectral signal collected in the current trial to the signal processing module; the signal processing module is used to extract the target neural features in the functional near-infrared spectral signal of the current trial, and generate the task difficulty parameter of the hand movement task of the next trial according to the target neural features of the current trial.

[0038] The purpose of this application is to integrate neurofeedback training for regulating neural activity in target brain areas with physical hand limb training, and to provide a targeted neurofeedback training system that is effective for the recovery of fine motor skills in the hands for people with specific needs. During the training process, the task difficulty of the next trial is adaptively adjusted according to the target neural features extracted in the current trial, reducing the additional interaction burden brought by the current neurofeedback method, and adjusting the neural features in an autonomous and implicit way without the user having to perform additional psychological activities, so that the user can complete the rehabilitation training more naturally. The effective regulation of the neural activity features of the target brain area is achieved through the adaptive adjustment of the parameters of the hand fine motor task, thereby achieving the purpose of improving the fine motor function of the hands of the target population.

[0039] This application adaptively adjusts the difficulty of the fine motor hand movement task of the next trial based on the target neural characteristics of the target brain area, and improves the neural characteristic indicators related to sensorimotor brain function activities through adaptive changes in task difficulty between trials. An important premise of this application is that the target neural characteristics have task difficulty parameter specificity, that is, when the user performs specified fine motor hand movement tasks of different difficulties, the target neural characteristics change more significantly. When the user's neural characteristic index value in the current trial is higher than that in the previous trial, the task difficulty of the next trial is increased; conversely, when the neural characteristics in the current trial are lower than those in the previous trial, the task difficulty of the next trial is reduced to maintain the target neural characteristics at a higher level.

[0040] In an exemplary embodiment, the target neural feature is an indicator that can characterize the intensity of neural activity in the visual-motor brain function network area of ​​the brain, and the target neural feature is obtained by the following steps:

[0041] The raw functional near-infrared spectroscopy signals of a set number of trials on each measurement channel in the prefrontal lobe, sensorimotor area and occipital lobe were collected, and the raw functional near-infrared spectroscopy signals were converted into oxyhemoglobin concentration and deoxyhemoglobin concentration through an offline preprocessing process, and the set number was 20.

[0042] More specifically, in the first step, for each measurement channel, each raw functional near-infrared spectral signal is converted into an optical density signal.

[0043] The second step is to find the standard deviation of the optical density signal, and then use the sliding window method to divide the optical density signal into a number of window signals (i.e., segments), and find the mean of these segments.

[0044] The third step is to extract motion artifact segments: if the mean of an optical density signal segment is greater than 5 times the standard deviation, the optical density signal segment is considered to be a motion artifact segment.

[0045] In the fourth step, the motion artifact segments are corrected using spline interpolation based on the optical density signals of adjacent channels.

[0046] The fifth step is to perform 0.01-0.1 Hz bandpass filtering on the corrected optical density signal, and use the modified Beer-Lambert law to calculate the change curve of oxygenated hemoglobin concentration and deoxygenated hemoglobin concentration over time on each measurement channel.

[0047] The change curves of oxyhemoglobin concentration and deoxyhemoglobin concentration on all channels in the bilateral sensorimotor areas were examined, and the channels showing obvious neural activation characteristics in the sensorimotor areas (i.e., oxyhemoglobin concentration first increased and then decreased to the baseline; deoxyhemoglobin concentration first decreased and then increased to the baseline) were taken as target channels, and the average peak value of oxyhemoglobin concentration on the target channels was taken as the target neural feature.

[0048] In an exemplary embodiment, the hand fine motor task involved in this application is a fine force-position coordinated control of the wrist under visual guidance, focusing on the flexibility of wrist movement and the stability of fine force output, requiring the user to control the hand to move along a specific trajectory while stably outputting a force of a specific amplitude under the guidance of visual information, and feedback to the user whether the actual output force amplitude is within the specified range through the trajectory color. Among them, the difficulty of the task is mainly determined by the allowable error range of the output force.

[0049] The interaction module supports users to perform fine hand movement tasks. The hardware system of the interaction module is mainly composed of a visual display device, a force feedback device 3 and a computer. The software part of the interaction module adopts a visual-touch fusion interaction software algorithm to build an operating environment for users to perform hand movement tasks. Specifically, the visual-touch fusion interaction function is realized by fusing the visual rendering thread and the tactile rendering thread.

[0050] The hand movement task execution process of this application is as follows Figure 2 As shown, the hand movement task involves a wrist fine force-position coordinated control task under visual guidance. The user needs to control the hand to move along a specific trajectory while stably outputting a force of a specific amplitude under the guidance of visual information, and perform continuous "position control" and "force control" operations. More specifically, during the hand movement task, the user sits in front of the screen and grasps the handle 2 of the force feedback device 3 in a natural and comfortable pen-holding posture. The end handle 2 of the force feedback device 3 can move freely with 6 degrees of freedom within a certain spatial range, and provide feedback force in the three directions of x, y, and z. The handle 2 of the force feedback device 3 is mapped to a tool in a three-dimensional virtual scene. When the user moves or rotates the handle 2 in the real physical space, the virtual brush 4 displayed on the screen will move or rotate synchronously. In this embodiment, the force feedback device 3 is preferably a desktop commercial force feedback device 3.

[0051] The interaction module includes a force feedback device 3 and a visual display device, wherein the visual display device is used to present a three-dimensional virtual scene, and a handle 2 at the end of the force feedback device 3 is mapped to a virtual paintbrush 4 of the three-dimensional virtual scene. When the user holds the handle 2 at the end of the force feedback device 3 and moves it, the virtual paintbrush 4 moves synchronously with the handle 2.

[0052] The three-dimensional virtual scene includes a virtual wall, which is a virtual wall constructed on the xy plane. The virtual wall serves as a virtual drawing board, and the virtual wall includes a circular groove, such as Figure 3As shown, the annular groove includes an inner circular wall 63, an outer circular wall 62, and an annular bottom surface 61 connecting the inner circular wall 63 and the outer circular wall 62. The annular bottom surface 61 is the bottom surface of the annular groove. A circular trajectory is drawn on the annular bottom surface 61 of the annular groove by the virtual brush 4, which is recorded as a trial. The drawn trajectory is as shown in FIG. Figure 4 By changing the contact stiffness between the virtual brush 4 and the annular bottom surface 61, and between the virtual brush 4 and the inner and outer circular walls 63 and 62 of the annular groove, the annular bottom surface 61, the outer and inner circular walls 62 and 63 are set to be impenetrable.

[0053] The principle of stiffness adjustment is as follows: in the mechanical model, the object in the virtual environment and the virtual brush 4 are equivalent to a spring torsion spring connection. When the two come into contact, that is, when the mechanical ball models constituting the two are embedded, the feedback force perceived by the user is equal to the product of the embedding depth between the mechanical ball models of the two and the stiffness matrix. The contact stiffness between the two can be adjusted by changing the value of the elements in the stiffness matrix.

[0054] The user touches the annular bottom surface 61, the outer circular wall 62 and the inner circular wall 63 of the annular groove through the virtual brush 4 at the virtual wall position 5 in the actual operation space, and feels that the annular bottom surface 61, the outer circular wall 62 and the inner circular wall 63 of the annular groove are very hard and impenetrable through the feedback force applied to the handle 2 by the force feedback device 3. When the user touches the annular groove through the virtual brush 4, the virtual brush 4 and the annular groove in the three-dimensional virtual scene are equivalent to a spring torsion spring connection; when the virtual brush 4 touches the annular bottom surface 61 of the annular groove, the user applies a contact force perpendicular to the annular bottom surface 61 between the virtual brush 4 and the annular bottom surface 61 of the annular groove, that is, a contact force along the z-axis direction. The "force control" in the tasks involved in this application refers to controlling the amplitude of the contact force along the z-axis direction.

[0055] The task difficulty parameter is determined according to the allowable force error range of the contact force, the width of the annular groove, or the completion time of a trial. The width of the annular groove is the distance between the inner circular wall 63 and the outer circular wall 62. For the force control difficulty, when the user needs to maintain a target contact force of a specific amplitude, the smaller the allowable force error range, the higher the force control difficulty. For the position control difficulty, the narrower the width of the annular groove, the higher the position control difficulty.

[0056] In an exemplary embodiment, the user operates the handle 2 of the force feedback device 3 to control the virtual brush 4 in the virtual scene, so that the brush tip draws a complete circle connected end to end on the annular bottom surface 61 of the annular groove, and tries not to touch the inner and outer walls of the annular ring as much as possible during the drawing process (i.e., precise position control is performed). Drawing a complete circular trajectory is recorded as one trial. While performing position control, the user also needs to maintain a constant contact force between the virtual brush 4 and the annular bottom surface 61.

[0057] When the task difficulty parameter is determined according to the allowable force error range of the contact force, each task difficulty parameter corresponds to a task difficulty level, and each task difficulty level corresponds to an allowable force target range. The contact force output by the user is allowed to fluctuate within a certain allowable force target range. When the user performs the hand movement task of the current trial, if the contact force (actually applied force) is within the allowable force target range corresponding to the task difficulty parameter of the current trial, the track drawn by the virtual brush 4 is the first color. If the contact force exceeds the maximum value of the allowable force target range corresponding to the task difficulty parameter of the current trial, the track drawn by the virtual brush 4 is the second color. If the contact force is lower than the minimum value of the allowable force target range corresponding to the task difficulty parameter of the current trial, the track drawn by the virtual brush 4 is the third color. The first color, the second color and the third color are different colors. The first color is green, the second color is red, and the third color is yellow. During the process of drawing the track, the user needs to continuously adjust the size of the output force according to the color of the track, and try to keep the drawn track all green, that is, to perform precise force control.

[0058] When the task difficulty parameter is determined according to the allowable force error range of the contact force, the target force amplitude is 0.5N, the task difficulty level is divided into 8 levels, and the 8-level task difficulty levels are F1, F2, F3, F4, F5, F6, F7 and F8 from low to high. When the task difficulty level is F1, the allowable force target range is ±0.4N, when the task difficulty level is F2, the allowable force target range is ±0.35N, when the task difficulty level is F3, the allowable force target range is ±0.3N, when the task difficulty level is F4, the allowable force target range is ±0.25N, when the task difficulty level is F5, the allowable force target range is ±0.2N, when the task difficulty level is F6, the allowable force target range is ±0.15N, when the task difficulty level is F7, the allowable force target range is ±0.1N, and when the task difficulty level is F8, the allowable force target range is ±0.05N. Taking F8 as an example, when the task difficulty level is F8, the user needs to always maintain the output force at 0.5N±0.05N (i.e. between 0.45N and 0.55N) when drawing a circular trajectory, so that the drawn trajectory will be green.

[0059] The process of the visual-touch fusion interactive software algorithm in this application is as follows Figure 5 As shown in the figure, the visual-touch fusion interaction software algorithm includes a visual rendering thread and a tactile rendering thread. The visual rendering thread completes the display of the graphics screen based on the OpenGL development kit and refreshes at a frequency of 30Hz to ensure the continuity of the visual screen. The tactile rendering thread is used to complete the force feedback effect and updates the position, posture and feedback force information at a refresh frequency of 1000Hz to provide a smooth force experience. The visual-touch fusion interaction software algorithm is executed by the interaction module in the system.

[0060] The specific execution process of the visual-touch fusion interactive software algorithm is as follows:

[0061] Step 1: Construct triangular patch models of virtual objects and virtual tools. The virtual object refers to the circular groove, and the virtual tool refers to the virtual brush.

[0062] The 3D modeling software is used to construct the 3D models of the cylindrical virtual tool and the circular groove-shaped virtual object. All vertex coordinates, vertex normal vectors and other information of the 3D model are stored in the .obj format file to obtain the triangular patch model of the virtual tool and the virtual object for the visual thread to call.

[0063] Step 2: Construct a ball tree model of virtual objects and virtual tools.

[0064] The octree ball tree model is used to model virtual tools and virtual objects. The higher the ball tree level, the more balls there are, which can better express the detailed features of the object. However, the calculation amount for collision detection and collision response in subsequent steps is correspondingly larger. This application adopts a three-level ball tree model.

[0065] Step 3: Obtain the real tool posture information in the physical space and display it in a visual graphic screen.

[0066] In the visual picture, the position of the virtual object (i.e., the inner circular wall 63, the outer circular wall 62, and the annular bottom 61) is fixed, and the position and posture of the virtual tool (i.e., the virtual brush 4) are synchronized with the real tool (i.e., the handle 2) in the physical space. According to the rigid body matrix transformation principle, the pose coordinates of the real tool in the physical space are converted into the pose coordinates in the virtual space, and the two-dimensional vector graphics of the virtual tool are drawn in real time and displayed on the screen. Among them, the pose coordinate information of the real tool in the physical space is sent by the force feedback device 3 at a frequency of 1000Hz. The OpenGL4.x version is used to call the API to realize the rendering and presentation of the two-dimensional vector graphics of the virtual tool. The basic principle is: the vertex data in the triangular face model is input into the vertex shader for operations related to vertex processing, the vertex position of the model is obtained through matrix transformation, the lighting formula is calculated to generate the color of each vertex, and the texture coordinate attributes of the vertex are generated, etc. In this process, the position coordinates, color, texture coordinates and other information of the vertex are generated for the vertex and sent to the subsequent stage. In the primitive assembly stage, vertices are combined into basic primitives in the form of points, lines, and surfaces, and then sent to the rasterization stage to be converted into a number of pixels. Finally, in the fragment shader, the correct color is determined for each pixel on the screen based on these basic primitive information to complete the display effect of the screen graphics scene.

[0067] Step 4: Calculate the positional relationship between the virtual tool and the virtual object to determine whether the two collide. If no collision occurs, return to step 3; if a collision occurs, proceed to step 5 to execute the collision response process.

[0068] The specific method for determining whether a collision occurs between a virtual tool and a virtual object is as follows: the position and posture coordinates of the real tool in the physical space at the current moment are converted into the position and posture coordinates of the ball-tree model of the virtual tool in the virtual space according to the rigid body matrix transformation principle, and then a multi-point contact collision detection is performed between the ball-tree model of the virtual tool and the ball-tree model of the virtual object. The detection principle is: the coordinates of the ball center in the virtual object model are marked as (x 1 ,y 1 ,z 1 ), the center coordinate of the virtual tool model is marked as (x 2 ,y 2 ,z 2 ), the sphere radius of the virtual object model and the virtual tool model are r 1 and r 2 If the distance between the centers of the two balls is less than the sum of their radii, it is considered that the two balls intersect and a collision occurs between the virtual object and the tool. 2 -x 1 ) 2 +(y 2 -y 1 ) 2 +(z2 -z 1 ) 2 <(r 2 +r 1 ) 2 .

[0069] The ball model is tested layer by layer using a hierarchical concept to improve the detection efficiency. The principle is: first determine whether the root node balls of the virtual object model and the virtual tool model intersect. If they do, perform the next level of sub-ball detection, discard the balls that do not intersect, and then use the intersecting balls to perform the next level of detection until there are no sub-balls on either the virtual object model or the virtual tool model. The ball pair information finally obtained is the result of the collision detection.

[0070] Step 5: Collision response, including tool graphic posture optimization and feedback force calculation output. When the virtual tool collides with the virtual object in the virtual space, a certain depth of embedding occurs between the two at the tactile ball model level. In order to make the user feel an impenetrable collision experience, the virtual tool needs to remain on the surface of the virtual object in the visual display, and cannot penetrate the object and embed inside. Figure 8 This is the schematic diagram for optimizing the tool graphics pose. is the actual physical position of the tool obtained from the force feedback device at time t, which is uniquely represented by six-dimensional variables:

[0071] Among them, the first three dimensions Indicates the position of the virtual tool at time t, and the three-dimensional The posture of the virtual tool at time t is expressed by Euler angles.

[0072] When there is no collision between the virtual tool and the virtual object at time t-1, the position and posture of the graphic tool Pose with physical tools Overlap; after the virtual tool collides with the virtual object at time t, the position and posture of the physical tool Embedded inside the object, the embedding depth is But the graphics tool stays on the surface of the object in the visual display The pose optimization process is to Convert to 's posture.

[0073] The principle of pose optimization is: the connection between the physical tool and the graphic tool is assumed to be a spring torsion spring connection. When the two are separated, the elastic potential energy between the two needs to be minimized, that is, the optimization objective function is minimized:

[0074]

[0075] Among them, G is a 6×6 stiffness matrix, and the values ​​on the diagonal are {k t ,k t ,k t ,k r ,k r ,k r}, k t is the maximum translational stiffness of the force feedback device, k r is the maximum value of the torsional stiffness of the force feedback device. The optimization constraint is that there is no embedding between the graphic tool and the object, that is, the balls that collide physically do not intersect in the graphic display, that is, the formula is satisfied: (x T -x O ) 2 +(y T -y O ) 2 +(z T -z O ) 2 ≥(r T +r O ) 2 .

[0076] Where (x O ,y O , z O ) represents the coordinates of the center of the virtual object model, (x T ,y T , z T ) represents the spherical center coordinates of the virtual tool model, r O and r T are the sphere radius of the virtual object model and the virtual tool model respectively. Afterwards, the effective set method is used for optimization iteration. In each iteration, the inequality constraints are converted into equality constraints for solution to obtain the optimized posture of the graphic tool. And display the graphic screen.

[0077] The feedback force calculation and output principle is: based on the assumption that the object tool and the graphic tool are connected by a spring torsion spring, and using the posture relationship between the two, the three-dimensional force F and torque required for feedback are calculated according to the following formula: Graphic tools refer to the location of the virtual brush displayed in the virtual scene, and object tools refer to the location that the brush has reached in the real world.

[0078] By calling the hdSetDoublev function in the OpenHaptics Toolkit, the calculated force and torque are output to the end handle of the force feedback device, completing a complete collision response process.

[0079] By looping through steps 3, 4, and 5, multiple high-frequency collision responses can be achieved, providing users with a continuous and smooth force interaction experience.

[0080] The above describes in detail the execution process of the visual-touch fusion interaction software algorithm according to steps 1 to 5, which involves the specific process of graphics posture optimization and feedback force calculation. The above visual-touch fusion interaction software algorithm is implemented by the interaction module in terms of hardware. The interaction module hardware includes force feedback equipment, visual display equipment, and a computer. The above visual-touch fusion interaction software algorithm is completed by this computer.

[0081] In terms of generating the task difficulty parameter of the hand movement task for the next trial based on the target neural feature of the current trial, the signal processing module is specifically used to: if the target neural feature of the current trial is higher than the target neural feature of the previous trial, then increase the task difficulty level of the next trial; if the target neural feature of the current trial is lower than the target neural feature of the previous trial, then reduce the task difficulty level of the next trial, and each task difficulty level corresponds to a task difficulty parameter.

[0082] The near-infrared spectrum signal acquisition module includes a head cap 1, on which a transmitting probe position S of a near-infrared light source and an absorbing probe position D of a receiving light source are arranged according to the international 10-20 system. Figure 6 As shown, the transmitting probe position of the near-infrared light source is used to arrange the light source transmitter, and the absorbing probe position of the receiving light source is used to arrange the detector. 18 light source transmitters and 19 detectors are used, which are symmetrically distributed on the left and right sides of the head cap 1. The distance between the light source transmitter and the detector is 3 cm. A measurement channel is formed between adjacent light source transmitters and detectors, with a total of 46 measurement channels. The position D10 of the detector No. 10 corresponds to the Cz position in the international 10-20 system. The measurement channels symmetrically cover the prefrontal cortex, sensorimotor cortex and visual cortex located in the occipital lobe on the left and right sides of the brain.

[0083] The target neural feature is the average peak value of oxygenated hemoglobin concentration on the target measurement channel in the sensory motor area of ​​the cerebral cortex; the target measurement channel is a measurement channel with hemodynamic response characteristics among multiple measurement channels formed between adjacent light source emitters and detectors; the measurement channel with hemodynamic response characteristics is a measurement channel in which the oxygenated hemoglobin concentration shows a trend of first increasing and then decreasing over time during the hand movement task test. The average value of the peak values ​​of oxygenated hemoglobin concentration on all target measurement channels is used as the target neural feature.

[0084] In the closed-loop neural feedback system of this application, the signal processing module completes three processes: (1) preprocessing the original near-infrared spectral signal collected by the subject when performing the task; (2) extracting the target neural features that reflect the level of sensory-motor related neural response; (3) calculating the task difficulty parameter of the next trial based on the target neural features. The specific process is as follows:

[0085] (1) Preprocessing of near-infrared spectral signals: First, the target channel that can reflect the functional neural activation of the sensorimotor area of ​​the brain is selected through a pre-training process. The specific method is as follows: the user first conducts 20 trials of pre-training, records the near-infrared spectral signals on all measurement channels, and converts the original optical signals into the concentrations of oxyhemoglobin and deoxyhemoglobin through an offline preprocessing process. Offline preprocessing is implemented through the built-in open source toolkit HOMER3 in MATLAB. For the closed-loop neural feedback training process, the preprocessing of near-infrared spectral signals is implemented through MATLAB programming design, and the specific process is consistent with the offline preprocessing process.

[0086] (2) Feature extraction process of neural signals:

[0087] When the user performs a fine motor task on the hand, the average peak value of the oxygenated hemoglobin concentration on all target channels during the task execution in the current trial is used as a neural feature indicator to generate the difficulty parameter of the task for the next trial.

[0088] In this application, a formal training phase includes 25 trials, and the task difficulty is adaptively adjusted according to the characteristics of the subject's neural activity. Each trial includes a 20-second task execution period and a 15-second rest period; in each trial, the circular groove and the virtual tool appear on the screen for 20 seconds, and the subject needs to draw a complete green circular trajectory that connects end to end as much as possible within the 20-second limit. If the subject fails to complete the drawing within the time limit, the circular groove and the virtual tool on the screen disappear, and there is no prompt on the screen, indicating that the subject can relax and keep still. The rest period lasts for 15 seconds before entering the task execution period of the next trial.

[0089] In this application, the data segment of oxygenated hemoglobin concentration in the time window of 5s to 25s after the start of the task is extracted and used as the signal source for feature extraction, and the oxygenated hemoglobin concentration data in the time window of -2s to 0s before the start of the task is used as the baseline. Figure 7The figure is a schematic diagram of the feature extraction and adaptive difficulty adjustment process. For each target channel, the peak value of the oxygenated hemoglobin concentration in the time window of 5s to 25s after the start of the task is subtracted from the mean value of the oxygenated hemoglobin concentration in the baseline time window to obtain the peak value of the relative oxygenated hemoglobin concentration during the task as the neural signal feature on the target channel. For n target channels, n neural signal feature values ​​are obtained, and the mean of the n feature values ​​is used as the target neural feature that characterizes the degree of neural activation in the sensorimotor area in the current trial, and the target neural feature is used to generate the task difficulty parameter for the next trial.

[0090] (3) The specific method for calculating the difficulty of the next trial task based on the characteristic value of the target neural feature is as follows:

[0091] When the user performs the task of the ith trial (i=1, 2, ..., 24), the difference between the neural feature values ​​of the ith trial and the i-1th trial is calculated according to the following formula: i =p i-1 HbO -p i HbO .

[0092] p i-1 HbO is the average peak value of oxygenated hemoglobin concentration on the target channel of the i-1th trial, p i HbO is the average peak value of oxygenated hemoglobin concentration on the target channel of the i-th trial, e i For p i-1 HbO With p i HbO The difference.

[0093] The task difficulty parameter δF of the i+1th trial is calculated by the following formula: i+1 :δF i+1 =δF i +0.05fix(k i ·e i ).

[0094] Among them, δF i+1 represents the task difficulty parameter of the i+1th trial, δF i represents the task difficulty parameter of the ith trial, and the coefficient 0.05 represents δF i ∈{0.4,0.35,0.3,0.25,0.2,0.15,0.1,0.05} The minimum difference between the elements in the set. δF i+1 Select from the set and use the fix function in MATLAB to fix p i-1 HbO and pi HbO The difference between them is an integer. i is the proportionality coefficient.

[0095] The proportionality coefficient is expressed as: Where a is a set percentage, and the coefficient a reflects the percentage of sensitivity of task difficulty changes to neural feature values. In this embodiment, a is 20%, which means that if p i HbO P i-1 HbO Decrease or increase by 20%, δF i+1 Relative to δF i Decrease or increase 0.05. To avoid abnormally small p i-1 HbO Values ​​may result in k i If the value is too large, the difficulty of the task will change too much between trials. Add the following constraint: δF i+1 -δF i ≤0.2.

[0096] Under this condition, if the δF calculated by the system is i+1 -δF i If the value of is greater than 0.2, it is set to 0.2 to avoid the task difficulty difference between adjacent trials exceeding four difficulty levels (for example, changing directly from F1 to F5).

[0097] In addition, in the embodiment of the present application, the width of the annular groove is set to 30 mm, and the force control difficulty is adjusted by changing the allowable error range δF of the output force, where δF is defined as the target force F 0 =0.5N and the absolute difference between the upper and lower limits of the target range. The target range is expressed as F 0 ±δF. The difficulty of the task is determined by the value of δF. The smaller δF is, the smaller the allowable output force fluctuation range is and the higher the task difficulty is. In the embodiment of the present application, 8 values ​​of δF are set, which are 0.40N, 0.35N, 0.30N, 0.25N, 0.20N, 0.15N, 0.10N and 0.05N (corresponding to F1 to F8 respectively), corresponding to eight task difficulty levels from the simplest to the most difficult. The mentioned task difficulty setting method and specific difficulty parameter values ​​are not restrictive and are not unique.

[0098] Before the subjects were trained using the closed-loop neural feedback system based on adaptive visual-tactile fusion interaction provided by this application, they were informed of the training requirements in writing: the difficulty of the task during the experiment will increase with the increase in the intensity of neural activity in the sensorimotor cortex, and the subjects need to do their best to complete the current trial, that is, keep the drawn track in a specific color for as long as possible, so that the difficulty of the task in the next trial increases. In the embodiment of this application, the specific color is green. In formal training, the system adaptively adjusts the difficulty of the task according to the above process to achieve an increase in the level of neural activity in the target brain area.

[0099] In an exemplary embodiment, the operation process of the closed-loop neural feedback system of the present application is as follows: the user wears a near-infrared signal acquisition head cap, sits in front of the screen and stares at the visual interface on the display device, holds the handle of the force feedback device in a comfortable pen-holding posture, and performs the above-mentioned hand fine motor tasks as required. During the task, the near-infrared spectrum signal acquisition module collects the user's near-infrared data in real time and transmits it to the signal processing module through the USB interface. The signal processing module performs real-time pre-processing on the original near-infrared signal, extracts the neural characteristic index of the target, generates parameters for task difficulty adjustment, and sends them to the interaction module. The interaction module adjusts the task difficulty of the next trial according to the received parameters, thereby forming a closed-loop feedback training system. The initial task difficulty is set artificially, and in this embodiment, it is set to the simplest F1 difficulty. The present application improves the degree of neural activation in specific brain areas by adaptively changing the task difficulty between trials.

[0100] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0101] This article uses specific examples to illustrate the principles and implementation methods of this application. The description of the above embodiments is only used to help understand the method and core ideas of this application. At the same time, for those skilled in the art, according to the ideas of this application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting this application.

Claims

1. A closed-loop neural feedback system, characterized in that: The closed-loop neural feedback system includes: a near-infrared spectral signal acquisition module, a signal processing module and an interaction module; the interaction module is used for the user to perform a hand movement task; the near-infrared spectral signal acquisition module is used to collect the functional near-infrared spectral signal of the cerebral cortex in real time during the user's performance of the hand movement task of the current trial, and send the functional near-infrared spectral signal collected in the current trial to the signal processing module; the signal processing module is used to extract the target neural features in the functional near-infrared spectral signal of the current trial, and generate the task difficulty parameter of the hand movement task of the next trial according to the target neural features of the current trial.

2. The closed-loop neural feedback system according to claim 1, characterized in that: The interaction module uses a visual-touch fusion interaction software algorithm to build an operating environment for users to perform hand movement tasks.

3. The closed-loop neural feedback system according to claim 2, characterized in that: The interaction module includes a force feedback device and a visual display device, wherein the visual display device is used to present a three-dimensional virtual scene, and a handle at the end of the force feedback device is mapped to a virtual brush of the three-dimensional virtual scene. When the user holds the handle at the end of the force feedback device and moves it, the virtual brush moves synchronously with the handle; The three-dimensional virtual scene includes a virtual wall, the virtual wall includes a circular groove, the circular groove includes an inner circular wall, an outer circular wall and an annular bottom surface connecting the inner circular wall and the outer circular wall, and a circular trajectory is drawn on the annular bottom surface of the circular groove by the virtual brush, which is recorded as one trial; When the user touches the annular groove with the virtual paintbrush, the virtual paintbrush and the annular groove in the three-dimensional virtual scene are equivalent to a spring torsion spring connection; when the virtual paintbrush touches the annular bottom surface of the annular groove, the user applies a contact force perpendicular to the virtual wall between the virtual paintbrush and the virtual wall; The task difficulty parameter is determined according to the allowable force error range of the contact force, the width of the annular groove or the completion time of a trial.

4. The closed-loop neural feedback system according to claim 3, characterized in that: When the task difficulty parameter is determined according to the allowable force error range of the contact force, each task difficulty parameter corresponds to a task difficulty level, and each task difficulty level corresponds to an allowable force target range. When the user performs the hand movement task of the current trial, if the contact force is within the allowable force target range corresponding to the task difficulty parameter of the current trial, the trajectory of the virtual paintbrush is a first color; if the contact force exceeds the maximum value of the allowable force target range corresponding to the task difficulty parameter of the current trial, the trajectory of the virtual paintbrush is a second color; if the contact force is lower than the minimum value of the allowable force target range corresponding to the task difficulty parameter of the current trial, the trajectory of the virtual paintbrush is a third color, and the first color, the second color, and the third color are different colors.

5. The closed-loop neural feedback system according to claim 3, characterized in that: When the task difficulty parameter is determined according to the allowable force error range of the contact force, the task difficulty level is divided into 8 levels, and the 8-level task difficulty levels are F1, F2, F3, F4, F5, F6, F7 and F8 from low to high. When the task difficulty level is F1, the allowable force target range is ±0.4N, when the task difficulty level is F2, the allowable force target range is ±0.35N, when the task difficulty level is F3, the allowable force target range is ±0.3N, when the task difficulty level is F4, the allowable force target range is ±0.25N, when the task difficulty level is F5, the allowable force target range is ±0.2N, when the task difficulty level is F6, the allowable force target range is ±0.15N, when the task difficulty level is F7, the allowable force target range is ±0.1N, and when the task difficulty level is F8, the allowable force target range is ±0.05N.

6. The closed-loop neural feedback system according to claim 1, characterized in that: In terms of generating a task difficulty parameter of a hand movement task for a next trial according to the target neural feature of the current trial, the signal processing module is specifically used to: If the target neural feature of the current trial is higher than the target neural feature of the previous trial, the task difficulty level of the next trial is increased; if the target neural feature of the current trial is lower than the target neural feature of the previous trial, the task difficulty level of the next trial is reduced. Each task difficulty level corresponds to a task difficulty parameter.

7. The closed-loop neural feedback system according to claim 1, characterized in that: The near-infrared spectral signal acquisition module includes a head cap, on which the transmitting probe position of the near-infrared light source and the absorbing probe position of the receiving light source are arranged according to the international 10-20 system. The transmitting probe position of the near-infrared light source is used to arrange the light source transmitter, and the absorbing probe position of the receiving light source is used to arrange the detector. A measurement channel is formed between adjacent light source transmitters and detectors.

8. The closed-loop neural feedback system according to claim 7, characterized in that: The target neural characteristic is the average peak value of the oxygenated hemoglobin concentration on the target measurement channel in the sensory motor area of ​​the cerebral cortex; the target measurement channel is a measurement channel with a hemodynamic response characteristic among multiple measurement channels formed between adjacent light source emitters and detectors; the measurement channel with a hemodynamic response characteristic is a measurement channel in which the oxygenated hemoglobin concentration shows a trend of first increasing and then decreasing over time during a hand movement task test.

9. The closed-loop neural feedback system according to claim 8, characterized in that: The task difficulty parameter is expressed as: δF i+1 =δF i +0.05fix(k i ·e i ), e i =p i-1 HbO -p i HbO ; Among them, δF i+1 represents the task difficulty parameter of the i+1th trial, δF i represents the task difficulty parameter of the ith trial, fix() represents the rounding function in MATLAB, k i is the proportionality coefficient, p i-1 HbO is the average peak value of oxygenated hemoglobin concentration on the target channel of the i-1th trial, p i HbO is the average peak value of oxygenated hemoglobin concentration on the target channel of the i-th trial, e i For p i-1 HbO With p i HbO The difference.

10. The closed-loop neural feedback system according to claim 9, characterized in that: The proportionality coefficient is expressed as: Among them, a is the set percentage.

Images