Cooperative interaction method, terminal and system for collecting eye movement information, brain fnirs detection data and electroencephalogram data

Abstract

The invention provides a collaborative interaction method, terminal and system for collecting eye movement information, brain fnirs detection data and electroencephalogram data. The method comprises the following steps: sequentially executing a preset number of task blocks for a subject at a preset first time interval by utilizing a display, and executing at least two smooth track motion tracking trials in each task block, the execution of the linear back-and-forth motion tracking trial comprises presenting a tracking target performing linear back-and-forth motion in a preset first time period; the execution of the curvilinear motion tracking test includes presenting a tracking target performing curvilinear motion in a preset second time period; eye movement information, brain fnirs detection data and electroencephalogram data of a subject aiming at a tracking target are collected. In this way, the obtained eye movement information, the brain fnirs detection data and the electroencephalogram data cooperate to objectively and comprehensively reflect the influence of different brain function conditions of the subject on each modal data.

Description

Technical Field

[0001] The present application relates to the technical field of brain function assessment, and in particular to a collaborative interaction method for collecting eye movement information, brain fNIRS detection data, and electroencephalogram data. Background Art

[0002] Mental disorders, such as schizophrenia and bipolar disorder, exhibit diverse and complex clinical symptoms that have long-term negative effects on patients' psychological functions and cognitive abilities, and sometimes even endanger their lives. When diagnosing such diseases, doctors often need to rely on their personal clinical experience, and doctors with different qualifications may have different diagnostic opinions. As the number of patients continues to increase, the pressure on doctors in diagnosis has also increased. To alleviate this problem, researchers have developed a variety of brain function assessment methods to increase the speed at which doctors make diagnoses.

[0003] In the related technology, although there are many ways to evaluate brain function related to mental disorders, different brain function evaluation devices have different specific needs in use. For example, the fNIRS signal collected based on fNIRS is a slow signal, and the EEG signal collected based on EEG is a fast signal. Therefore, each time a brain function evaluation related to mental disorders is performed, only a single brain function evaluation method is often used. If you want to obtain accurate evaluation indicators in different dimensions, you need to perform multiple brain function evaluations, which brings inconvenience to the examinee. Although researchers have also tried to combine data from multiple modalities to perform brain function evaluations, these signals of different modalities are usually collected in discrete processes, so there is still a lot of room for improvement. Summary of the invention

[0004] This application is proposed in view of the above technical problems existing in the prior art. This application aims to provide a collaborative interaction method for collecting eye movement information, brain fNIRS detection data, and EEG data, which can provide an eye movement-fNIRS-EEG three-modal task paradigm, and can achieve intuitive, convenient and efficient collaborative interaction with the subject using a single display. In the process of collaborative interaction, eye movement information, brain fNIRS detection data, and EEG data of different time periods are obtained. The acquired eye movement information, brain fNIRS detection data, and EEG data can be coordinated to objectively and comprehensively reflect the impact of different brain function conditions of the subject on each modality data.

[0005] According to a first scheme of the present application, a collaborative interaction method for collecting eye movement information, brain fNIRS detection data, and electroencephalogram (EEG) data is provided, the collaborative interaction method comprising: using a display to sequentially execute a preset number of task blocks for a subject at a preset first time interval, executing at least two smooth trajectory motion tracking trials in each task block, the at least two smooth trajectory motion tracking trials executed include a straight line reciprocating motion tracking trial and a curved line motion tracking trial, executing the straight line reciprocating motion tracking trial includes presenting a tracking target that performs a straight line reciprocating motion within a preset first time period; executing the curved line motion tracking trial includes presenting a tracking target that performs a curved line motion within a preset second time period; and collecting the subject's eye movement information for the tracking target, the brain fNIRS detection data during the first time interval and in each task block, and the electroencephalogram (EEG) data from a preset third time period before the start of presentation of the tracking target in each tracking trial to the end of the current tracking trial.

[0006] According to a second scheme of the present application, a terminal for providing a trimodal paradigm of eye movement information, brain fNIRS detection data, and electroencephalogram data is provided, wherein the terminal is configured to use a processor to: sequentially execute a preset number of task blocks for a subject at a preset first time interval, execute at least two smooth trajectory motion tracking trials in each task block, the at least two smooth trajectory motion tracking trials executed include a straight line reciprocating motion tracking trial and a curved motion tracking trial, executing the straight line reciprocating motion tracking trial includes presenting a tracking target that performs a straight line reciprocating motion within a preset first time period; executing the curved motion tracking trial includes presenting a tracking target that performs a curved motion within a preset second time period; wherein the brain fNIRS detection data during the first time interval and in each task block, and the electroencephalogram data from a preset third time period before the start of presentation of the tracking target in each tracking trial to the end of the current tracking trial, are used to coordinately analyze with the subject's eye movement information for the tracking target to obtain the subject's brain function assessment result.

[0007] According to the third scheme of the present application, a data acquisition system is provided, which includes an eye movement information acquisition device, an fNIRS device, and an electroencephalogram device, and each device is used to execute the collaborative interaction method for collecting eye movement information, brain fNIRS detection data, and electroencephalogram data as described in the embodiments of the present application.

[0008] According to the fourth scheme of the present application, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed, the collaborative interaction method for collecting eye movement information, brain fNIRS detection data, and EEG data as described in the embodiments of the present application is implemented.

[0009] According to a fifth scheme of the present application, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed, a processor is used to provide a tri-modal paradigm of eye movement information, brain fNIRS detection data, and EEG data, specifically comprising: executing a preset number of task blocks in sequence for a subject at a preset first time interval, executing at least two smooth trajectory motion tracking trials in each task block, the at least two smooth trajectory motion tracking trials executed include a straight line reciprocating motion tracking trial and a curved motion tracking trial, executing the straight line reciprocating motion tracking trial includes presenting a tracking target performing a straight line reciprocating motion within a preset first time period; executing the curved motion tracking trial includes presenting a tracking target performing a curved motion within a preset second time period.

[0010] Compared with the prior art, the beneficial effects of the embodiments of the present application are: The collaborative interaction method for collecting eye movement information, brain fNIRS detection data, and EEG data provided in the embodiment of the present application uses a display to sequentially execute a preset number of task blocks for the subject at a preset first time interval, and executes at least two smooth trajectory motion tracking trials in each task block. The at least two smooth trajectory motion tracking trials executed include a straight line reciprocating motion tracking trial and a curved motion tracking trial. Executing the straight line reciprocating motion tracking trial includes presenting a tracking target performing a straight line reciprocating motion within a preset first time period; executing the curved motion tracking trial includes presenting a tracking target performing a curved motion within a preset second time period; in this way, for brain fNIRS detection data, since the task block contains at least two smooth trajectory motion tracking trials, it is sufficient to collect effective and differentiated brain fNIRS detection data, and the block-based paradigm design It can also meet the data collection requirements of fNIRS equipment, thereby obtaining high-quality brain fNIRS detection data; for EEG data, since the movement directions of at least two smooth trajectory motion tracking trials are different, and in the tracking trial where the tracking target performs a relatively simple straight-line motion, the tracking target performs a straight-line reciprocating motion, the expected effect of the subject on the tracking trajectory can be reduced under the premise that the overall difficulty of the trial is moderate, so high-quality EEG data can be collected in both the straight-line reciprocating motion tracking trial and the curved motion tracking trial; for eye movement information, different types of smooth trajectory motion tracking trials can meet the use requirements of brain function evaluation in different dimensions, and the eye movement information obtained by performing a straight-line reciprocating motion tracking trial can effectively evaluate the brain function related to mental disorders, while performing a curved motion tracking trial can evaluate and verify the brain function related to mental disorders. Therefore, when the subject performs the above task blocks, the eye movement-fNIRS-EEG trimodal data can be collected with high quality at the same time, and the data collected in each modality can be used for brain function evaluation related to mental disorders.

[0011] The collaborative interaction method provided in the embodiment of the present application enables the subject to accurately collect eye movement information, brain fNIRS detection data, and EEG data by only executing one task paradigm on a single terminal, thereby accurately evaluating the brain function of the subject, especially patients with mental illness or subjects with a tendency to mental illness, and also making it convenient for doctors to distinguish patients with mental disorders based on the objective biological indicators obtained from the evaluation.

[0012] The above description is only an overview of the technical solution of the present application. In order to more clearly understand the technical means of the present application, it can be implemented in accordance with the contents of the specification. In order to make the above description and other purposes, features and advantages of the present application more obvious and easy to understand, the specific implementation methods of the present application are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] In the drawings, which are not necessarily drawn to scale, the same reference numerals may describe similar parts in different views. Similar reference numerals with letter suffixes or different letter suffixes may represent different examples of similar components. The accompanying drawings generally illustrate various embodiments by way of example and not by way of limitation, and together with the description and claims, are used to illustrate the disclosed embodiments. Such embodiments are illustrative and exemplary and are not intended to be exhaustive or exclusive embodiments of the present method, apparatus, system, or non-transitory computer-readable medium having instructions for implementing the method.

[0014] Figure 1 A flowchart of a collaborative interaction method for collecting eye movement information, brain fNIRS detection data, and EEG data provided by some embodiments of the present disclosure is shown; Figure 2 A schematic diagram showing two smooth trajectory motion tracking trials in a task block in a collaborative interaction method for collecting eye movement information, brain fNIRS detection data, and EEG data provided by some embodiments of the present disclosure is shown; Figure 3 A schematic diagram showing three smooth trajectory motion tracking trials in a task block in a collaborative interaction method for collecting eye movement information, brain fNIRS detection data, and EEG data provided by some embodiments of the present disclosure is shown; Figure 4 A schematic diagram showing four smooth trajectory motion tracking trials in a task block in a collaborative interaction method for collecting eye movement information, brain fNIRS detection data, and EEG data provided by some embodiments of the present disclosure is shown; Figure 5A schematic diagram showing five smooth trajectory motion tracking trials included in a task block in a collaborative interaction method for collecting eye movement information, brain fNIRS detection data, and EEG data provided by some embodiments of the present disclosure. DETAILED DESCRIPTION

[0015] In order to enable those skilled in the art to better understand the technical solution of the present application, the present application is described in detail below in conjunction with the accompanying drawings and specific implementation examples. The embodiments of the present application are further described in detail below in conjunction with the accompanying drawings and specific implementation examples, but are not intended to limit the present application.

[0016] The words "first", "second" and similar words used in this application do not indicate any order, quantity or importance, but are only used to distinguish. The words "include" or "comprises" and similar words used in this application mean that the elements before the word include the elements listed after the word, and do not exclude the possibility of covering other elements. In this application, the arrows shown in the figures of each step are only examples of the execution order, not limitations. The technical solution of this application is not limited to the execution order described in the embodiments. The various steps in the execution order can be combined, decomposed, or the order can be changed, as long as the logical relationship of the execution content is not affected.

[0017] All terms (including technical terms or scientific terms) used in this application have the same meaning as those understood by ordinary technicians in the field to which this application belongs, unless otherwise specifically defined. It should also be understood that terms defined in general dictionaries should be interpreted as having a meaning consistent with their meaning in the context of the relevant technology, and should not be interpreted in an idealized or extremely formal sense, unless explicitly defined here. Techniques and equipment known to ordinary technicians in the relevant field may not be discussed in detail, but where appropriate, the techniques and equipment should be considered as part of the specification.

[0018] See also Figure 1 As shown, it is a flow chart of a collaborative interaction method for collecting eye movement information, brain fNIRS detection data, and EEG data provided by an embodiment of the present disclosure, the method includes S101-S102, wherein: S101: Utilizing a display, sequentially executing a preset number of task blocks at a preset first time interval for the subject, executing at least two smooth trajectory motion tracking trials in each task block, wherein the at least two smooth trajectory motion tracking trials executed include a straight line reciprocating motion tracking trial and a curved motion tracking trial, executing the straight line reciprocating motion tracking trial includes presenting a tracking target performing a straight line reciprocating motion within a preset first time period; executing the curved motion tracking trial includes presenting a tracking target performing a curved motion within a preset second time period.

[0019] S102: Collect the subject's eye movement information for the tracking target, fNIRS detection data of the brain during the first time interval and in each task block, and EEG data from a preset third time period before the start of presentation of the tracking target to the end of the current tracking trial in each tracking trial.

[0020] The following is a detailed description of the above steps.

[0021] For S101, the display can be understood as a device with a display function, which can be a device with both data processing and display functions such as a laptop, a tablet computer, a smart phone, or an external display screen. A task block contains at least two task trials. After the subject completes all task trials in the current task block, the task trial in the next task block can be executed through the display until all task blocks are completed. The smooth trajectory motion tracking trial can be understood as the motion trajectory of the tracking target is smooth in the tracking trial. Specifically, the straight reciprocating motion tracking trial can be understood as the motion of the tracking target in the tracking trial is a reciprocating straight line motion, and the motion direction of the tracking target can be horizontal, vertical, etc.; the curved reciprocating motion tracking trial can be understood as the motion of the tracking target in the tracking trial is a reciprocating curved motion, and the motion direction of the tracking target can be clockwise or counterclockwise, and the trajectory of the curved motion of the tracking target can be semicircular, circular, etc.; the shape of the tracking target can be circular, square, etc.

[0022] In some embodiments, in order to facilitate the subject to track the tracking target, the background color in the display can be black or other colors that are not easy to cause visual fatigue, and the color of the tracking target can be set to red or other colors that are easy to attract the subject's attention. In addition, in the case of color blindness or color weakness of the subject, the color of the tracking target can also be set to a color that the subject can still clearly distinguish and pay attention to in the specific case of color blindness or color weakness, for example, for a subject with red and green color blindness, the color of the tracking target is set to blue or yellow.

[0023] In some embodiments, the preset number can be set to 1-7, for example, 2, 3, 4, 5, 6; the first time interval can be set to 10-60s, for example, 15s, 20s, 30s, 40s, 50s; the first time period can be set to 5-40s, for example, 10s, 20s, 30s; the second time period can be set to 7-50s.

[0024] Thus, setting the first time interval of 10-60s can meet the time requirements for extracting and analyzing the fNIRS index, so as to ensure the accuracy of the fNIRS index finally obtained. The first time period of 5-40s and the second time period of 7-50s are suitable for allowing various subjects (including schizophrenia patients) to make interactive responses consistent with their mental conditions and brain cognition. The continuous brain response of the subject caused by the first time period and the continuous brain response of the subject caused by the second time period rarely continue to the interactive time domain of the next adjacent motion tracking trial, thereby reducing the overlapping confusion of brain responses in the time domain of adjacent motion tracking trials in the task paradigm, so that the detection data of each modality collected in the time domain of each motion tracking trial is more targeted.

[0025] In some embodiments, a rest trial may be set between two consecutive smooth trajectory motion tracking trials, and the duration of the rest trial may be 10-60 seconds. This can not only meet the time requirements for extracting and analyzing fNIRS indicators, but also reduce the task load of the subject.

[0026] For example, a task block containing two smooth trajectory motion tracking trials can be shown as follows: Figure 2 As shown, one task block includes one straight-line reciprocating motion tracking trial and one curved motion tracking trial. The tracking target is a gray ball, and part of the motion trajectory of the tracking target is shown by the dotted line.

[0027] Among them, in one straight line reciprocating motion tracking trial, the movement direction of the tracking target can be to move right first and then to the left, realizing one straight line reciprocating motion. Two straight line reciprocating motions can be included in one straight line reciprocating motion tracking trial to provide sufficient brain fNIRS detection data for the fNIRS device used in conjunction with the eye movement device. At the same time, the reciprocating motion can also prevent the subject from having expectations about the tracking trajectory, thereby better providing high-quality EEG data for the EEG device used in conjunction with the eye movement device.

[0028] The study found that when schizophrenia patients track their eyes, their eyes often lag behind the target, especially when the target moves horizontally. Therefore, adding straight-line round-trip motion tracking trials can effectively assess the brain function of the subjects' tendency to suffer from mental illness and provide objective indicators with reference value for doctors' subjective diagnosis.

[0029] In one curve motion tracking trial, the tracking target can use the center point of the display as the center of the circle and make counterclockwise circular motion with a preset radius. In one curve motion tracking trial, two complete circular motions can be included, that is, the motion trajectory contains two complete circular motion trajectories. In this way, different from the straight-line round-trip motion trajectory, the curve motion can provide tracking trials with higher difficulty and different types. Combined with the data collection method under the three modes, it can provide more objective biological indicators for the diagnosis of clinical mental illness by further exploring the differences in multimodal results under different motion directions and paths.

[0030] For example, a task block containing three smooth trajectory motion tracking trials can be shown as follows: Figure 3 As shown, the at least three smooth trajectory motion tracking trials performed include at least 2 straight-line reciprocating motion tracking trials and 1 curved motion tracking trial. The movement directions of the tracking target in the two straight-line reciprocating motion tracking trials are different, and a partial motion trajectory of the tracking target is shown as a dotted line.

[0031] Among them, in the first of the two straight line reciprocating motion tracking trials, the movement direction of the tracking target may be first moving right and then moving left, realizing one straight line reciprocating motion, and the first straight line reciprocating motion tracking trial may include two straight line reciprocating motions; in the second of the two straight line reciprocating motion tracking trials, the movement direction of the tracking target may be first moving downward and then moving upward, realizing one straight line reciprocating motion, and the second straight line reciprocating motion tracking trial may include two straight line reciprocating motions.

[0032] In this way, compared with the task block that only includes one straight-line reciprocating motion tracking trial, the present implementation includes two straight-line reciprocating motion tracking trials with different motion directions, which can obtain more data for analyzing the subject's condition and provide objective indicators with reference value for the doctor's subjective diagnosis.

[0033] In one curve motion tracking trial, the tracking target can make counterclockwise circular motion with a preset radius with the center point of the display as the center of the circle. In one curve motion tracking trial, two complete circular motions can be included, that is, the motion trajectory contains two complete circular motion trajectories.

[0034] In this way, unlike straight-line reciprocating motion trajectories, curved motion can provide tracking trials with higher difficulty and different types. Combined with the data collection method under three modalities, it can further explore the differences in multimodal results under different motion directions and paths, and provide doctors with more objective biological indicators for the diagnosis of clinical mental illness.

[0035] For example, a task block containing four smooth trajectory motion tracking trials can be shown as follows: Figure 4 As shown, the at least four smooth trajectory motion tracking trials performed include at least two straight line reciprocating motion tracking trials and two curved motion tracking trials, the movement directions of the tracking target in the two straight line reciprocating motion tracking trials are different, the movement directions of the tracking target in the two curved motion tracking trials are different, and a partial motion trajectory of the tracking target is shown as a dotted line.

[0036] Among them, in the first straight line reciprocating motion tracking trial of the two straight line reciprocating motion tracking trials, the movement direction of the tracking target may be to move rightward first and then to move leftward, realizing one straight line reciprocating motion, and one straight line reciprocating motion tracking trial may include two straight line reciprocating motions; in the second straight line reciprocating motion tracking trial of the two straight line reciprocating motion tracking trials, the movement direction of the tracking target may be to move downward first and then to move upward, realizing one straight line reciprocating motion, and one straight line reciprocating motion tracking trial may include two straight line reciprocating motions.

[0037] In the first of the two curved motion tracking trials, the tracking target can take the center point of the display as the center of the circle and make counterclockwise circular motion with a preset radius, and one curved motion tracking trial can include two complete circular motions, that is, the motion trajectory includes two complete circular motion trajectories; in the second of the two curved motion tracking trials, the tracking target can take the center point of the display as the center of the circle and make clockwise circular motion with a preset radius, and one curved motion tracking trial can include two complete circular motions.

[0038] For example, a task block containing five smooth trajectory motion tracking trials can be shown as follows: Figure 5 As shown, the at least five smooth trajectory motion tracking trials performed include at least three straight line reciprocating motion tracking trials and two curved motion tracking trials, and the three straight line reciprocating motion tracking trials include at least one straight line reciprocating motion tracking trial in a first direction and at least one straight line reciprocating motion tracking trial in a second direction; the movement directions of the tracking target in the two curved motion tracking trials are different, and the partial motion trajectory of the tracking target is shown as the dotted line.

[0039] Among them, in the first straight line reciprocating motion tracking trial among the three straight line reciprocating motion tracking trials, the movement direction of the tracking target may be to move to the right first and then to the left, realizing one straight line reciprocating motion, and one straight line reciprocating motion tracking trial may include two straight line reciprocating motions; in the second straight line reciprocating motion tracking trial among the three straight line reciprocating motion tracking trials, the movement direction of the tracking target may be to move to the left first and then to the right, realizing one straight line reciprocating motion, and one straight line reciprocating motion tracking trial may include two straight line reciprocating motions; in the third straight line reciprocating motion tracking trial among the three straight line reciprocating motion tracking trials, the movement direction of the tracking target may be to move downward first and then to move upward, realizing one straight line reciprocating motion, and one straight line reciprocating motion tracking trial may include two straight line reciprocating motions.

[0040] In the first of the two curved motion tracking trials, the tracking target can take the center point of the display as the center of the circle and make counterclockwise circular motion with a preset radius, and one curved motion tracking trial can include two complete circular motions, that is, the motion trajectory includes two complete circular motion trajectories; in the second of the two curved motion tracking trials, the tracking target can take the center point of the display as the center of the circle and make clockwise circular motion with a preset radius, and one curved motion tracking trial can include two complete circular motions.

[0041] In this way, for the specific performance of schizophrenia patients in tracking horizontal moving objects, at least 5 smooth trajectory motion tracking trials, including 2 straight-line reciprocating motion tracking trials with different motion directions (and 2 different horizontal tracking trials), and 2 curve motion tracking trials with different motion directions, can induce a better fNIRS signal response and facilitate the superposition and averaging of EEG data and near-infrared response. On the other hand, the data acquisition method under the three modes can provide objective indicators with reference value for doctors' subjective diagnosis by further exploring the differences in the three-modal results under different motion directions and paths.

[0042] It should be noted that the specific arrangement order of the above-mentioned movement directions can be adjusted according to actual needs, and the embodiments of the present disclosure do not limit the specific arrangement order of the movement directions.

[0043] In some embodiments, the movement of the tracking target in the smooth trajectory motion tracking trial may include an acceleration movement phase and a uniform speed movement phase performed sequentially, wherein: The tracking target can perform accelerated motion at a preset acceleration in the accelerated motion stage until the speed of the tracking target reaches a preset speed; the tracking target can perform uniform motion at the preset speed in the uniform motion stage, thereby achieving a motion effect of first accelerating and then uniform motion.

[0044] Here, the preset speed and the preset acceleration can be set with reference to actual environmental parameters such as the display area of ​​the display, the size of the tracking target, and other actual environmental parameters, and can be set to values ​​that the human eye can perceive and track. For example, the preset speed can be set to 1m / s, 2m / s, etc., and the preset acceleration can be set to 1m / s², 2m / s², etc.

[0045] In some embodiments, the movement of the tracking target in the smooth trajectory motion tracking trial may further include a deceleration movement phase.

[0046] Specifically, the tracking target can accelerate at a preset acceleration in the acceleration stage until the speed of the tracking target reaches a preset speed; the tracking target can move at a uniform speed at the preset speed in the uniform motion stage; the tracking target can decelerate with the preset speed as the initial speed in the deceleration stage until a complete linear motion or circular motion is completed.

[0047] Exemplarily, in a complete circular motion, the tracking target can accelerate in the first stage, move at a constant speed in the second stage, and decelerate in the third stage. The speed change of the tracking target in a complete circular motion can be from 0-preset speed-0, that is, gradually accelerating from 0 to the preset speed, and then gradually decelerating from the preset speed to 0.

[0048] In this way, by setting different types of movement stages including acceleration, constant speed, and deceleration, the tracking target can be made to have a certain degree of tracking difficulty during the movement, avoiding the subjects from having expectations for overly simple tracking tasks, thereby better providing high-quality EEG data for the EEG equipment used in conjunction with the eye movement equipment, and also better providing high-quality brain fNIRS detection data for the fNIRS equipment used in conjunction with the eye movement equipment.

[0049] In some embodiments, a fixation point may be presented in a fourth time period before executing the first task block. The fourth time period may be set to 5-60 seconds.

[0050] Here, presenting the fixation point can attract the subject's attention and calm the subject's emotions before the task block officially begins, so that the subject can perform the task block smoothly.

[0051] S102: Collect the subject's eye movement information for the tracking target, fNIRS detection data of the brain during the first time interval and in each task block, and EEG data from a preset third time period before the start of presentation of the tracking target to the end of the current tracking trial in each tracking trial.

[0052] Here, the step of collecting eye movement information, brain fNIRS detection data, and EEG data in S102 can be executed simultaneously with the task block of S101, that is, while using the display to provide the subject with the task block, the above three-modal data of the subject can be collected, thereby realizing data collection under the three-modal paradigm.

[0053] In some embodiments, the third time period may be 200-2000ms. By collecting EEG data 200-2000ms before presenting the tracking target, the collected EEG data may be used as a baseline in subsequent EEG data analysis, thereby being able to accurately analyze changes in the EEG data of the subject when tracking the tracking target.

[0054] In some embodiments, the collected eye movement information of the subject for the tracking target can represent at least one of the following attributes: the change of the tracking rate, the relative position relationship between the tracking target and the fixation point, the eye saccade index, etc. The relative position relationship between the tracking target and the fixation point can be represented by the tracking gain.

[0055] In some embodiments, in order to reduce the eye movement artifacts in the EEG data, an artifact removal algorithm may be used to process the collected EEG data. The artifact removal algorithm may be, for example, any one or more of independent component analysis (ICA), a combination algorithm based on wavelet transform (WT), ensemble empirical mode decomposition (EEMD) and ICA (WT-EEMD-ICA), etc. By removing the eye movement artifacts in the EEG data, the accuracy of the final brain function assessment results may be improved.

[0056] In some embodiments, the data collected under each modality can be used for joint assessment of the subject's brain function.

[0057] Specifically, the processor can be used to obtain the brain function assessment result of the subject based on the collected eye movement information, EEG data and brain fNIRS detection data of the subject.

[0058] Among them, the brain function assessment result is an objective biological indicator of the subject, which can assist the doctor in evaluating the mental state / brain function of the subject in combination with other subjective and objective indicators such as behavioral science. The brain function assessment result can be expressed in the form of a predicted value representing the normal degree of brain function, the level of concern that requires the doctor's special attention, etc.

[0059] In some embodiments, when the processor is used to obtain the brain function assessment result of the subject based on the collected eye movement information, EEG data and brain fNIRS detection data of the subject, any of the following methods can be used: Method 1: Rule-based data analysis method.

[0060] Here, the usage rules for eye movement information, EEG data, and brain fNIRS detection data can be formulated in advance, and brain function assessment can be performed according to the usage rules.

[0061] The usage rules may include the order of using data / information in different modalities, thresholds indicating normal brain function of the subject, etc.

[0062] Exemplarily, a first screening can be performed based on EEG data and brain fNIRS detection data, and subjects who do not meet the preset normal standards for EEG data and / or brain fNIRS detection data are selected as subjects for the second screening, and the subjects who participate in the second screening are screened for the second time using eye movement information. At this time, according to the results of the two screenings, the brain function evaluation results of subjects who do not meet the preset normal standards for eye movement information can be set as key attention; for subjects who meet the preset normal standards for eye movement information but do not meet the preset normal standards for EEG data and / or brain fNIRS detection data, their brain function evaluation results can be set as general attention; for subjects who meet the preset normal standards for eye movement information, and meet the preset normal standards for EEG data and brain fNIRS detection data, their brain function evaluation results can be set as no need for attention.

[0063] Exemplarily, a first screening may be performed based on the brain fNIRS detection data, and then a second screening may be performed based on the EEG data for subjects who do not meet the preset normal standards for the brain fNIRS detection data. Then, a third screening may be performed based on the eye movement information for subjects who do not meet the preset normal standards for the EEG data and the normal standards for the brain fNIRS detection data, and a brain function assessment result corresponding to the subject is generated according to whether the subject passes the three screenings.

[0064] In this way, by pre-setting the rules for using data under each modality, the subjects can be divided according to the data volume advantage of multimodal data to obtain brain function assessment results. Doctors can also be more targeted in their subsequent assessment of the subjects' mental state, thereby improving the efficiency of doctors' assessment of subjects.

[0065] Method 2: Model-based data analysis method.

[0066] Here, the data of the subject under three modalities can be analyzed synchronously based on the pre-trained model to obtain brain function assessment results.

[0067] Specifically, the training process of the model can be supervised, and the labels used in the training can include at least one of the evaluation results such as the doctor's subjective evaluation results, the scale evaluation results of the examination, etc.; in the input data processing stage of the model, the feature values ​​corresponding to each modality can be spliced ​​according to the preset modal feature splicing order to obtain the features to be analyzed containing the features under the three modalities, and then the features to be analyzed can be classified and processed to obtain the corresponding brain function evaluation results of the subjects.

[0068] In some embodiments, the characteristic value corresponding to the eye movement modality can be obtained by performing feature extraction on indicators such as tracking rate, tracking gain, and eye saccade frequency in the eye movement information; the characteristic value corresponding to the fNIRS modality can be obtained by performing feature extraction on the change in the brain fNIRS detection data in each task block compared to the brain fNIRS detection data during the first time interval; the characteristic value corresponding to the EEG modality can be obtained by performing feature extraction on the EEG data in each task block compared to the change in the EEG data in a third time period preset before the start of presenting the tracking target, and can also be obtained by performing time domain analysis on the EEG data in each task block.

[0069] In this way, the subjects only need to perform one task paradigm to achieve accurate collection of eye movement information, brain fNIRS detection data, and EEG data, thereby accurately evaluating the brain function of the subjects, especially those with mental illness or those with a tendency to mental illness. It also makes it easier for doctors to distinguish patients with mental disorders based on the objective biological indicators obtained from the evaluation.

[0070] In some other embodiments, a terminal for providing a trimodal paradigm of eye movement information, brain fNIRS detection data, and EEG data is provided, and the terminal is configured to use a processor to: sequentially execute a preset number of task blocks for a subject at a preset first time interval, execute at least two smooth trajectory motion tracking trials in each task block, the at least two smooth trajectory motion tracking trials executed include a straight line reciprocating motion tracking trial and a curved motion tracking trial, executing the straight line reciprocating motion tracking trial includes presenting a tracking target performing a straight line reciprocating motion within a preset first time period; executing the curved motion tracking trial includes presenting a tracking target performing a curved motion within a preset second time period; wherein, the brain fNIRS detection data during the first time interval and in each task block, and the EEG data from a preset third time period before the start of presentation of the tracking target in each tracking trial to the end of the current tracking trial, are used to coordinately analyze with the subject's eye movement information for the tracking target to obtain the subject's brain function assessment result.

[0071] For example, the terminal may be a tablet computer, VR glasses, AR glasses, mobile phone or other portable terminals, or a desktop PC, TV or other terminals. Specifically, the terminal may visually present the movement of the tracking target along the trajectory in each motion tracking trial on its own display screen, optionally visually or auditorily prompting the subject to keep tracking the movement of the tracking target, or project these visual images onto other media, such as but not limited to space, a screen or a display screen, which will not be elaborated here.

[0072] In some other embodiments, a data acquisition system is also provided, which includes an eye movement information acquisition device, an fNIRS device, and an electroencephalogram device, and each device is used to execute the collaborative interaction method for collecting eye movement information, brain fNIRS detection data, and electroencephalogram data as described in any of the above embodiments.

[0073] The processor may be a processing device including one or more general-purpose processing devices, such as a microprocessor, a central processing unit (CPU), etc. More specifically, the processor may be a complex instruction set computing (CISC) microprocessor, a reduced instruction set computing (RISC) microprocessor, a very long instruction word (VLIW) microprocessor, a processor running other instruction sets, or a processor running a combination of instruction sets. The processor may also be one or more special-purpose processing devices, such as an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), a digital signal processor (DSP), a system on a chip (SoC), etc.

[0074] This application describes various operations or functions that may be implemented as software codes or instructions or defined as software codes or instructions. Such content may be source code or differential code ("incremental" or "patch" code) that can be directly executed ("object" or "executable" form). Software codes or instructions may be stored in a computer-readable storage medium, and when executed, may cause a machine to perform the described functions or operations, and include any mechanism for storing information in a form accessible to a machine (e.g., a computing device, an electronic system, etc.), such as recordable or non-recordable media (e.g., read-only memory (ROM), random access memory (RAM), magnetic disk storage media, optical storage media, flash memory devices, etc.).

[0075] The exemplary methods described in this application can be implemented at least in part by a machine or computer. In some embodiments, a computer-readable storage medium stores computer program instructions, which, when executed by a processor, cause the processor to execute the collaborative interaction method for collecting eye movement information, brain fNIRS detection data, and EEG data as described in various embodiments of this application.

[0076] In some other embodiments, a computer-readable storage medium is also provided, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed, a processor is used to provide a tri-modal paradigm of eye movement information, brain fNIRS detection data, and EEG data, specifically including: executing a preset number of task blocks in sequence for the subject at a preset first time interval, executing at least two smooth trajectory motion tracking trials in each task block, the at least two smooth trajectory motion tracking trials executed include a straight line reciprocating motion tracking trial and a curved motion tracking trial, executing the straight line reciprocating motion tracking trial includes presenting a tracking target performing a straight line reciprocating motion within a preset first time period; executing the curved motion tracking trial includes presenting a tracking target performing a curved motion within a preset second time period.

[0077] The implementation of such a method may include software code, such as microcode, assembly language code, high-level language code, etc. Various software programming techniques may be used to create various programs or program modules. For example, a program portion or program module may be designed with or by means of Java, Python, C, C++, assembly language, or any known programming language. One or more of such software portions or modules may be integrated into a computer system and / or a computer-readable medium. Such software code may include computer-readable instructions for executing various methods. The software code may form a part of a computer program product or a computer program module. In addition, in an example, the software code may be tangibly stored on one or more volatile, non-temporary or non-volatile tangible computer-readable media, such as during execution or at other times. Examples of these tangible computer-readable media may include, but are not limited to, hard disks, removable disks, removable optical disks (such as optical disks and digital video disks), cassettes, memory cards or memory sticks, random access memories (RAMs), read-only memories (ROMs), etc.

[0078] In addition, although exemplary embodiments have been described herein, the scope includes any and all embodiments based on the present application with equivalent elements, modifications, omissions, combinations (e.g., various embodiments intersecting schemes), adaptations or changes. The elements in the claims will be interpreted broadly based on the language adopted in the claims, and are not limited to the examples described in this specification or during the execution of the application, and the examples will be interpreted as non-exclusive. Therefore, this specification and examples are intended to be considered as examples only, and the true scope and spirit are indicated by the claims and the full scope of their equivalents.

[0079] The above description is intended to be illustrative rather than restrictive. For example, the above examples (or one or more schemes thereof) may be used in combination with each other. For example, a person of ordinary skill in the art may use other embodiments when reading the above description. In addition, in the above-mentioned specific embodiments, various features may be grouped together to simplify the present application. This should not be interpreted as an intention that a disclosed feature that is not required to be protected is necessary for any claim. On the contrary, the subject matter of the present application may be less than all the features of a particular disclosed embodiment. Thus, the claims are incorporated herein as examples or embodiments in the specific embodiments, wherein each claim is independently a separate embodiment, and it is considered that these embodiments may be combined with each other in various combinations or arrangements. The scope of the present application should be determined with reference to the attached claims and the full scope of equivalent forms granted by these claims.

[0080] The above embodiments are only exemplary embodiments of the present application and are not intended to limit the present application. The protection scope of the present application is defined by the claims. Those skilled in the art may make various modifications or equivalent substitutions to the present application within the essence and protection scope of the present application, and such modifications or equivalent substitutions shall also be deemed to fall within the protection scope of the present application.

Claims

1. A collaborative interaction method for collecting eye movement information, brain fNIRS detection data, and EEG data, characterized in that: The collaborative interaction method comprises: Using a display, a preset number of task blocks are sequentially performed for the subject at a preset first time interval, at least two smooth trajectory motion tracking trials are performed in each task block, the at least two smooth trajectory motion tracking trials performed include a straight line reciprocating motion tracking trial and a curved line motion tracking trial, performing the straight line reciprocating motion tracking trial includes presenting a tracking target performing a straight line reciprocating motion within a preset first time period; performing the curved line motion tracking trial includes presenting a tracking target performing a curved line motion within a preset second time period; and The subject's eye movement information for the tracking target, brain fNIRS detection data during the first time interval and in each task block, and EEG data from the preset third time period before the start of the tracking target presentation to the end of the current tracking trial in each tracking trial are collected.

2. The collaborative interaction method according to claim 1, characterized in that: The task block includes at least three smooth trajectory motion tracking trials; wherein, The at least three smooth trajectory motion tracking trials performed include at least two straight-line reciprocating motion tracking trials and one curved motion tracking trial, and the movement directions of the tracking target in the two straight-line reciprocating motion tracking trials are different.

3. The collaborative interaction method according to claim 1, characterized in that: The task block includes at least four smooth trajectory motion tracking trials; wherein, The at least four smooth trajectory motion tracking trials performed include at least two straight-line reciprocating motion tracking trials and two curved-line motion tracking trials, the movement directions of the tracking targets in the two straight-line reciprocating motion tracking trials are different, and the movement directions of the tracking targets in the two curved-line motion tracking trials are different.

4. The collaborative interaction method according to claim 1, characterized in that: The task block includes at least five smooth trajectory motion pursuit trials; The at least five smooth trajectory motion tracking trials performed include at least three straight-line reciprocating motion tracking trials and two curved motion tracking trials, wherein the three straight-line reciprocating motion tracking trials include at least one straight-line reciprocating motion tracking trial in a first direction and at least one straight-line reciprocating motion tracking trial in a second direction; the movement directions of the tracking targets in the two curved motion tracking trials are different.

5. The collaborative interaction method according to any one of claims 1 to 4, characterized in that: The movement of the tracking target in the smooth trajectory motion tracking trial at least includes an acceleration motion phase and a uniform motion phase performed sequentially, wherein: The tracking target performs accelerated motion at a preset acceleration during the accelerated motion phase until the speed of the tracking target reaches a preset speed; The tracking target moves at a uniform speed at the preset speed during the uniform motion stage.

6. The collaborative interaction method according to any one of claims 1 to 4, characterized in that: The preset number is 1-7, the first time interval is 10-60s, the first time period is 5-40s, the second time period is 7-50s, and the third time period is 200-2000ms.

7. The collaborative interaction method according to claim 1, characterized in that: The method further comprises: The fixation point is presented in a fourth time period before executing the first task block, and the fourth time period is 5-60s.

8. The collaborative interaction method according to any one of claims 1 to 4, characterized in that: The subject's eye movement information for the tracking target represents at least one of the following attributes: Changes in tracking rate, relative position of tracking target and fixation point, and saccade indicators.

9. A terminal for providing a three-modal paradigm of eye movement information, brain fNIRS detection data, and EEG data, characterized in that: The terminal is configured to utilize a processor to: The subject is asked to perform a preset number of task blocks in sequence at a preset first time interval, and at least two smooth trajectory motion tracking trials are performed in each task block. The at least two smooth trajectory motion tracking trials performed include a straight line reciprocating motion tracking trial and a curved line motion tracking trial. The straight line reciprocating motion tracking trial includes presenting a tracking target that performs a straight line reciprocating motion within a preset first time period; the curved line motion tracking trial includes presenting a tracking target that performs a curved line motion within a preset second time period; Among them, the fNIRS detection data of the brain during the first time interval and in each task block, and the EEG data from the third time period preset before the start of presentation of the tracking target in each tracking trial to the end of the current tracking trial are used to coordinately analyze with the subject's eye movement information for the tracking target to obtain the subject's brain function assessment results.

10. A data acquisition system, characterized in that: The data acquisition system includes an eye movement information acquisition device, an fNIRS device, and an EEG device, and is configured to execute the collaborative interaction method for collecting eye movement information, brain fNIRS detection data, and EEG data as described in any one of claims 1 to 8.

11. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed, the collaborative interaction method for collecting eye movement information, brain fNIRS detection data, and electroencephalogram data as described in any one of claims 1 to 8 is implemented.

12. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed, the processor provides a three-modal paradigm of eye movement information, brain fNIRS detection data, and EEG data, specifically including: The subject performs a preset number of task blocks in sequence at a preset first time interval, performs at least two smooth trajectory motion tracking trials in each task block, and the at least two smooth trajectory motion tracking trials performed include a straight line reciprocating motion tracking trial and a curved motion tracking trial. Performing the straight line reciprocating motion tracking trial includes presenting a tracking target performing a straight line reciprocating motion within a preset first time period; performing the curved motion tracking trial includes presenting a tracking target performing a curved motion within a preset second time period.

Images