Self-adaptive determination method and system for emotion cognition difficulty of interaction task

Abstract

The invention provides a self-adaptive determination method and system for the emotion cognition difficulty of an interaction task, and the method comprises the steps: determining the emotion cognition difficulty for a first round of interaction task; for each round of interaction task after the first round, the method comprises a model online updating step and an emotion cognition difficulty determining step. And an emotion cognition difficulty determination step: determining the difficulty of the task situation in the current round of interaction task, and determining the difficulty of the limb action corresponding to the emotion to be recognized in the current round of interaction task, thereby presenting the corresponding task situation and limb action to the subject through the immersive interaction technology. Obtaining an interaction result of the subject in the process of executing the interaction task of the current round; and a model online updating step: updating the baseline model and the difficulty adjustment probability model based on the interaction result corresponding to the previous round of interaction task, so that the current round of interaction task is executed based on the updated baseline model and difficulty adjustment probability model. According to the invention, the accuracy of cognitive assessment can be improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of human-computer interaction, and in particular to an adaptive determination method and system for emotional cognitive difficulty of an interactive task. BACKGROUND

[0002] In recent years, the development of human-computer interaction and wearable technology has brought new opportunities for the diagnosis of cognitive impairment research. Especially immersive interaction technology, non-invasive electroencephalogram (EEG) acquisition and eye tracking (Eye Tracking) means, with their portability and high sensitivity to cognitive state, have gradually become an important data source for emotional cognitive research.

[0003] However, the current emotional cognitive interactive task with a single difficulty level is used to judge the cognitive state of the subject, which cannot fully activate the cognitive ability of the subject. Therefore, there is an urgent need for a method to adaptively adjust the difficulty level of the emotional cognitive interactive task to fully activate the cognitive ability of the subject, thereby improving the accuracy of judging the cognitive ability and state of the subject. SUMMARY

[0004] In view of this, the embodiments of the present application provide an adaptive determination method and system for emotional cognitive difficulty of an interactive task, which can fully activate the cognitive ability of the subject during the emotional cognitive interaction process, thereby improving the accuracy of judging the cognitive ability and state of the subject.

[0005] One aspect of the present application provides an adaptive determination method for emotional cognitive difficulty of an interactive task. For the first round of interactive tasks, the method includes an emotional cognitive difficulty determination step. For each round of interactive tasks after the first round, the method includes an online model updating step and an emotional cognitive difficulty determination step. The emotional cognitive difficulty determination step: determining the difficulty of the task context in the current round of interactive tasks and the to-be-identified emotion corresponding to the current round of interactive tasks, and determining the difficulty of the body movement corresponding to the to-be-identified emotion determined in the current round of interactive tasks, so as to present the corresponding task context and body movement to the subject through immersive interaction technology, and obtain the interaction result during the execution of the current round of interactive tasks by the subject; wherein the difficulty of the task context and the difficulty of the body movement corresponding to the to-be-identified emotion in the first round of interactive tasks are predetermined; for each round of interactive tasks after the first round, the difficulty of the task context is adaptively determined based on the interaction result corresponding to the last round of interactive tasks and the difficulty adjustment probability model of the current round, and the difficulty of the body movement is adaptively determined based on the interaction result corresponding to the last round of interactive tasks and the baseline model of the current round; The model online updating step: updating the baseline model and the difficulty adjustment probability model based on the interaction result corresponding to the last round of interaction task, so that the current round of interaction task is executed based on the updated baseline model and difficulty adjustment probability model; wherein the interaction result corresponding to the last round of interaction task includes the eye movement data and the electroencephalogram data collected during the subject performing the last round of interaction task, and the emotional cognitive result of the subject on the limb movement in the last round of interaction task; the baseline model and the difficulty adjustment probability model of the second round are respectively obtained by updating the initial baseline model and the initial difficulty adjustment probability model constructed in advance.

[0006] In some embodiments of the application, for each round of interaction task after the first round, the difficulty of the limb movement in the current round of interaction task is determined by the following method: In the case that the emotional cognitive result in the last round of interaction result meets the set condition, comparing the pre-baseline obtained based on the eye movement data and the electroencephalogram data in the last round of interaction result with the current round baseline obtained based on the updated baseline model, if the pre-baseline is less than the current round baseline, then for the determined limb movement corresponding to the to-be-identified emotion, the difficulty is the first difficulty level higher than the difficulty of the limb movement in the last round of interaction task, otherwise the difficulty is the second difficulty level higher than the first difficulty level; In the case that the emotional cognitive result in the last round of interaction result does not meet the set condition, for the determined limb movement corresponding to the to-be-identified emotion, the difficulty is lower than the difficulty of the limb movement in the last round of interaction task.

[0007] In some embodiments of the application, the pre-baseline is obtained by extracting the eye movement features and the electroencephalogram features from the eye movement data and the electroencephalogram data in the last round of interaction result, and respectively assigning the pre-set weights to the eye movement features and the electroencephalogram features; The current round baseline is obtained by inputting the eye movement data and the electroencephalogram data in the last round of interaction result into the current round baseline model, and outputting the current round baseline through the current round baseline model.

[0008] In some embodiments of the application, for each round of interaction task after the first round, the difficulty of the task context in the current round of interaction task is determined by the following method: The input of the difficulty adjustment probability model of the current round is obtained based on the eye movement data and the electroencephalogram data in the last round of interaction result, the adjustment probability of the task context difficulty is output by the difficulty adjustment probability model of the current round, and the difficulty of the task context in the current round of interaction task is determined based on the adjustment probability and the difficulty of the task context in the last round of interaction task.

[0009] In some embodiments of the application, the to-be-identified emotion corresponding to the current round of interaction task is determined by the following method: the to-be-identified emotion corresponding to the first round of interaction task is a preset all types of to-be-identified emotion; for each round of interaction task after the first round of interaction task, the to-be-identified emotion corresponding to the current round of interaction task is determined according to the preset cognitive order of each type of to-be-identified emotion.

[0010] In some embodiments of the present application, the to-be-identified emotion includes the following types: anger, happiness, disgust, fear, surprise, sadness and neutrality; The body movement corresponding to each type of to-be-identified emotion is designed according to the following correspondence: For the to-be-identified emotion of the type of anger, the corresponding region of interest is the arm and leg; For the to-be-identified emotion of the type of happiness, the corresponding region of interest is the leg; For the to-be-identified emotion of the type of disgust, the corresponding region of interest is the head and the core region of the torso; For the to-be-identified emotion of the type of fear, the corresponding region of interest is the arm and leg; For the to-be-identified emotion of the type of surprise, the corresponding region of interest is the arm and head; For the to-be-identified emotion of the type of sadness, the corresponding region of interest is the head; For the to-be-identified emotion of the type of neutrality, the corresponding region of interest is the arm, leg, head and core region of the torso.

[0011] In some embodiments of the present application, the difficulty of the task context is determined based on the difficulty of the scene and the difficulty of the subject's response to the task; The task context presented to the subject is selected from a task context library containing multiple difficulties according to the determined difficulty of the task context; The task context library includes the following four task contexts: For the first task context, the scene is a background-free indoor living room, and the subject's response task is to identify emotions according to specific body movements; For the second task context with a higher difficulty than the first task context, the scene is an indoor living room with background noise interference of the first decibel level, and the subject's response task is to identify emotions according to specific body movements; For the third task context with a higher difficulty than the second task context, the scene is an indoor living room with background noise interference of the second decibel level, and the subject's response task is to identify emotions according to specific body movements and respond; For a fourth task situation which is more difficult than the third task situation, the scene is an outdoor forest park with third decibel level noise interference and visual flow interference in the background, and the subject's response task is to identify and respond to emotions according to specific limb movements.

[0012] Another aspect of the present application provides a system for adaptive determination of emotional cognitive difficulty of an interactive task, comprising a processor, a memory, and a computer program / instructions stored on the memory, the processor being configured to execute the computer program / instructions, and the system implementing the steps of the method according to any one of the above embodiments when the computer program / instructions are executed.

[0013] Another aspect of the present application provides a computer readable storage medium having stored thereon a computer program / instructions, the computer program / instructions being executed by a processor to implement the steps of the method according to any one of the above embodiments.

[0014] Another aspect of the present application provides a computer program product comprising a computer program / instructions, the computer program / instructions being executed by a processor to implement the steps of the method according to any one of the above embodiments.

[0015] The adaptive determination method and system for emotional cognitive difficulty of an interactive task according to the present application can update the baseline model and the difficulty adjustment probability model according to the interactive result of the previous round of interactive task when the subject performs multiple rounds of interactive task, and can adaptively adjust the difficulty level of the task situation and the limb movement based on the updated baseline model and the difficulty adjustment probability model, so as to achieve the purpose of adjusting the difficulty level of the emotional cognitive interactive task. The adaptive determination method for emotional cognitive difficulty according to the present application can fully activate the cognitive ability of the subject when the subject performs the emotional cognitive interactive task for each type of emotion to be identified, thereby improving the accuracy of the emotional cognitive ability evaluation.

[0016] Additional advantages, objects, and features of the application will be set forth in part by the description that follows, and will become apparent to those skilled in the art upon examination of the following or can be learned by practice of the application. The objects and other advantages of the application can be realized and attained by the structure particularly pointed out in the written description and claims hereof as well as the appended drawings.

[0017] It will be understood by those skilled in the art that the objects and advantages of the present application can be realized and attained by the structure particularly pointed out in the written description and claims hereof as well as the appended drawings. BRIEF DESCRIPTION OF DRAWINGS

[0018] The accompanying drawings, which are included to provide a further understanding of the application and are incorporated in and constitute a part of this application, illustrate embodiments of the application and together with the description serve to explain the principles of the application. In the drawings: Figure 1 A process diagram for emotion recognition of a subject in a human-computer interaction according to an embodiment of the present application.

[0019] Figure 2 A process diagram for emotion recognition of a subject in a human-computer interaction according to an embodiment of the present application.

[0020] Figure 3 A process diagram for emotion recognition of a subject in a human-computer interaction according to an embodiment of the present application.

[0021] Figure 4 A process diagram for emotion recognition of a subject in a human-computer interaction according to an embodiment of the present application.

[0022] Figure 5 A process diagram for emotion recognition of a subject in a human-computer interaction according to an embodiment of the present application.

[0023] Figure 6 A process diagram for emotion recognition of a subject in a human-computer interaction according to an embodiment of the present application.

[0024] Figure 7 A process diagram for emotion recognition of a subject in a human-computer interaction according to an embodiment of the present application.

[0025] Figure 8 A process diagram for emotion recognition of a subject in a human-computer interaction according to an embodiment of the present application.

[0026] Figure 9 A process diagram for emotion recognition of a subject in a human-computer interaction according to an embodiment of the present application.

[0027] Figure 10 A process diagram for emotion recognition of a subject in a human-computer interaction according to an embodiment of the present application. DETAILED DESCRIPTION

[0028] In order to make the objects, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the embodiments and the accompanying drawings. Herein, the illustrative embodiments of the present application and the descriptions thereof are used to explain the present application but do not limit the present application.

[0029] It should be noted that, in order to avoid the present application being obscured by unnecessary details, only the structures and / or processing steps closely related to the solutions according to the present application are shown in the accompanying drawings, and other details not closely related to the present application are omitted.

[0030] It should be emphasized that the term "comprising" as used herein is used to mean that features, elements, steps or components that are listed are present, but does not exclude the presence or addition of one or more other features, elements, steps or components.

[0031] Hereinafter, embodiments of the present application will be described with reference to the accompanying drawings. In the drawings, the same reference numerals represent the same or similar components, or the same or similar steps.

[0032] Immersive interaction technology is a human-computer interaction method aiming to provide users with a more natural, intuitive and immersive experience. With the help of immersive interaction technology, the implementation of emotional cognitive assessment of subjects has become an important part of research. In this application, the process of emotional cognitive assessment of subjects with the help of immersive interaction technology can be referred to as the process of subjects performing emotional cognitive interaction tasks (hereinafter referred to as interaction tasks). Moreover, subjects can use virtual reality (VR), augmented reality (AR) and mixed reality (MR) and other technologies to implement emotional cognitive interaction tasks with the help of immersive interaction technology. The present application does not specifically limit the specific type of immersive interaction technology. For example, VR helps to induce cognitive responses and assess cognitive abilities in a more realistic behavioral context.

[0033] However, the current emotional cognitive assessment process based on immersive interaction technology has the following defects: ① The single modal interaction data acquisition method makes the collected data relatively simple and the analysis limited, and it is impossible to comprehensively evaluate the emotional cognitive ability of the subjects; ② The existing emotional stimulation paradigm relies too much on facial expression recognition or film emotional induction, and lacks standardized induction tasks designed according to the cognitive characteristics of the subjects, which limits the accuracy and practicality of cognitive assessment; ③ The interaction difficulty is relatively single, which cannot fully activate the cognitive ability of the subjects.

[0034] Based on this, the present application designs to synchronously collect the eye movement data and the electroencephalogram data of the subjects in the interaction process, realizes multi-modal data fusion analysis, and fully excavates the cooperative correlation mechanism between emotional cognitive ability and brain-electro-ocular dual modal signals; Moreover, the present application innovatively designs a body movement and eye movement induction paradigm, which eliminates the interference of facial expressions by making certain movements of a virtual character without expression and without clothes, and purifies the emotional cognitive behavior of the subjects on the movement; Further, the present application designs a virtual reality interaction induction task with multiple difficulty gradients, which can adjust the cognitive difficulty level of the interaction task (the cognitive difficulty level can also be referred to as the difficulty level or the difficulty in this application) in multiple rounds of interaction by collecting the interaction results in real time, so as to fully activate the cognitive ability of the subjects and realize high-precision emotional cognitive assessment.

[0035] That is, asFigure 1 As shown, in order to solve the problems in the existing emotion recognition evaluation process, the present application can design to realize accurate evaluation of emotion recognition ability through the following multiple stages: in the fixation induction paradigm (i.e. eye movement induction paradigm) design stage, expressionless body movements can be designed to effectively remove the confusion factors of facial emotions, so as to construct a mapping model of the emotions to be recognized corresponding to the body movements, that is, to design expressionless body movement paradigm to induce emotion recognition of subjects; in the interaction stage, the immersive interaction environment and multiple difficulty emotion recognition interaction tasks designed are used to determine the difficulty of the emotion recognition interaction tasks and perform emotion recognition evaluation; in the data capture stage, eye movement data and electroencephalogram data of the subjects during the execution of the emotion recognition interaction tasks are collected, as well as the emotion recognition results of the subjects; in the data preprocessing stage, the data obtained in the data capture stage is processed; in the adaptive adjustment stage, the cognitive difficulty level of the emotion recognition interaction tasks is re-determined according to the captured data, and the interaction is performed again in the interaction stage. Through the above multiple stages, not only the closed-loop interaction task difficulty adaptive adjustment process can be realized, but also by using the immersive interaction technology to integrate the body movement fixation induction paradigm, the emotion recognition interaction task and the multi-modal interaction data collection, the shortcomings of traditional scales and static test methods can be effectively made up.

[0036] As an example, as shown in Figure 2 The task situation and body movement designed in the fixation induction paradigm design stage and the interaction stage are presented to the subjects through the interaction end, so that the subjects perform the interaction task to detect emotion recognition; the interaction results of the subjects can be collected in the data capture stage through the eye movement collection device and the electroencephalogram collection device; after the interaction results are filtered, data aligned and feature extracted, the multi-level task difficulty can be adjusted in combination with the task adaptive adjustment mechanism, so as to fully activate the cognitive ability of the subjects.

[0037] In some embodiments of the present application, considering that the cognitive difficulty level of the emotion-cognition interactive task is adjusted according to the data collected when the subject performs the emotion-cognition interactive task (i.e., the interaction result corresponding to the emotion-cognition interactive task) in the adaptive adjustment stage, for the first round of emotion-cognition interactive task performed by the subject, the difficulty of the first round of emotion-cognition interactive task cannot be adaptively adjusted according to the interaction result, and the difficulty of each round of emotion-cognition interactive task after the first round performed by the subject can be adaptively determined according to the interaction result of the last round. Therefore, for different rounds of emotion-cognition interactive tasks performed by the subject, the adaptive difficulty determination method of the emotion-cognition interactive task proposed in the present application performs different steps: for the first round of interactive task, the method proposed in the present application includes an emotion-cognition difficulty determination step; for each round of interactive task after the first round, the method proposed in the present application includes an online model updating step and an emotion-cognition difficulty determination step. The interaction result corresponding to the emotion-cognition interactive task includes eye movement data, electroencephalogram data collected during the subject performing the interactive task, and the emotion-cognition result of the subject.

[0038] As an example, the emotion-cognition difficulty determination step aims to determine the cognitive difficulty level of the emotion-cognition interactive task, and the emotion-cognition interactive task in the present application aims to, in a specific scene in the task, present a specific body movement in the task to the subject to make the subject respond accordingly, so the emotion-cognition interactive task can be considered to be composed of a task context (the task context includes a scene and the subject's response to the task, and the result of the subject's response to the task can be used as the emotion-cognition result of the subject performing the emotion-cognition interactive task to realize subsequent adaptive adjustment) and a body movement. The above determination of the cognitive difficulty level of the emotion-cognition interactive task can be understood as the determination of the cognitive difficulty of the task context in the interactive task, and the determination of the cognitive difficulty of the body movement in the interactive task (i.e., the difficulty of the interactive task is determined based on the difficulty of the task context and the difficulty of the body movement, and the difficulty levels of the task context and the body movement in a round of interactive task can be different). That is, in combination with the task context and the body movement of multiple cognitive difficulty levels, the present application can design multiple interactive tasks with multiple cognitive difficulty levels, so as to realize the dynamic matching of the task complexity and the current cognitive state of the subject according to the adaptive adjustment mechanism, thereby improving the adaptability of the interactive process and the individual sensitivity of the evaluation.

[0039] Therefore, for each round of interaction task, the emotion cognition difficulty determination step can include: step S01, determining the difficulty of the task context in the current round of interaction task; step S02, determining the to-be-identified emotion corresponding to the current round of interaction task, and determining the difficulty of the body movement corresponding to the to-be-identified emotion in the current round of interaction task. After determining the difficulty of the interaction task in the adaptive adjustment phase of the current round, the corresponding task context and body movement of the determined difficulty level can be presented to the subject again in the interaction phase through the immersive interaction technology, so that the subject performs the next round of interaction task. Wherein, the subject is presented with a task context of a specific difficulty level and a body movement corresponding to the to-be-identified emotion each time the subject performs emotion cognition detection.

[0040] More specifically, for the first round of emotion cognition interaction task, the difficulty of the task context is predetermined; for each round of interaction task after the first round, the difficulty of the task context in the current round of interaction task is determined based on the interaction result corresponding to the last round of interaction task (which can also be referred to as the last round of interaction result) and the difficulty adjustment probability model of the current round, that is, for each round of interaction task after the first round, the input of the difficulty adjustment probability model of the current round can be obtained based on the last round of interaction result, the adjustment probability of the difficulty of the task context can be output by the difficulty adjustment probability model of the current round, and the difficulty of the task context in the current round of interaction task is determined based on the adjustment probability and the difficulty of the task context in the last round of interaction task, and then the task context in the current round of interaction task is determined.

[0041] As an example, the interaction result of the last round can be mapped to corresponding features, and the mapped features are input into the difficulty adjustment probability model to output the possibility of task context difficulty decreasing, increasing or remaining unchanged. The present application does not specifically limit the form of data input into the difficulty adjustment probability model, and the above-mentioned features are only examples. Moreover, the input of the difficulty adjustment probability model can be obtained based on only the eye movement data and the electroencephalogram data in the interaction result, or can be obtained based on the eye movement data, the electroencephalogram data and the emotion cognition result in the interaction result, and the present application does not specifically limit the type of data input into the difficulty adjustment probability model, which can be determined according to training requirements. In addition, the present application does not limit the model type of the difficulty adjustment probability model, for example, the difficulty adjustment probability model can be constructed based on a classifier architecture, or can be constructed based on a deep learning network.

[0042] Moreover, for each round of interactive task after the first round, the difficulty adjustment probability model can output a first probability of difficulty reduction, a second probability of no change in difficulty, and a third probability of difficulty increase, and the difficulty adjustment mode with the largest probability value is selected to adjust the difficulty of the task context in the previous round of interactive task, to obtain the difficulty of the task context in the current round of interactive task. For example, if the difficulty adjustment probability model outputs a first probability of 0.8, a second probability of 0.6, and a third probability of 0.9, then the difficulty of the task context in the previous round of interactive task is selected to be increased by one difficulty level, and the difficulty of the task context in the current round of interactive task is obtained.

[0043] Further, after determining the difficulty of the task context in the current round of interactive task, the determination of the task context in the current round of interactive task mentioned in the present application can be to select a task context of a specific difficulty level from a pre-set task context library containing multiple cognitive difficulty level task contexts, or to generate a task context of a corresponding difficulty level using an AI model, or to improve a task context of a specific difficulty level based on an existing task context. For example, taking a task context library containing multiple difficulty level task contexts as an example, since only one specific difficulty level task context needs to be presented to the subject during human-computer interaction, if there is only one corresponding task context for a difficulty level in the task context library, then the task context in the current round of interactive task can be determined after determining the difficulty level of the task context; if there are multiple corresponding different task contexts for a difficulty level in the task context library, then a task context can be selected from multiple different task contexts of the same difficulty level as the task context in the current round of interactive task after determining the difficulty level of the task context. That is, the method proposed in the present application only specifically explains the determination method of the difficulty level of the task context, and the process of determining the task context under the condition of known difficulty level is not specifically limited in the present application.

[0044] In some embodiments of the present application, considering that the task context includes a scene and a subject response task, the difficulty level of the task context can be determined based on the difficulty level of the scene and the difficulty level of the subject response task. For example, an example of the task context can be as shown in Table 1, wherein the difficulty level L1 of the first task context < the difficulty level L2 of the second task context < the difficulty level L3 of the third task context < the difficulty level L4 of the fourth task context, and the first decibel level < the second decibel level < the third decibel level.

[0045] Table 1: Example of multiple difficulty level task contexts As an example, the task context example in Table 1 can be specifically presented in the process of human-computer interaction as: For the first task situation of L1 difficulty level, the scene is an indoor living room without background interference factors, and the subject response task is to identify emotions after watching specific limb action animations; for the second task situation of L2 difficulty level, the scene is an indoor living room with background playing of 60dB±3dB soothing piano solo interference, and the subject response task is to identify emotions after watching specific limb action animations; for the third task situation of L3 difficulty level, the scene is an indoor living room with background playing of 65dB±3dB simulated neighbor white noise interference, and the subject response task is to identify emotions and make responses after watching specific limb action animations; for the fourth task situation of L4 difficulty level, the scene is an outdoor forest park (with background playing of 70dB±3dB environmental noise and random appearance of 3-5 passers-by as visual flow interference), and the subject response task is to identify emotions and make responses after watching specific limb action animations. For example, the response in the subject response task can specifically refer to the behavior response of the subject, such as positive response (hugging or comforting, etc.), negative response (running away or avoiding, etc.), or neutral response (observation or friendly exchange, etc.).

[0046] Among them, the purpose of setting background noise interference in the scene of the present application is to trigger specific brain electrical response, and the double environment selection of indoor living room and outdoor forest park can make the subject perform dynamic fixation. Moreover, when performing the subject response task, the subject can select the corresponding emotion to be identified through pre-set emotion identification options (as shown in Figure 3 and Figure 4 ), and select the corresponding behavior response through pre-set response options (as shown in Figure 5 ), and the emotion identification and response can also be performed through other ways such as text input, and the present application is not limited thereto. In addition, in order to further increase the difficulty of the task situation, the duration of the response window in the task situation of L3 level can be set to 2.0s±0.2s, and the duration of the response window in the task situation of L3 level can be set to 1.8s±0.2s.

[0047] In the present application, a task situation including multiple scenes (including an indoor living room and an outdoor forest park) can be constructed based on the Unity engine, and then presented by means of immersive interaction technology. Moreover, in order to improve the friendliness of the user interaction interface, for the interactive environment of the indoor living room, the following setting parameters can be used: the initial user viewpoint is fixed at the virtual sofa position (for example, spatial coordinates x = 1.2 ± 0.1 m, y = 0.8 ± 0.1 m, z = 0.0 m), the central carpet area is a circular interaction area with a diameter of 1.5 m, and the indoor living room also includes interactive environmental element furniture such as a TV set, a tea table, and a wall painting, and the light intensity of the indoor living room can be set to 400 ± 50 lux; for the interactive environment of the outdoor forest park, the following setting parameters can be used: the interactive environment is set as an open area of 10 m x 10 m, the noise interference in the environment includes natural white noise such as conversation, bird song, and wind sound, and the visual flow interference includes dynamic vegetation and moving pedestrian NPCs, wherein the dynamic vegetation density of the outdoor forest park is ≥ 70%, the moving speed of the pedestrian NPCs can be a moving speed of 0.5-1.2 m / s, and the moving path of the pedestrian NPCs is controlled by means of NavMesh and the like.

[0048] In some embodiments of the present application, when designing the eye movement induced paradigm, multiple types of emotions to be recognized can be set, and facial expressionless body movements of different difficulty levels corresponding to each type of emotion to be recognized can be designed to realize adaptive difficulty adjustment of the body movements, that is, multiple types of emotions to be recognized can correspond to body movements of multiple difficulty levels, so that the subjects perform cognitive detection of different emotions. For example, the emotions to be recognized can include multiple types such as anger, happiness, disgust, fear, surprise, sadness, and neutrality, and the present application does not specifically limit the types of emotions to be recognized.

[0049] More specifically, when step S02 is performed in the adaptive difficulty adjustment phase, the emotion to be recognized corresponding to the current round of tasks needs to be determined first, and then the difficulty of the body movement corresponding to the emotion to be recognized is determined, and the body movement of the determined difficulty is taken as the body movement in the current round of interactive tasks. Therefore, the present application proposes that the emotion to be recognized corresponding to the current round of interactive tasks can be determined in the following manner: in order to provide sufficient emotional data support, the emotion to be recognized corresponding to the first round of interactive tasks can be all types of pre-set emotions to be recognized; for each round of interactive tasks after the first round, the emotion to be recognized corresponding to the current round of interactive tasks can be determined in accordance with the cognitive order of the pre-set types of emotions to be recognized. That is, the subject needs to respond to the body movements corresponding to all types of emotions to be recognized in a specific scene when performing the first round of interactive tasks; the subject needs to respond to the body movements corresponding to a type of emotion to be recognized in a specific scene when performing each round of interactive tasks after the first round.

[0050] The determining the to-be-recognized emotion corresponding to the current round of interaction task according to the preset cognitive order of each type of to-be-recognized emotion includes: if the difficulty levels of the task context and the body movement in the last round of interaction task are different from those in the first two rounds of interaction task, the to-be-recognized emotion corresponding to the current round of interaction task is the to-be-recognized emotion corresponding to the last round of interaction task (the to-be-recognized emotion does not change); if the difficulty levels of the task context and the body movement in the last round of interaction task do not change compared with those in the first two rounds of interaction task, the next emotion type of the to-be-recognized emotion corresponding to the last round of interaction task in the cognitive order is taken as the to-be-recognized emotion corresponding to the current round of interaction task; or starting from the second round of interaction task, after each round of interaction task is executed, the next to-be-recognized emotion in the cognitive order is automatically switched to as the to-be-recognized emotion corresponding to the next round of interaction task.

[0051] Further, after the to-be-recognized emotion of the current round is determined, the difficulty level of the body movement in the current round of interaction task can be determined: for the first round of emotion cognitive interaction task, the difficulty level of the body movement corresponding to the to-be-recognized emotion can be predetermined, and the difficulty levels of the body movements corresponding to all to-be-recognized emotions can be the same; for each round of interaction task after the first round, the difficulty of the body movement in the current round of interaction task can be determined based on the interaction result corresponding to the last round of interaction task and the baseline model of the current round (which can also be referred to as a cognitive baseline model), and then the body movement in the current round of interaction task is determined.

[0052] In some embodiments of the present application, for each round of interaction task after the first round, the process of determining the difficulty of the body movement corresponding to the emotion to be identified based on the interaction result and the baseline model is as follows: ① According to the error feedback compensation mechanism, if the emotion recognition result in the last round of interaction result does not meet the set condition, the body movement with lower difficulty than the body movement in the last round of interaction task is selected as the body movement corresponding to the emotion to be identified in the current round of interaction task; ② If the emotion recognition result in the last round of interaction result meets the set condition, the pre-baseline obtained based on the last round of interaction result is compared with the current round baseline obtained based on the baseline model of the current round, if the pre-baseline is less than the current round baseline, the body movement B with higher difficulty than the body movement A in the last round of interaction task is selected as the body movement corresponding to the emotion to be identified in the current round of interaction task, otherwise the body movement C with higher difficulty than the body movement B is selected as the body movement corresponding to the emotion to be identified in the current round of interaction task (the difficulty level of the body movement C (the second difficulty level) > the difficulty level of the body movement B (the first difficulty level) > the difficulty level of the body movement A). Wherein, the pre-baseline can be obtained by extracting features from the eye movement data, electroencephalogram data and emotion recognition result in the last round of interaction result, and then assigning a preset weight to each extracted feature, and then calculating the weighted result; the current round baseline can be obtained by inputting the last round of interaction result into the current round baseline model, and then outputting the result through the baseline model of the current round. That is, the pre-baseline is a baseline calculated according to the interaction result, and the current round baseline is a baseline output by the baseline model of the current round according to the interaction result.

[0053] For example, if the emotion recognition result in the last round of interaction result meets the set condition, the difficulty of the body movement can be dynamically selected according to the neurophysiological indicators collected during the execution of the last round of interaction task according to the neural indicator adaptation mechanism: if the pre-baseline obtained from the neurophysiological indicators is lower than the current round baseline, the first body movement with higher difficulty than the body movement in the last round of interaction task is selected, if the pre-baseline is not lower than the current round baseline, the second body movement with higher difficulty than the first body movement is selected.

[0054] It is worth noting that when determining the difficulty of the body movement in the current round of interaction task in step S02, the type of the emotion to be identified corresponding to the last round of interaction task and the current round of interaction task is not considered, and only the difficulty level of the body movement in the last round of interaction task is used to determine the difficulty level of the body movement in the current round of interaction task. Moreover, the above-mentioned method of determining the body movement in the current round of interaction task only specifically illustrates the determination method of the difficulty level of the body movement, that is, the process of determining the body movement under the condition of known difficulty level is not specifically limited by the present application.

[0055] As an example, the above-mentioned setting condition can be that the emotion recognition result accuracy is higher than a set threshold, and the application does not specifically limit the setting condition, which can be designed by the emotion recognition result itself. Similarly, the application does not specifically limit the data form and data type of the input baseline model, which can be determined according to the training requirements. In addition, the application does not limit the model type of the baseline model, for example, the baseline model can be constructed based on the Transformer architecture, or can be constructed based on a deep learning network.

[0056] In some embodiments of the application, the application can construct a mapping model of the to-be-recognized emotion-limb action through an eye movement induction paradigm dedicated to emotional cognition, so as to use expressionless limb actions as emotional induction media, and then capture the interaction results of the subjects performing emotional cognition tasks. Moreover, the mapping model of the to-be-recognized emotion-limb action can be designed based on Blender in the design stage of the fixation induction paradigm. In addition, in human-computer interaction, the virtual character can express anger by using strong limb conflict actions, express happiness by using happy limb language, express disgust by using repulsive actions, express fear by using defensive actions, express surprise mainly by arm actions, express sadness mainly by actions such as covering the face and avoiding, and express neutral emotions by actions with no obvious emotional tendency.

[0057] According to the motion posture of the virtual character, the application proposes that the limb action design can be performed according to the region of interest (ROI) shown in Table 2. In the application, the ROI is defined as the motion key points of the head (H), left arm (LH), right arm (RH), left leg (LL), and right leg (RL), and the motion trajectory buffer thereof.

[0058] Table 2: Correspondence table of each type of to-be-recognized emotion and ROI, typical fixation mode In addition, as shown in Table 2, the application also designs a mapping relationship between the emotion type and the motion observation mode, which can be performed in the eye movement tracking process. That is, the subjects can exhibit a specific ROI attention mode for different types of to-be-recognized emotions, for example, the angry emotion mainly causes rapid saccades (frequency > 3 times / s) to the arm region, the happy emotion generates a circular scanning trajectory in the leg region, and so on. The specific saccade speed and fixation density quantitative indicators can be designed to design the typical fixation mode.

[0059] The expression of the saccade speed is: ; wherein, total number of sampling points representing saccade path, representing the eye movement data point, representing the Euclidean distance or displacement between adjacent sampling points, representing the time interval between adjacent sampling points.

[0060] gaze density The expression is: ; wherein, representing the total gaze time of the subject in the ROI corresponding to the emotion to be identified, representing the total gaze time of the subject when performing the interactive task, representing the area of the entire visual scene during human-computer interaction, representing the area of the ROI.

[0061] As an example, only one specific difficulty level of limb movement needs to be presented to the subject during human-computer interaction, even if there are multiple different limb movements at one difficulty level. One limb movement is selected as the limb movement presented to the subject in the current round of interactive task. Moreover, similar to the task context, after determining the difficulty of the limb movement in the current round of interactive task, the determination of the limb movement in the current round of interactive task mentioned in the present application can be obtained based on the limb movement library, AI model or existing limb movement. For example, during the design phase of the gaze induction paradigm, by quantitatively mapping the kinematic parameters (such as speed, movement amplitude and movement symmetry, etc.) of each type of emotion to be identified and the corresponding movement, multiple difficulty levels of limb movements shown in Table 3 can be designed for each type of emotion to be identified, so that through immersive interaction technology, the subject is presented with the limb movement animation in Table 3 (such as each limb movement lasting 25.0s±4s). In Table 3, each type of emotion to be identified corresponds to 4 standardized movement templates.

[0062] Table 3: Examples of limb movements corresponding to each type of emotion to be identified The above-mentioned designed limb movements are only examples, and the present application does not limit the types of emotions to be identified, the number of limb movements of different difficulty levels corresponding to each emotion to be identified, and the specific limb movement animation. It can be designed according to the interactive task.

[0063] In some embodiments of the present application, as shown in Figure 6 Table 1, taking the task context difficulty level in Table 1 and the limb movement difficulty level in Table 3 as an example, the process of determining the difficulty level of the interactive task can be: For the first round of interaction task, the to-be-identified emotions are seven types of anger, happiness, disgust, fear, surprise, sadness and neutral, the first task context L1 (the subject performs emotion recognition in a background-free indoor living room) is selected, and the difficulty level L2 of the body movement corresponding to each type of to-be-identified emotion is selected. In the selected task context, the subject is sequentially shown the body movements corresponding to each type of to-be-identified emotion (the presentation order of each type of to-be-identified emotion is designed by the subject, for example, the order is anger → sadness → neutral → joy → surprise → fear → disgust), and each body movement animation can be displayed for 25.0±4s, as shown in the selection interface shown in the following figure. Figure 3 The selection interface can pop up after the action is completed, allowing the subject to make a selection, and while the subject is watching the body movement animation, 14-channel EEG signals can be collected at a sampling rate of 256Hz, and gaze point distribution patterns (i.e., eye movement data and brain electrical data of the subject are collected synchronously, and the subject's response is recorded) are recorded at a frequency of 90Hz. After each type of to-be-identified emotion corresponding body movement is displayed, the second round of interaction task is started. In addition, the baseline model and the difficulty adjustment probability model corresponding to the second round of interaction task are respectively updated based on the pre-constructed initial baseline model and the initial difficulty adjustment probability model.

[0064] For the second round of interaction task, the to-be-identified emotion of the current round (assuming anger) is determined according to the cognitive order of the emotion, and the baseline model and the difficulty adjustment probability model are updated based on the interaction results of the previous round; if the accuracy rate of the emotion cognitive result of the first round reaches the set threshold, the pre-baseline obtained according to the neural index adaptation mechanism is compared with the current round baseline obtained based on the baseline model, if the pre-baseline is lower than the current round baseline, the body movement corresponding to the anger emotion with a difficulty level of L3 is selected, if the pre-baseline is not lower than the current round baseline, the body movement corresponding to the anger emotion with a difficulty level of L4 is selected; if the accuracy rate of the emotion cognitive result of the first round does not reach the set threshold, the body movement corresponding to the anger emotion with a difficulty level of L1 is selected. At the same time, the difficulty of the task context of the current round can also be determined based on the interaction results corresponding to the first round of interaction task and the difficulty adjustment probability model of the current round, and then the task context in the current round of interaction task can be determined according to Table 1. Finally, the human-computer interaction in the current round can be realized according to the selected task context and body movement.

[0065] After the second round of interaction task is completed, if the emotion sequence is not completed, the adaptive detection round is repeated until the emotion sequence is completed, and finally the interaction results of the subject in all adaptive detection rounds can be obtained.

[0066] As an example, the difficulty level adjustment formula of the task context or the body movement can be expressed as follows: ; wherein, This indicates the difficulty level of the physical actions or task situation in the previous round of tasks. This indicates the difficulty level corresponding to the physical actions or task context in the current round of tasks. The first adjustment condition can be that the probability value of the difficulty increasing output by the difficulty adjustment probability model is maximized, or the emotional cognition result in the previous round of interaction meets the set conditions; the second adjustment condition can be that the probability value of the difficulty decreasing output by the difficulty adjustment probability model is maximized, or the emotional cognition result in the previous round of interaction does not meet the set conditions; other conditions can indicate that the probability value of the difficulty remaining unchanged output by the difficulty adjustment probability model is maximized.

[0067] Furthermore, when adaptively adjusting the difficulty level of the task context or physical actions, if it is determined that the difficulty level of the current round is lower than the difficulty level of the previous round, and the difficulty level of the previous round is the lowest difficulty level (or, if it is determined that the difficulty level of the current round is higher than the difficulty level of the previous round, and the difficulty level of the previous round is the highest difficulty level), then the difficulty level of the current round can be the same as the difficulty level of the previous round.

[0068] During each round of the interactive task, the participants' eye-tracking patterns, EEG data updates, and behavioral responses can be monitored in triple parallelism. Furthermore, the interaction results collected during the task can be preprocessed to determine the participants' cognitive load at different task difficulties, enabling closed-loop feedback to adjust the task's difficulty. The steps for collecting interaction results in the data capture phase and processing them in the data preprocessing phase are as follows: Figure 7 As shown, specifically: eye-tracking data of the subject during the interactive task is collected in real time using an eye-tracking acquisition device (such as a VR device or eye tracker), and electroencephalogram (EEG) data of the subject during the interactive task is collected in real time using a head-mounted EEG device with a high channel count. After filtering and noise reduction, the eye-tracking data and EEG data are timestamped, and finally, the corresponding eye-tracking features and EEG features can be extracted. For example, the eye-tracking subsystem (ETS) built into the PICO 4Pro head-mounted display can be used to achieve immersive induction and interaction detection of virtual character limb movements, recording the subject's gaze data at each ROI of the virtual character at a sampling rate of 120Hz, and a 14-channel EEG signal can be collected using an Emotiv EPOC+ device.

[0069] In some embodiments of the present invention, eye-tracking data represents changes in gaze position over time. Analyzing eye-tracking data can yield information such as eye-tracking trajectory, first fixation latency, saccade speed, and ROI switching frequency. Electroencephalogram (EEG) data represents the collected EEG frequencies over time. Emotional cognition results may include response time (including the time to complete recognition and the time to respond), emotion recognition results, and response results. Specific examples of emotion cognition results are shown in Table 4 and...Figure 8 as shown.

[0070] Table 4: Emotional cognition result diagram As an example, the interaction platform can receive the raw data packets transmitted by the eye movement collection device and the electroencephalogram collection device through the TCP / IP protocol (port number 8867), and split the data packets according to the format as shown in Table 4 to obtain the following field information: user identification field, detection information field (which can include detection timestamp, task difficulty level, subject emotional cognition result (which can be the encoding of user options), eye movement calibration data, eye movement point sequence, and EEG sequence. For example, the user identification field can be represented as a 16-byte hash value (SHA-256 first 128 bits), the precision of the detection timestamp can reach 1 ms, the eye movement calibration data can be the average error Eavg (Eavg≤0.12) of 6 calibration points, the eye movement point sequence includes multiple frames of data, and each frame of data includes gaze coordinates (resolution 1920x1080 pixels).

[0071] Since the eye movement data and the electroencephalogram data can be collected based on different sampling frequencies, a resampling strategy based on linear interpolation can be used to resample to the same time step to align the multi-modal data; and an EEG time axis and an eye movement time axis can also be established respectively, and the collected eye movement data and electroencephalogram data are aligned through a unified timestamp. For example, the timestamp can be generated by the same system clock or a unified external synchronization source to ensure the time consistency between the multi-modal data.

[0072] As shown in Figure 9 , to achieve high-precision eye movement tracking calibration, a six-point spatial calibration protocol (unit: screen ratio) can be performed before human-computer interaction to determine the overall error of the current device, and calibration can be performed according to the overall error. The specific process is as follows: Assuming that the coordinates of the 6 calibration points are represented as: Data collection and screening is performed for each calibration point at a sampling rate of 50 Hz for 3 seconds (a total of 150 samples), which can obtain a calibration data set . Therefore, the Euclidean distance between each sample and the corresponding calibration point can be calculated , so as to determine the valid sample data based on If , the sample is determined as valid sample data. After screening all valid sample data, a valid sample set can be formed.​​

[0073] Furthermore, utilizing each calibration point The corresponding set of valid samples The average error at each calibration point can be calculated. Therefore, the total error can be calculated: .

[0074] In some embodiments of the present invention, the eye movement characteristics and EEG frequency band representation of subjects interacting with a non-expressionable human figure model can be analyzed using a dynamic gaze-EEG coupling analysis algorithm. Specifically, the preprocessing procedure for the eye movement data is as follows: ① The raw eye-tracking data is smoothed using an exponential moving average filter (e.g., a smoothing coefficient of 0.25) to remove high-frequency jitter and obtain a smooth gaze point trajectory.

[0075] ② Synchronize the scene, camera, and body movements in the interactive platform and eye-tracking acquisition device through eye-tracking data mapping: Eye-tracking data collected by eye-tracking devices can be transformed into two-dimensional eye-tracking fixation data by projecting the direction vector onto the model plane after quaternion rotation transformation. ; ; in, A unit quaternion representing the direction the head is facing. Represents the forward unit vector. The camera position represents the coordinates of the head or eye movement origin. Distance to the projection plane The gaze direction vector after rotation by quaternion. These are the coordinates of the gaze point in a two-dimensional screen coordinate system.

[0076] The eye-tracking data is mapped onto screen coordinates using an eye-tracking data mapping algorithm, as shown in the following formula: ; in, Let be the projection matrix. For the camera's world coordinates, Based on and Obtained.

[0077] ③Based on the mapping points of eye-tracking data on a two-dimensional plane Calculate the intersection time between the mapping point and the 3D model within the task time. Furthermore, they statistically analyzed eye movement characteristics such as fixation trajectory, fixation duration, average saccade speed, fixation time percentage of each ROI, and pupil change rate.

[0078] For example, through time series analysis, the eye fixation time and fixation duration of the subject on each ROI of the virtual character are counted to obtain the fixation time proportion of each ROI : ; wherein, is the interaction time accumulation of the eye fixation point and each ROI region of the limb movement model, is the limb movement animation playing time of the virtual character.

[0079] ④ Count the two-dimensional heat map in the eye movement feature For the fixation coordinate data collected by the eye movement collection device, the original eye movement data is processed for outlier rejection, normalization and image mapping to generate a heat map for visualization or modeling analysis. The generation process of the heat map includes the following steps: (1) Outlier rejection: By ignoring the original data of the Z axis in the original eye movement data , and transforming the original data of the X axis and Y axis, the collected original eye movement data can be mapped to a two-dimensional plane to form a two-dimensional fixation point set . Among them, is the coordinate formed by mapping the original fixation point to a two-dimensional plane.

[0080] Calculate the mean and standard deviation of the original data of the X axis and Y axis in the original eye movement data: ; ; ; ; Based on the preset outlier judgment threshold ( which can be set according to experience), filter out the collection points that meet the following conditions: , and (2) Coordinate normalization: coordinate normalization processing is performed on the reserved effective point set: For example, set the margin parameter , and expand the maximum and minimum value range of the original eye movement data to: ; ; ; ; For each fixation point in the raw eye movement data , it is normalized to the interval according to the following formula: ; ; wherein , , and represent the coordinate boundaries.

[0081] (3) Map the coordinate data of the acquisition device coordinate system to the pixel space: The point set formed by normalization is mapped to a fixed resolution texture space for drawing a heat map, and the mapping formula is as follows: ; ; wherein represents the size of the heat map pixel space (for example, it can be 512x512 pixels), represents the floor function.

[0082] (4) Take each fixation point as the center, and perform heat decay superposition in the range determined by the heat decay coefficient , and the heat function can adopt the form of a two-dimensional Gaussian distribution, and the formula is as follows: ; wherein is the heat decay coefficient (the value of can be 0.05), is the coordinate of mapped to the heat map pixel space.

[0083] (5) Generate a two-dimensional heat matrix according to the heat values of all points in the heat map, and normalize it: ; wherein represents the two-dimensional heat matrix, represents the maximum value of the heat matrix at all positions .

[0084] (6) Map to RGB values (R channel linear, G / B channel nonlinear) using the Viridis color scale, and map the heat values to the RGB color space to generate a color heat map, as shown in Figure 10 .

[0085] A head-mounted EEG acquisition device was used to simultaneously record multi-channel EEG signals at a specific sampling rate (e.g., 256Hz). The preprocessing procedure for the EEG data is as follows: ① The acquired raw EEG signal data was denoised using a Chebyshev Type-II bandpass filter. During this process, the filter range could be set to 0.3~40Hz and the filter order to 5 to achieve a filtering effect that balances accuracy and computational efficiency.

[0086] ② Use a sliding window with a length of 20 seconds and an overlap rate of 80% to segment the denoised EEG signal and generate multiple signal segments.

[0087] ③ Extract the power spectral density of three frequency bands (ranges including α [4~7Hz], θ [8~13Hz], and β [14~30Hz]) from each EEG segment, and calculate the relative power spectral density (rPSD) of each frequency band to represent the subject's neural activity state, thereby achieving cross-subject feature standardization. The Welch method can be used to calculate PSD features, as shown in the following formula: ; in, For the k-th EEG signal segment, It is a Fast Fourier Transform. It is the total number of EEG signal segments obtained by the sliding window method. Represents frequency The power spectral density at that location.

[0088] Considering the significant differences in EEG amplitudes among different individuals, the relative power spectral density can be further extracted using the following formula: ; in, Indicates the first The relative power spectral density of each frequency band Indicates the total number of frequency bands.

[0089] Finally, the ratio of the relative power spectral density of the α band and the β band (Alpha / Beta, ABR) can be calculated, as well as the ratio of the relative power spectral density of the θ band and the β band (Theta / Beta, TBR), and ABR and TBR can be used as EEG features.

[0090] As an example, in the data capture and preprocessing stage, a data storage module for storing eye movement data, electroencephalogram data, eye movement features and electroencephalogram features can also be set, and data storage in the data storage module can be in the form of a hash table. The raw data collected by the eye movement device and the electroencephalogram device can be stored in CSV format, and the field information stored can include user identification information (such as user hash ID), timestamp, eye movement data and electroencephalogram data, and processed eye movement data and electroencephalogram data, such as EEG frequency band power (alpha / beta, beta / theta). In addition, an offline query module can also be set, which can quickly query the file path in the data storage module by username when using offline query historical detection data, and obtain local historical data according to the file path.

[0091] In some embodiments of the present application, the present application controls the interactive task by task context and body movement, and the mechanism can determine the cognitive load of the subject based on the interaction result in the adaptive stage, and then dynamically adjust the task difficulty level to stimulate the cognitive attention of the subject. In this process, the adaptive emotional cognitive classification model needs to be designed to analyze the cognitive difficulty of the subject. The adaptive emotional cognitive classification model includes a baseline model and a difficulty adjustment probability model, the baseline model is used to output a personalized cognitive baseline to adjust the difficulty level of the body movement, and the difficulty adjustment probability model is used to predict the difficulty level adjustment probability of the task context.

[0092] Before each round of interactive task execution, the baseline model and the difficulty adjustment probability model need to be updated to adapt to the cognitive ability of the subject. In order to distinguish the baseline model and the difficulty adjustment probability model used in the determination of the difficulty of different rounds of interactive tasks, the present application refers to the baseline model and the difficulty adjustment probability model used to determine the difficulty of the current round of interactive task as the baseline model of the current round and the difficulty adjustment probability model of the current round. For the first round of interactive task, since the difficulty level of the body movement and the task context is pre-set (without using the adaptive emotional cognitive classification model to determine), the baseline model of the first round and the difficulty adjustment probability model of the first round can be the initial baseline model and the difficulty adjustment probability model (which can be referred to as the initial probability model) constructed in advance; for each round of interactive task after the first round, the model parameters in the baseline model of the last round and the model parameters in the difficulty adjustment probability model of the last round can be updated based on the interaction result corresponding to the last round of interactive task to obtain the baseline model of the current round and the difficulty adjustment probability model of the current round, and for each round of interactive task after the first round, the difficulty of the body movement and the difficulty of the task context need to be determined by means of the updated baseline model and the updated difficulty adjustment probability model, therefore, before the emotion cognitive difficulty determination step is executed, the online model updating step needs to be executed to obtain the baseline model of the current round and the difficulty adjustment probability model of the current round.

[0093] As an example, the baseline model and the difficulty adjustment probability model can be updated only according to the electroencephalogram data in the last round of interaction results, or the model can be updated using the eye movement data and the electroencephalogram data in the last round of interaction results, or the model can be updated using the eye movement data, the electroencephalogram data and the emotional cognitive results of the subject in the last round of interaction results. The present application does not specifically limit the type of data used to update the model, and the cognitive judgment demand can be designed by itself.

[0094] Taking the eye movement data and the electroencephalogram data as examples, the logic regression weights of the baseline model and the difficulty adjustment probability model can be updated by the following online learning method according to the last round of interaction results, so as to gradually optimize the model and the individual decision threshold: ①The electroencephalogram features and the eye movement features in each time window are spliced to construct a tensor , and the tensor is normalized to obtain . For example, the Z-score method can be used to standardize the features to .

[0095] ②In order to balance the model generalization ability and the sample balance, the stratified strategy can be used when the training data is divided, so as to ensure that the feedback of each category is evenly distributed in the training set and the test set.

[0096] ③Model optimization (1) The linear combination of the electroencephalogram features and the eye movement features is mapped to the feature space to the category probability space, and for the multi-classification case, the multinomial logistic regression can be used. Assuming that the sample feature vector is , the category output by the difficulty adjustment probability model is , and the probability expression of the prediction of the th category is: ; wherein is the weight vector of the th category, is the bias term of the th category.

[0097] The category with the maximum prediction probability is selected as the difficulty level adjustment mode of the current round of task situation. The difficulty adjustment probability model of the current round can be obtained by the following training method: the cross-entropy loss function is used as the objective function, and it is innovatively integrated into the cost-sensitive mechanism (log loss), so the loss function can be expressed as: ; wherein, represents the number of samples, , is used to represent that the i-th sample does not belong to the j-th class, is used to represent that the i-th sample belongs to the j-th class, is the predicted probability that the i-th sample belongs to the j-th class output by the normalized exponential function.

[0098] (2) The baseline model of the current round may be represented as: ; wherein, and are parameters of the baseline model, represents the normalized eye movement feature, represents the normalized electroencephalogram feature.

[0099] Before the human-computer interaction of the current round, the parameters of the baseline model are updated according to the following formula, and the individualized action selection strategy can be further implemented based on the baseline model of the current round updated and the interaction result of the last round: ; ; wherein, and represent the parameters of the baseline model of the current round, and represent the parameters of the baseline model of the last round, and represent the parameters of the baseline model of the previous two rounds, and are learning rates, and represent the gradient vector of the loss function at the current parameter position.

[0100] The above-mentioned manner of updating the baseline model and the difficulty adjustment probability model is only an example, and any existing online updating of the two models can be used.

[0101] ​​​​​​As an example, since there is a certain linear separability between ABR, TBR and the task feedback of the subjects, the boundary decision processing can be performed by constructing a hierarchical logistic regression framework, which can include a feature enhancement layer, a regularization decision layer and a dynamic feedback layer. Among them, the feature enhancement layer is used to recalculate the features of the input baseline model and the difficulty adjustment probability model to enhance the characteristics between the features, for example, the original EEG features can be decomposed into multiple sub-frequency bands (such as α, β, θ, δ and γ bands) through wavelet transform, and the nonlinear coupling coefficients are calculated; the regularization decision layer is used to optimize the baseline model and the difficulty adjustment probability model by using elastic network regularization; the dynamic feedback layer is used to dynamically adjust the task difficulty according to the difficulty adjustment decision.

[0102] The emotion cognitive difficulty adaptive difficulty determination method proposed in the present application has the following significant advantages: ① Based on the interaction results obtained when the subjects perform the emotion cognitive interaction task, and the constructed baseline model and difficulty adjustment probability model, a complete task difficulty online adaptive adjustment mechanism is constructed: through real-time data acquisition of the interaction results corresponding to the task, intelligent discrimination and dynamic adjustment of the difficulty of the task situation and body movement in the interaction task, a closed-loop, individualized and dynamic response function cognitive training method is constructed, realizing the integration of "evaluation-training" adaptive intervention, and providing theoretical support and practical scheme for augmented reality interaction, brain-computer interface training and personalized learning system.

[0103] ② The task situation and body movement of multiple levels of difficulty are designed, so that the cognitive load of the subjects is dynamically maintained in the optimal region by adaptively adjusting the difficulty of the interaction task, and the eye movement behavior and EEG response mode of the subjects when responding to the change of cognitive load can be observed according to the interaction results, which helps to find the difference in cognitive flexibility, thereby improving the discrimination ability and sensitivity.

[0104] ③ Unlike existing facial expression stimuli, the present application realizes the precise induction of emotion cognitive ability, working memory and visual attention by designing the corresponding relationship between the body movement without facial expression and the additional prompt of emotional expression, thereby distinguishing from traditional static neuropsychological tests.

[0105] ④ In terms of extracting EEG features, strong difference frequency domain features are extracted, and a sliding window is used to construct time-stable multi-channel EEG features to improve the discriminability of the baseline model and the difficulty adjustment probability model; in terms of extracting eye movement features, various eye movement features can be combined to reflect the eye movement tracking situation.

[0106] Combined with the method and immersive interaction technology proposed in the present application, a multi-modal auxiliary diagnosis method with low cost, non-invasive and suitable for early screening of cognitive impairment population (such as mild cognitive impairment and early Alzheimer's disease population) can be further proposed to improve the early screening rate and intervention timeliness.

[0107] The diagnostic method has significant advantages in multi-modal data analysis, neurobehavioral feature extraction, diagnostic accuracy and interactive experience, and can provide a scientific, efficient and generalizable technical path for early diagnosis of cognitive impairment. Specifically, through the emotion cognitive difficulty self-adaptive difficulty determination method, the adaptive difficulty emotion cognitive interactive task of different subjects can be determined based on the innovative design of the expressionless limb movement induction paradigm, so as to activate the cognitive attention of the subjects in the subsequent interactive process; through the immersive interactive technology (such as virtual reality technology), the interactive results (including multi-channel EEG and eye movement data, etc.) of the subjects performing the emotion cognitive interactive task are collected in real time, so as to further assist the adaptive adjustment of the interactive task difficulty. Finally, through the analysis of the interactive results collected when the subjects perform the interactive task, efficient identification and risk assessment of the cognitive impairment population can be realized.

[0108] Compared with the traditional method, the diagnostic method not only improves the accuracy and sensitivity of diagnosis, but also has wider applicability. For example, the diagnostic method is suitable for clinical auxiliary screening (such as early cognitive impairment risk screening in primary hospitals and nursing homes), home rehabilitation monitoring (such as long-term tracking of cognitive state changes with training tasks), scientific research data collection (the diagnostic method supports standardized experimental tasks and reproducible data processing flow), and education and popularization applications (such as popularizing cognitive health assessment awareness for ordinary users).

[0109] Corresponding to the above method, the present application also provides an emotion cognitive difficulty self-adaptive determination system for interactive tasks, which comprises a computer device, the computer device comprising a processor and a memory, the memory storing computer programs / instructions, and the processor being configured to execute the computer programs / instructions stored in the memory, so that the system realizes the steps of the above method.

[0110] The embodiment of the present application also provides a computer readable storage medium, which stores computer programs / instructions, and the computer programs / instructions are executed by the processor to realize the steps of the above edge computing server deployment method. The computer readable storage medium can be a tangible storage medium, such as random access memory (RAM), memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, register, floppy disk, hard disk, removable storage disk, CD-ROM, or any other form of storage medium known in the art.

[0111] The embodiments of the present application also provide a computer program product, which stores a computer program / instruction. The computer program / instruction is executed by a processor to implement the steps of the aforementioned edge computing server deployment method. The computer program product can be a tangible product, such as a random access memory (RAM), a memory, a read only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a floppy disk, a hard disk, a removable disk, a CD-ROM, or any other form of product known in the technical field.

[0112] Those skilled in the art should understand that the exemplary components, systems and methods described in connection with the embodiments disclosed herein can be implemented in hardware, software or a combination thereof. The decision to implement in hardware or software depends on the particular application and design constraints imposed on the technical solution. Those skilled in the art can use different methods to implement the described functions for each particular application, but such implementation should not be considered beyond the scope of the present application. When implemented in hardware, it can be, for example, an electronic circuit, an application specific integrated circuit (ASIC), appropriate firmware, a plug-in, a functional card, etc. When implemented in software, the elements of the present application are program or code segments used to perform the required tasks. The program or code segments can be stored in a machine readable medium or transmitted through a data signal carried in a carrier wave over a transmission medium or communication link.

[0113] It should be noted that the present application is not limited to the specific configurations and processes described above and shown in the drawings. For the sake of brevity, detailed descriptions of known methods are omitted. In the above embodiments, several specific steps are described and shown as examples. However, the method processes of the present application are not limited to the specific steps described and shown, and those skilled in the art can make various changes, modifications and additions, or change the order of the steps, after understanding the spirit of the present application.

[0114] In the present application, the features described and / or illustrated for one embodiment can be used in the same way or in a similar way in one or more other embodiments, and / or in combination with or instead of features of other embodiments.

[0115] The above description is only preferred embodiments of the present application and is not intended to limit the present application. The embodiments of the present application can be variously changed and modified by those skilled in the art. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the scope of the present application.

Claims

1. An adaptive method for determining the difficulty of emotion cognition in interactive tasks, characterized in that, For the first round of interactive tasks, the method includes a step to determine the difficulty of emotion cognition; for each round of interactive tasks after the first round, the method includes an online model update step and a step to determine the difficulty of emotion cognition. The steps for determining the difficulty of emotion recognition are as follows: First, determine the difficulty of the task context and the corresponding emotion to be identified in the current round of interactive tasks. Second, determine the difficulty of the physical actions corresponding to the identified emotion in the current round of interactive tasks. Then, present the corresponding task context and physical actions to the participants using immersive interactive technology, and obtain the interaction results during the participants' execution of the current round of interactive tasks. Specifically, the difficulty of the task context and the difficulty of the physical actions corresponding to the identified emotion in the first round of interactive tasks are predetermined. For each subsequent round of interactive tasks, the difficulty of the task context is adaptively determined based on the interaction results of the previous round and the difficulty of the current round using an adjusted probability model. The difficulty of the physical actions is adaptively determined based on the interaction results of the previous round and the baseline model of the current round. The online model update process involves updating the baseline model and the difficulty adjustment probability model based on the interaction results of the previous interactive task, so that the current interactive task can be executed based on the updated baseline model and difficulty adjustment probability model. The interaction results of the previous interactive task include eye-tracking data and electroencephalogram data collected during the subject's execution of the previous interactive task, as well as the subject's emotional cognition results regarding the body movements in the previous interactive task. The baseline model and difficulty adjustment probability model for the second round are updated based on the pre-constructed initial baseline model and initial difficulty adjustment probability model, respectively.

2. The method according to claim 1, characterized in that, For each round of interactive tasks after the first round, the difficulty of the physical movements in the previous round of interactive tasks is determined in the following way: If the emotional cognition results in the previous round of interaction meet the set conditions, compare the pre-baseline obtained based on the eye movement data and EEG data in the previous round of interaction with the current round baseline obtained based on the updated baseline model. If the pre-baseline is less than the current round baseline, the difficulty of the limb movement corresponding to the determined emotion to be identified is the first difficulty level, which is higher than the difficulty of the limb movement in the previous round of interaction task; otherwise, the difficulty is the second difficulty level, which is higher than the first difficulty level. If the emotion recognition results in the previous round of interaction do not meet the set conditions, the difficulty of the body movements corresponding to the identified emotion is lower than the difficulty of the body movements in the previous round of interaction task.

3. The method according to claim 2, characterized in that, The pre-baseline is obtained by extracting features from the eye movement data and EEG data in the previous round of interaction results to obtain eye movement features and EEG features, and assigning preset weights to the eye movement features and EEG features respectively. The current round baseline is obtained by taking the eye movement data and EEG data from the previous round of interaction as input to the current round baseline model and outputting it through the current round baseline model.

4. The method according to claim 1, characterized in that, For each round of interactive tasks after the first round, the difficulty of the task situation in the current round of interactive tasks is determined in the following way: The input to the difficulty adjustment probability model for the current round is obtained based on the eye-tracking and EEG data from the previous round of interaction results. The difficulty adjustment probability of the task situation is obtained by outputting the difficulty adjustment probability model for the current round. The difficulty of the task situation in the current round of interaction is determined based on the adjustment probability and the difficulty of the task situation in the previous round of interaction.

5. The method according to claim 1, characterized in that, The emotion to be identified for the current round of interaction task is determined in the following way: For the emotions to be identified in the first round of interactive tasks, all preset types of emotions to be identified are included. For each subsequent round of interactive tasks, the emotion to be identified for the current round of interactive tasks is determined according to the pre-defined cognitive order of the various types of emotions to be identified.

6. The method according to claim 5, characterized in that, The emotions to be identified include the following types: anger, happiness, disgust, fear, surprise, sadness, and neutrality; The corresponding body movements for each type of emotion to be identified are designed according to the following correspondence: For the emotion to be identified as anger, the corresponding regions of interest are the arms and legs; For the emotion to be identified as "happy", the corresponding region of interest is the legs; For the emotion to be identified as aversion, the corresponding regions of interest are the head and core torso. For the emotion to be identified as fear, the corresponding regions of interest are the arms and legs; For the emotion to be identified as surprise, the corresponding regions of interest are the arms and head; For the emotion to be identified as sadness, the corresponding region of interest is the head; For neutral emotions to be identified, the corresponding regions of interest are the arms, legs, head, and core torso.

7. The method according to claim 1, characterized in that, The difficulty of the task scenario is determined based on the difficulty of the scenario and the difficulty of the subjects responding to the task; The task scenarios presented to the participants were selected from a database of task scenarios with varying levels of difficulty, based on the determined difficulty of the task scenarios. The task scenario library includes the following four task scenarios: For the first task scenario, the scene was an indoor family living room without background interference, and the subject's response task was to identify emotions based on specific body movements. For the second task scenario, which is more difficult than the first task scenario, the scene is an indoor family living room with background noise interference at the first decibel level. The subject's response task is to identify emotions based on specific body movements. For the third task scenario, which is more difficult than the second task scenario, the scene is an indoor family living room with background noise interference at the second decibel level. The participants' response task is to identify emotions based on specific body movements and respond accordingly. For the fourth task scenario, which is more difficult than the third task scenario, the scene is an outdoor forest park with background noise interference of level 3 and visual motion interference. The subject's response task is to identify emotions based on specific body movements and make a response. Among them, the level 3 is greater than the level 2, and the level 2 is greater than the level 1.

8. An adaptive system for determining the difficulty of emotion cognition in interactive tasks, comprising a processor, a memory, and a computer program / instructions stored in the memory, characterized in that, The processor is configured to execute the computer program / instructions, and when the computer program / instructions are executed, the system implements the steps of the method as described in any one of claims 1 to 7.

9. A computer-readable storage medium having a computer program / instructions stored thereon, characterized in that, When the computer program / instructions are executed by the processor, they implement the steps of the method as described in any one of claims 1 to 7.

10. A computer program product comprising a computer program / instructions, characterized in that, When the computer program / instructions are executed by the processor, they implement the steps of the method as described in any one of claims 1 to 7.

Images