AR (Augmented Reality) adaptive scene memory training system and method based on electroencephalogram neural feedback

Abstract

The invention provides an AR (Augmented Reality) adaptive scene memory training system and method based on electroencephalogram neural feedback. The system comprises head display equipment and an EEG (Electroencephalogram) data acquisition module, the data processing and prediction module is used for outputting a memory state prediction result of the current test EEG to the self-adaptive adjustment and feedback module; the self-adaptive adjustment and feedback module is used for judging the memory performance of the user based on the familiarity and recall index in the memory state prediction result of the current test round, determining the required task scene difficulty parameter and the mnemonic strategy according to the memory performance of the user, and finally generating an adjustment instruction according to the task scene difficulty parameter and the mnemonic strategy; and a Unity 3D engine platform. According to the invention, a set of self-adaptive scene memory training system is constructed based on EEG (electroencephalogram) and AR (augmented reality) technologies, and the defects of low ecological efficiency, lack of real-time feedback and difficulty in personalized adjustment of an existing scene memory intervention system are effectively overcome.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of cognitive training, and particularly relates to an AR adaptive episodic memory training system and method based on electroencephalogram neural feedback. BACKGROUND

[0002] With the aggravation of population aging, the prevalence of Mild Cognitive Impairment (MCI) and Alzheimer's Disease (AD) is increasing year by year. MCI is considered to be a prodromal stage of AD, and about 10%-15% of MCI patients will progress to AD each year. Episodic memory impairment is one of the most common early symptoms of MCI, and is closely related to the degenerative changes of hippocampus and its associated brain areas. Therefore, early identification and intervention of episodic memory impairment is of great significance for delaying the progression of AD. At present, cognitive training is one of the main non-drug intervention methods for improving the memory function of MCI patients. Traditional cognitive training methods include memory strategy training (such as location method, associative memory, etc.) and neural regulation technology (such as transcranial magnetic stimulation TMS). However, these methods have obvious limitations in terms of individual adaptation, ecological validity and real-time feedback.

[0003] In recent years, Augmented Reality (AR) technology and Electroencephalography (EEG) neurofeedback technology have shown great potential in the field of cognitive training. AR technology can provide a high ecological validity training environment, simulate real-life scenarios, and improve user immersion and engagement. EEG technology is widely used in memory assessment and neurofeedback training due to its real-time and neural sensitivity. Previous studies have found that the EEG neural oscillation patterns during the encoding stage, such as increased gamma power, alpha suppression, and prefrontal-hippocampal phase synchronization, are closely related to subsequent memory success rates and have been used to predict individual memory performance in specific situations. This differential neural activity that characterizes the encoding process and its impact on subsequent memory is known as the subsequent memory effect (SME). SME research not only helps identify critical neural mechanisms during the encoding process but also provides mechanistic support for personalized memory training based on neural measurements. At the characterization level, several studies have successfully used machine learning algorithms (such as SVM and random forests) to classify and model encoding period EEG signals, achieving memory state prediction. At the intervention level, based on SME research findings, some studies have explored how to optimize encoding strategies during the encoding phase to enhance the subsequent memory effect, including context cues, deep semantic processing, and attention guidance to maximize information storage in long-term memory. However, most of these studies remain at the offline analysis stage and have not yet formed a complete "prediction-feedback-retraining" closed-loop intervention process.

[0004] In summary, the defects existing in the prior art are: 1. The closed-loop intervention of EEG neurofeedback for episodic memory is missing: 1) Neurofeedback and memory training are separated. Existing methods usually separate neurofeedback training (such as Theta band regulation) from memory tasks, and only evaluate the effect by comparing before and after training, lacking real-time monitoring and dynamic feedback of the memory encoding process. For example, the study by Berner et al. (2020) used offline neurofeedback training (up-regulating Sigma / Beta activity) and evaluated memory performance before and after training, but could not optimize the intervention strategy in real time during the memory task execution. 2) Lack of closed-loop regulation based on memory state: Traditional neurofeedback only focuses on the power regulation of specific frequency bands (such as Theta, Alpha), and does not combine the subsequent memory effect (SME) to predict the success or failure of a single memory trial in real time, resulting in a disconnection between feedback and memory encoding needs. 2. The deficiencies of EEG neurofeedback combined with AR: 1) The AR episodic memory paradigm is not perfect: Existing AR memory training systems mostly transplant traditional working memory tasks (such as N-back), lacking design for episodic memory characteristics (such as spatiotemporal binding, multi-modal encoding). 2) Neurofeedback is not deeply integrated into AR interaction: Most AR systems only use EEG as a passive monitoring tool, and do not use real-time neurofeedback to dynamically adjust AR scene parameters (such as object complexity, spatial distribution). SUMMARY

[0005] The purpose of the present application is to provide an AR adaptive episodic memory training system and method based on EEG neurofeedback, effectively solving the defects of low ecological validity, lack of real-time feedback, and difficulty in personalized adjustment existing in existing episodic memory intervention systems. The technical solutions adopted are as follows:

[0006] An AR adaptive episodic memory training system based on EEG neurofeedback, comprising:

[0007] a head-mounted device;

[0008] an EEG data acquisition module for real-time acquisition of EEG signals and real-time transmission to a data processing and prediction module;

[0009] a data processing and prediction module for processing and classifying EEG signals, and outputting memory state prediction results of the current trial EEG signals to an adaptive adjustment and feedback module;

[0010] The adaptive adjustment and feedback module judges the memory performance of the user based on the familiarity and recall degree indicators in the memory state prediction results of the current trial round, determines the difficulty parameters and mnemonic strategies related to the task scene according to the memory performance of the user, and finally generates adjustment instructions according to the difficulty parameters and mnemonic strategies;

[0011] Each trial round includes N continuous EEGs;

[0012] and a Unity 3D engine platform connected with the head-mounted device, which receives the adjustment instructions to adjust the corresponding variables in the task scene to display the adjusted task scene.

[0013] Preferably, the adaptive adjustment and feedback module comprises:

[0014] a sequence of an adjustment triggering module, an adjustment module, and an output module; the adjustment module comprises a difficulty parameter adjustment module and a dynamic mnemonic strategy adjustment module;

[0015] The adjustment triggering module receives the memory state prediction result of the current trial round and calculates the familiarity and recall degree, then judges the memory performance of the user according to the familiarity and recall degree, and determines the required task scene difficulty parameter and mnemonic strategy according to the memory performance of the user;

[0016] The difficulty parameter adjustment module is used to generate corresponding adjustment instructions according to the determined task scene difficulty parameter;

[0017] The dynamic mnemonic strategy adjustment module is used to generate corresponding adjustment instructions according to the determined mnemonic strategy;

[0018] The output module outputs the adjustment instructions to the Unity 3D engine platform.

[0019] Preferably, the data processing and prediction module comprises:

[0020] a data processing module that receives EEGs, and outputs the EEGs in the encoding stage;

[0021] and a prediction module that receives the EEGs in the encoding stage, and outputs the memory state prediction result.

[0022] Preferably, the prediction module comprises at least one machine learning classification model.

[0023] Preferably, the EEG data acquisition module is a Brain Products LiveAmp system, and the head-mounted device is a Meta Quest 3 head-mounted device;

[0024] The Brain Products LiveAmp system comprises an EEG amplifier and an acquisition module connected with the output end of the EEG amplifier.

[0025] Preferably, the Meta Quest 3 head-mounted device integrates multiple task scenes, and the task scenes are developed based on the Unity 3D engine.

[0026] Preferably, the memory state prediction result comprises: forgetting False, remembering all attributes SC, remembering only color CC, remembering only location LC, and forgetting all attributes SI.

[0027] An AR adaptive episodic memory training method based on electroencephalogram neural feedback, based on the AR adaptive episodic memory training system based on electroencephalogram neural feedback, comprising the following steps:

[0028] Step 1, build an AR adaptive episodic memory training system based on electroencephalogram neural feedback, then the user wears a head-mounted device and EEG acquisition hardware, and finally starts the task scene for training on the Unity 3D engine platform;

[0029] Step 2, synchronously open the AR device and the EEG data acquisition module;

[0030] Step 3, according to the current trial electroencephalogram signal EEG collected, obtain the memory state prediction result of the current trial electroencephalogram signal EEG;

[0031] Step 4, when the memory state prediction results corresponding to N trial electroencephalogram signals EEG are collected, calculate the familiarity index and the recall index, determine the difficulty parameter and the mnemonic strategy of the task scene, generate the corresponding adjustment instruction, and send it to the Unity 3D engine platform in real time.

[0032] Preferably, the step 4 specifically comprises the following steps:

[0033] Step 4A, calculate the familiarity index and the recall index representing memory performance, then determine the difficulty interval of the user's memory performance, and finally determine the adjustment level required by the user's memory performance according to the difficulty interval:

[0034] Based on the memory state prediction results of the continuous N trial electroencephalogram signals EEG, the proportion of remembering Hit is calculated as the familiarity index; the proportion of Hit = the number of trial remembering Hit / N;

[0035] Based on the memory state prediction results of the continuous N trial electroencephalogram signals EEG, the proportion of remembering all attributes SC is calculated as the recall index; the proportion of remembering all attributes SC = the number of trial remembering all attributes SC / the number of trial remembering all attributes SC + the number of trial remembering only color CC + the number of trial remembering only location LC + the number of trial forgetting all attributes SI;

[0036] Step 4B, adjust the difficulty parameter and the mnemonic strategy according to the adjustment strategy corresponding to the adjustment level;

[0037] Step 4C, generate the respective adjustment instructions according to the difficulty parameter and the mnemonic strategy determined in step 4B;

[0038] Step 4D, adjust the instruction transmission to the Unity 3D engine platform dynamic adjustment.

[0039] Compared with the prior art, the advantages of the present application are:

[0040] 1. Closed-loop EEG neurofeedback mechanism: based on SME, a real-time EEG prediction model is constructed to dynamically identify the memory state (remember / forget) in the encoding stage and trigger adaptive adjustment (such as extending the presentation time, increasing or decreasing semantic expansion).

[0041] 2. AR and EEG deep coupling: map the neurofeedback directly to the AR task parameters, and realize the "EEG-scene" collaborative optimization through real-time communication between Unity and Python. AR task parameters include: difficulty parameter and mnemonic strategy.

[0042] 3. Scenario memory paradigm design: design an AR old / new recognition task that integrates deep processing (semantic judgment, spatial rationality), and combine multi-modal feedback (visual expression, voice prompt) to enhance the encoding effect. BRIEF DESCRIPTION OF DRAWINGS

[0043] Figure 1 Figure 1 is a flow chart of the offline training process of the data processing and prediction module;

[0044] Figure 2 Figure 2 is a diagram of the training principle mechanism of the data processing and prediction module;

[0045] Figure 3 Figure 3 is a diagram of the adaptive interaction mechanism;

[0046] Figure 4 Figure 4 is a diagram of the overall architecture of the AR adaptive scenario memory training system based on EEG neurofeedback;

[0047] Figure 5 Figure 5 is a diagram of the principle of determining the difficulty parameter and mnemonic strategy according to the user's performance. DETAILED DESCRIPTION

[0048] The AR adaptive scenario memory training system and method based on EEG neurofeedback of the present application will be described in more detail below in conjunction with the accompanying drawings, which represent the preferred embodiments of the present application. It should be understood that those skilled in the art can modify the present application described herein while still achieving the advantageous effects of the present application. Therefore, the following description should be understood as a broad understanding for those skilled in the art, and not as a limitation on the present application.

[0049] As Figures 3 to 5 An AR adaptive scenario memory training system based on EEG neurofeedback, comprising:

[0050] A head-mounted device for presenting a high-ecological validity training scene and interacting with a user;

[0051] An EEG data acquisition module for acquiring electroencephalogram (EEG) signals in real time and transmitting them to a data processing and prediction module in real time;

[0052] A data processing and prediction module for processing and classifying electroencephalogram (EEG) signals, outputting memory state prediction results of the current trial electroencephalogram (EEG) signals to an adaptive adjustment and feedback module;

[0053] An adaptive adjustment and feedback module for determining the memory performance of the user based on the familiarity and recall indicators in the memory state prediction results of the current trial, determining the required task scene difficulty parameters and mnemonic strategies according to the memory performance of the user, and finally generating adjustment instructions according to the task scene difficulty parameters and mnemonic strategies.

[0054] Each trial round includes N consecutive trial electroencephalogram (EEG) signals; a trial electroencephalogram (EEG) signal is a trial round.

[0055] The difficulty parameters include: number of colors, number of objects, presentation duration, and spatial distribution.

[0056] The mnemonic strategies include: difficulty adjustment voice notification; increasing or decreasing semantic expansion; and concentration reminder.

[0057] A Unity 3D engine platform connected to the head-mounted device, which receives adjustment instructions to adjust corresponding variables in the task scene to display the adjusted task scene. The Unity 3D engine platform is also used for interactive feedback presentation and user behavior data back transmission.

[0058] The variables in the task scene correspond to the difficulty parameters and mnemonic strategies, including stimulus presentation time, object quantity, spatial density, and mnemonic prompt type.

[0059] As shown in Figure 4 The EEG data acquisition module is a Brain Products LiveAmp system.

[0060] The Brain Products LiveAmp system includes an electroencephalograph (EEG) amplifier Brain Products LiveAmp and an acquisition module Brainvision Pycorder connected to the output end of the electroencephalograph (EEG) amplifier.

[0061] The electroencephalograph (EEG) amplifier acquires and encodes electroencephalogram (EEC) signals in the acquisition and retrieval stages, and the acquisition module outputs the electroencephalogram (EEC) signals.

[0062] The multi-channel LiveAmp EEG system of German Brain Products worn on the head of the user supports real-time signal transmission, contains 32 channels such as Fz, Cz, Pz, and focuses on monitoring memory-related neural oscillations in the frontal lobe and parietal lobe region. The sampling rate is 500Hz-1000Hz, supports wireless transmission, low-noise amplification, and is suitable for mobile experiments.

[0063] Figure 1 For the offline training process of the EEG classification model (data processing and prediction module).

[0064] The head-mounted device is a Meta Quest 3 head-mounted device.

[0065] The Meta Quest 3 head-mounted device integrates multiple task scenarios, and the task scenarios are all developed based on the Unity 3D engine. In other embodiments, a portable electromotor can also be used to collect the EEG signals of the user.

[0066] Specifically, the mixed display toolkit Meta XR suitable for the development of the Meta Quest 3 is loaded on the Unity 3D engine platform.

[0067] The preset scenario is loaded as a memory task scenario through the AR platform (Unity engine 3D), the target object or event is embedded in the spatial environment, and the user is guided to complete the encoding task (such as remembering the position and color of the object).

[0068] The Meta Quest 3 head-mounted device is an integrated VR device, which includes a controller.

[0069] The Meta Quest 3 head-mounted device is responsible for spatial scanning and presenting virtual objects in space.

[0070] The controller of the Meta Quest 3 head-mounted device is used to receive user input and record behavior data.

[0071] As shown in Figure 4 , the data processing and prediction module includes:

[0072] The data processing module outputs the EEG signal in the encoding stage. The trial is about 5s, and the segmentation and extraction of the trial are completed by the data processing module.

[0073] The data processing module is used for band-pass filtering, notch filtering, ICA de-artifacting, and feature extraction (including power spectral density, time domain features, etc.) on the received EEG data.

[0074] The prediction module receives the EEG signal in the encoding stage and outputs the memory state prediction result.

[0075] The prediction module is one of SVM, random forest, or XGboost. The memory state prediction results include: forget (False), remember all attributes (SC), remember only color (CC), remember only position (LC), and forget all attributes (SI).

[0076] That is, the prediction module includes at least one machine learning classification model, such as support vector machine (SVM), random forest (RF), and gradient boosting tree (XGBoost), and adopts a multi-model integration strategy (majority voting method) to classify the EEG data in the encoding stage, and outputs the memory state prediction result of the current trial EEG signal to drive the subsequent adaptive adjustment and feedback module.

[0077] Specifically, memory results are categorized as Hit / Miss. Items remembered can be further categorized based on recall into four categories: all attributes remembered, color only, location only, and all attributes forgotten. Therefore, the two-category (remember / forget) model can be expanded to five categories, coded as Forgotten (False), All attributes remembered (Source Correct, SC), Color only remembered (Color Correct, CC), Location only remembered (Location Correct, LC), and All attributes forgotten (Source Incorrect, SI).

[0078] like Figure 2 The figure shows the technical implementation diagram of EEG classification model training.

[0079] The data processing and prediction module, adaptive adjustment and feedback module, and Unity 3D engine platform are all integrated into a desktop computer.

[0080] The EEG signals collected in real time during the encoding phase are transmitted to a desktop computer.

[0081] A desktop computer with an i9-12900H processor and an NVIDARTX 3070Ti GPU, running a Windows 11 operating system, was used to manage and control the entire experiment.

[0082] like Figure 3 As shown, the adaptive adjustment and feedback module includes: an adjustment trigger module, an adjustment module and an output module connected in sequence; the adjustment module includes a difficulty parameter adjustment module and a dynamic mnemonic strategy adjustment module.

[0083] The adjustment trigger module receives the memory state prediction results of the current test round and calculates familiarity and recall. It then judges the user's memory performance based on familiarity and recall, and determines the required task scenario difficulty parameters and mnemonic strategies based on the user's memory performance.

[0084] The specific steps of obtaining the memory state prediction result of the current trial are: through the data processing and prediction module of the Python backend, the collected electroencephalogram signals are predicted in real time, and the memory state (one of the five classifications) of the current trial is output.

[0085] In the embodiment, every 10 trials of electroencephalogram signals are a trial round, the overall performance of 10 trials is counted (two indicators need to be counted in combination with 10 predicted memory states: a. Familiarity, b. Recall degree), the adjustment level corresponding to the memory performance of the user is judged, and then the overall difficulty parameter and the adjustment direction of the mnemonic strategy are determined.

[0086] The memory performance of the user is represented by the familiarity index and the recall degree index. The threshold values of the two indexes are determined according to the pre-experiment, and the average performance level ± 1 variance SD of the user is calculated.

[0087] As shown in Figure 5 , the adjustment level includes: difficulty reduction, difficulty unchanged and difficulty increase.

[0088] For example, the difficulty interval corresponding to difficulty increase includes: familiarity > 75% and 55% < recall degree < 70%, 62% < familiarity < 75% and recall degree > 70%.

[0089] That is, when the memory performance of the user corresponds to one of the above difficulty intervals, the adjustment level corresponding thereto is difficulty increase.

[0090] The difficulty parameter adjustment module is used to generate corresponding adjustment instructions according to the determined task scene difficulty parameter.

[0091] As shown in Figure 5 , the difficulty parameter adjustment module defines color quantity, body block quantity, presentation time length, space distribution and other parameter variables. The adjustment mode follows the principle of gradual adjustment, and the adjustment strategy is as shown in Figure 5 .

[0092] That is, if the adjustment level corresponding to the memory performance of the user is difficulty increase, the parameter is adjusted according to the adjustment mode as shown in Figure 5 .

[0093] The dynamic mnemonic strategy adjustment module is used to generate corresponding adjustment instructions according to the determined mnemonic strategy.

[0094] The dynamic mnemonic strategy adjustment module is coupled with the difficulty parameter adjustment module and is used for dynamic adjustment after every 10 trials.

[0095] For example, when the system determines that the difficulty needs to be reduced (the adjustment level is difficulty reduction), the determined mnemonic strategy includes: the voice prompts the user to "the current task difficulty is reduced, please keep your attention", increases semantic expansion (such as "the deep pot is on the desktop"), and assists the user in remembering;

[0096] When the system determines that the difficulty needs to be increased (the adjustment level is difficulty increase), the determined mnemonic strategy includes: reducing semantic expansion.

[0097] Secondly, after each single test, the prediction result needs to be adjusted in real time. If the prediction result is not remembered, the object appears repeatedly at the end of this experiment; if the prediction result is to remember the object, the next object of the same type appears. Each experiment contains 4 test rounds.

[0098] The output module outputs the adjustment instruction to the Unity 3D engine platform.

[0099] As shown in Figure 3 "System feedback and user perception" shows the adjustment instructions generated by difficulty parameters and mnemonic strategies respectively, and the Unity 3D engine platform adjusts the task scene according to the adjustment instructions.

[0100] Among them, the adjustment instructions generated by the mnemonic strategy include 2 types of visual instructions and auditory instructions, and the adjustment instructions generated by the difficulty parameters also include 2 types of visual instructions and auditory instructions.

[0101] Specifically, the adaptive adjustment and feedback module dynamically adjusts the AR task adaptive parameters and mnemonic strategies according to the prediction results according to the adjustment thresholds of familiarity and recall degree confirmed by pre-experiment, and optimizes the user's memory strategy through multi-modal feedback (emoji, voice prompt, etc.).

[0102] The core module functions of the system include: (1) Scene memory AR training task design module in the memory encoding and recall retrieval stage (2) Double-index adjustment strategy module based on EEG neural feedback, which dynamically judges the current cognitive state of the user for the first time by combining the predicted familiarity (overall recognition level) and recall degree (detail recall integrity) double index. (3) Multi-modal feedback module, which optimizes user interaction and memory strategy by combining visual (emoji, color prompt), auditory (voice prompt), interface prompt and other multi-modal ways.

[0103] Working principle of AR adaptive scene memory training system based on EEG neural feedback:

[0104] Step 1, build an AR adaptive scene memory training system based on EEG neural feedback, then the user wears a head-mounted device and an EEG amplifier in the EEG data acquisition module: finally start the task scene for training in the Unity 3D engine platform.

[0105] Step 2, synchronously open the head-mounted device and the EEG data acquisition module;

[0106] Step 3, according to the current trial EEG signal EEG collected, the memory state prediction result of the current trial EEG signal EEG is obtained.

[0107] Step 4, when the memory state prediction results corresponding to the N trial EEG signals EEG are collected, the familiarity index and the recall index are calculated, the difficulty parameter and the mnemonic strategy of the task scene are determined, the corresponding adjustment instruction is generated, and is sent to the Unity 3D engine platform in real time.

[0108] Step 4 specifically includes the following steps:

[0109] Step 4A, calculate the familiarity index and the recall index representing memory performance, then determine the difficulty interval of the user's memory performance, and finally determine the adjustment level required by the user's memory performance according to the difficulty interval.

[0110] Based on the memory state prediction results of the continuous N trial EEG signals EEG, the proportion of remembering Hit is taken as the familiarity index, and the proportion of Hit = the number of trial remembering Hit / N;

[0111] Based on the memory state prediction results of the continuous N trial EEG signals EEG, the proportion of remembering SC with all attributes is taken as the recall index; the proportion of remembering SC with all attributes = the number of trial remembering SC with all attributes / the number of trial remembering SC with all attributes + the number of trial remembering color CC only + the number of trial remembering location LC only + the number of trial forgetting SI with all attributes;

[0112] Specifically, the 10-trial overall performance calculation is based on the prediction results of the last 10 trials.

[0113] Familiarity index: the proportion of remembering (including SC, CC, LC, SI) in 10 trials, reflecting the overall recognition level of the user to the stimulus;

[0114] Recall index: the proportion of remembering SC (remembering all attributes) in the above remembered trials, reflecting the complete recall level of the user to the detail attributes.

[0115] The system has set the threshold of the medium difficulty interval in the two indexes according to the pre-experiment, and according to Figure 5 Adjust the difficulty parameter and the mnemonic strategy.

[0116] The difficulty parameter includes: the number of colors, the number of blocks, the presentation duration, and the spatial distribution.

[0117] Mnemonic strategies include: difficulty adjustment voice notification; increase or decrease semantic expansion; focus attention reminder.

[0118] Step 4B, adjust the difficulty parameter and mnemonic strategy according to the adjustment strategy corresponding to the adjustment level;

[0119] Step 4C, generate respective adjustment instructions according to the difficulty parameters and mnemonic strategies determined in step 4B;

[0120] Based on the determined difficulty parameters and mnemonic strategies, specific adjustment instructions are generated, including:

[0121] Difficulty parameter adjustment (such as {"adjust":"stim_count","change":+1}, indicating increasing the number of objects);

[0122] Mnemonic strategy adjustment (such as {"adjust":"mnemonic_hint","enable":true}, indicating enabling semantic hints).

[0123] Step 4D, the adjustment instructions are transmitted to the Unity 3D engine platform for dynamic adjustment.

[0124] The above adjustment instructions are encapsulated and sent to the Unity 3D engine platform in real time through the WebSocket protocol, and the Unity parses the instructions to dynamically adjust the task scene, mnemonic hints and user interaction feedback, realizing closed-loop adaptive adjustment.

[0125] In addition, step 3 also includes the following mnemonic strategy adjustment steps for single trial.

[0126] Single trial instant state prediction results adopt a five-classification system, including False, SC, CC, LC, and SI. The mnemonic strategy based on single trial adjustment includes: if it is False, the last repeated object appears; if it is any one of SC, CC, LC, and SI, the next object of the same type appears.

[0127] In summary, the system aims to solve the problems in the prior art such as the disconnection between EEG neurofeedback and AR memory training, the lack of real-time closed-loop intervention, and the insufficient optimization of scenario memory encoding stage.

[0128] The present application is based on electroencephalogram (EEG) and augmented reality (AR) technology, and constructs an adaptive scenario memory training system (Amemory), effectively solving the technical defects of the existing scenario memory intervention system, such as low ecological validity, lack of real-time feedback, and difficulty in individualized adjustment. The main advantages and beneficial effects are as follows:

[0129] 1. High ecological validity training paradigm, enhancing user immersion and training motivation.

[0130] Design immersive AR episodic memory tasks to simulate real-life scenarios and improve the transfer effect of memory training.

[0131] Specifically, AR technology is used to build memory training scenarios close to daily life (home scenarios and life objects) to replace traditional laboratory word list or picture memory tasks that are detached from reality. Through spatial positioning and three-dimensional visual interaction, the realism and context relevance of the training task are improved. This design not only improves memory encoding quality, but also significantly enhances user motivation and training compliance, providing more favorable conditions for long-term memory consolidation.

[0132] 2. Real-time prediction mechanism driven by EEG signals, real-time memory state prediction, and individual memory state recognition.

[0133] Based on the Subsequent Memory Effect (SME), EEG signals are used to predict the user's memory state (remember / forgotten) during the encoding phase.

[0134] Specifically: The system collects EEG signals in real time during the encoding phase of the AR episodic memory training paradigm, extracts oscillation features of memory-related frequency bands such as θ, α, and γ, and uses machine learning models to predict the "remember / forgotten (not remembered)" state based on the Subsequent Memory Effect (SME) principle. According to this specific paradigm, a four-class classification of detailed memory states (color / position memory) is further conducted, breaking through the technical limitations of traditional paradigms and providing objective and continuous physiological basis for task adjustment.

[0135] 3. Multi-dimensional adaptive training strategy design, realizing real-time closed-loop neural feedback for adaptive task adjustment.

[0136] According to the EEG prediction results, dynamically adjust the AR training difficulty parameters (including stimulus color quantity, body block quantity, presentation duration, spatial distribution, etc.), and couple dynamic mnemonic strategies such as cue repetition, state notification, semantic mnemonic strategy, and focused attention reminders to ensure that the user is in the optimal challenge interval.

[0137] According to the prediction results of each trial, the system automatically adjusts the difficulty parameters and mnemonic strategies for the next round.

[0138] 4. Multi-modal neural feedback: combining visual, auditory, and semantic cues (increasing or decreasing semantic expansion) to enhance user perception of memory state and active adjustment ability.

[0139] Support multi-modal feedback mode (visual, prompt animation, voice guide), enhance the depth of coding. This strategy effectively solves the problem of "too difficult to give up / too easy to be ineffective" under fixed training difficulty. That is, through multiple sensory channels (visual, auditory, language, action, etc.) into the brain, form a more rich neural representation.

[0140] The system integrates EEG acquisition, state prediction, strategy adjustment, and Unity task linkage into a closed-loop system. The system updates the state prediction results immediately after each training trial is completed and drives the training content to change, realizing a continuous adaptive training process.

[0141] Compared with the traditional offline analysis and manual intervention method, the closed-loop structure improves the intelligent degree of training and the intervention efficiency.

[0142] The above is only the preferred embodiment of the present application, and does not have any limiting effect on the present application. Any person skilled in the art, without departing from the scope of the technical solution of the present application, makes any form of equivalent replacement or modification of the technical solution and technical content disclosed by the present application, etc. change, does not depart from the content of the technical solution of the present application, still belongs to the protection scope of the present application.

Claims

1. An AR adaptive situational memory training system based on EEG neurofeedback, characterized in that: include: Head-mounted display (HMD); EEG data acquisition module, used to collect EEG signals in real time and transmit them to the data processing and prediction module in real time; The data processing and prediction module is used to process the EEG signal, classify and predict it, and output the memory state prediction result of the current trial EEG signal to the adaptive adjustment and feedback module; The adaptive adjustment and feedback module judges the user's memory performance based on the familiarity and recall indicators in the memory status prediction results of the current trial round, determines the required difficulty parameters and mnemonic strategies related to the task scenario based on the user's memory performance, and finally generates adjustment instructions based on the difficulty parameters and mnemonic strategies; Each trial round includes N consecutive trial EEG signals; and the Unity 3D engine platform, which is connected to the head-mounted display device, and receives adjustment instructions to adjust corresponding variables in the task scene to display the adjusted task scene.

2. The AR adaptive situational memory training system based on EEG neurofeedback according to claim 1 is characterized in that: The adaptive adjustment and feedback module includes: An adjustment trigger module, an adjustment module and an output module connected in sequence; the adjustment module includes a difficulty parameter adjustment module and a dynamic mnemonic strategy adjustment module; The adjustment trigger module receives the memory state prediction result of the current test round and calculates the familiarity and recall, then judges the user's memory performance based on the familiarity and recall, and determines the required task scenario difficulty parameters and mnemonic strategies based on the user's memory performance; The difficulty parameter adjustment module is used to generate corresponding adjustment instructions according to the determined task scenario difficulty parameters; The dynamic mnemonic strategy adjustment module is used to generate corresponding adjustment instructions according to the determined mnemonic strategy; The output module outputs the adjustment instruction to the Unity 3D engine platform.

3. The AR adaptive situational memory training system based on EEG neurofeedback according to claim 1 is characterized in that: The data processing and prediction module includes: Data processing module, which outputs EEG signals in the encoding stage; And a prediction module, which receives the EEG signal in the encoding stage and outputs the memory state prediction result.

4. The AR adaptive situational memory training system based on EEG neurofeedback according to claim 3 is characterized in that: The prediction module includes at least one machine learning classification model.

5. The AR adaptive situational memory training system based on EEG neurofeedback according to claim 1 is characterized in that: The EEG data acquisition module is a Brain Products LiveAmp system, and the head-mounted display device is a Meta Quest 3 head-mounted display device; The Brain Products LiveAmp system includes an EEG amplifier and an acquisition module connected to the output end of the EEG amplifier.

6. The AR adaptive situational memory training system based on EEG neurofeedback according to claim 5 is characterized in that: The Meta Quest 3 head-mounted display device integrates multiple mission scenarios, which are developed based on the Unity 3D engine.

7. The AR adaptive situational memory training system based on EEG neurofeedback according to claim 1 is characterized in that: The memory state prediction results include: forgotten False, all attributes remembered SC, only remember the color CC, only remember the position LC, and all attributes forgotten SI.

8. An AR adaptive situational memory training method based on EEG neurofeedback, based on the AR adaptive situational memory training system based on EEG neurofeedback according to any one of claims 1 to 7, characterized in that: The following steps are involved: Step 1: Build an AR adaptive situational memory training system based on EEG neurofeedback. The user then wears a head-mounted display (HMD) and EEG acquisition hardware, and finally launches the training task scene on the Unity 3D engine platform. Step 2: Synchronously start the AR device and EEG data acquisition module; Step 3: Obtain a memory state prediction result of the current trial EEG signal based on the collected current trial EEG signal; Step 4: When the memory state prediction results corresponding to N trial EEG signals are collected, the familiarity index and recall index are calculated, the difficulty parameters and mnemonic strategies of the task scenario are determined, the corresponding adjustment instructions are generated, and sent to the Unity 3D engine platform in real time.

9. The AR adaptive situational memory training method based on EEG neurofeedback according to claim 8, characterized in that: The step 4 specifically includes the following steps: Step 4A: Calculate the familiarity index and recall index representing the memory performance, then determine the difficulty range of the user's memory performance, and finally determine the adjustment level required for the user's memory performance based on the difficulty range: Based on the memory state prediction results of N consecutive EEG trials, the proportion of hits remembered is counted as the familiarity index; the proportion of hits = the number of trials in which hits are remembered / N; Based on the memory state prediction results of N consecutive EEG trials, the proportion of attributes remembered SC is counted as the recall index; the proportion of attributes remembered SC = the number of trials with attributes remembered SC / the number of trials with attributes remembered SC + the number of trials with only color CC remembered + the number of trials with only location LC remembered + the number of trials with all attributes forgotten SI; Step 4B, adjusting the difficulty parameter and the mnemonic strategy according to the adjustment strategy corresponding to the adjustment level; Step 4C: Generate corresponding adjustment instructions according to the difficulty parameter and mnemonic strategy determined in step 4B; Step 4D: The adjustment instruction is transmitted to the Unity 3D engine platform for dynamic adjustment.

Images