Multimodal human cognitive training method, apparatus, edge computing device, and medium

Abstract

The present disclosure provides a multi-modal human cognitive training method, device, edge computing device and medium, relating to the technical fields of cognitive rehabilitation, human-computer interaction and the like. The multi-modal human cognitive training method comprises: obtaining task configuration parameters of a cognitive training task, the task configuration parameters at least comprising a target frequency; based on the task configuration parameters, generating sound stimuli and visual stimuli corresponding to the cognitive training task, and synchronously outputting the sound stimuli and the visual stimuli according to the target frequency in the process of executing the cognitive training task; obtaining a cognitive training result obtained by a user executing the cognitive training task; and adjusting a next cognitive training task based on the cognitive training result.

Description

Technical Field

[0001] This disclosure relates to the technical fields of cognitive rehabilitation and human-computer interaction, and in particular to a multimodal cognitive training method, device, edge computing equipment and medium. Background Technology

[0002] Patients with various types of brain dysfunction, neurological abnormalities, and cognitive deficits commonly experience a decline in multiple dimensions of cognitive abilities, including attention, working memory, executive function, and sensory integration, severely impacting their daily information processing, perceptual coordination, and behavioral regulation abilities. Therefore, achieving high-precision, personalized, and adaptive multimodal cognitive rehabilitation training is a pressing technical challenge in current cognitive rehabilitation technology.

[0003] Cognitive rehabilitation training often focuses on single cognitive tasks, paper-and-pencil exercises, or general computerized tasks, making it difficult to simultaneously provide stable neural rhythm stimulation and structured task training. Patient participation, compliance, and training effectiveness vary among individuals. Audiovisual stimulation programs typically rely on dedicated lighting, external sound sources, or independent stimulation devices, separating visual and auditory stimulation from cognitive tasks. This results in the audiovisual stimulation process failing to precisely align with cognitive task events, leading to poor training outcomes. Summary of the Invention

[0004] One of the technical problems that this disclosure aims to solve is that the audio-visual stimulation process cannot be accurately aligned with cognitive task events, resulting in poor training effects.

[0005] To address the aforementioned technical problems, this disclosure provides a multimodal cognitive training method, comprising: acquiring task configuration parameters for a cognitive training task, the task configuration parameters including at least a target frequency; generating auditory and visual stimuli corresponding to the cognitive training task based on the task configuration parameters, and synchronously outputting the auditory and visual stimuli according to the target frequency during the execution of the cognitive training task; acquiring the cognitive training results obtained by the user in executing the cognitive training task; and adjusting the next cognitive training task based on the cognitive training results.

[0006] In some embodiments, obtaining the cognitive training results obtained by the user performing a cognitive training task includes: obtaining timestamp data of the user performing the cognitive training task and the user's feedback data, wherein, under the timestamp data, sound stimuli and visual stimuli are output synchronously, and the timestamp data and feedback data are based on the master clock data; correcting the timestamp data based on pre-calibrated display link compensation and audio link compensation to obtain the actual visual stimulus presentation time and the actual sound stimulus playback time; determining visual phase data and sound phase data based on the target frequency, the actual visual stimulus presentation time, and the actual sound stimulus playback time; and performing multimodal event alignment processing on the timestamp data, visual phase data, sound phase data, and user feedback data to obtain the cognitive training results.

[0007] In some embodiments, visual stimuli are output through a display, and auditory stimuli are output through an audio player. The display link compensation amount and audio link compensation amount are obtained by: acquiring offline calibration parameters and scheduling delay data, wherein the offline calibration parameters include the display link delay amount corresponding to the display and the audio link delay amount corresponding to the audio player; determining joint delay data based on the display link delay amount, audio link delay amount, and scheduling delay data; and determining the display link compensation amount and audio link compensation amount based on the joint delay data.

[0008] In some embodiments, the sound stimulus is output through an audio player, which includes multiple channels; before outputting the sound stimulus through the audio player, the method further includes: determining a delayed channel among the multiple channels and a delay compensation amount corresponding to the delayed channel; determining the number of offset sampling points in a buffer based on a preset audio sampling rate and the delay compensation amount, wherein the buffer stores the sound stimulus; and adjusting the playback position of the delayed channel in the buffer based on the number of offset sampling points.

[0009] In some embodiments, after outputting sound stimuli, the method further includes: when the user performs a cognitive training task, obtaining the playback position of the buffer corresponding to the multi-channel; when the difference between the playback positions of any two channels is greater than a preset threshold and less than or equal to a reset threshold, adjusting the pointer of the phase accumulator or the buffer, wherein the buffer determines the playback position in the form of a pointer; when the difference between the playback positions of any two channels is less than or equal to the preset threshold, continuing to output sound stimuli based on the current playback position; when the difference between the playback positions of any two channels is greater than the reset threshold, resetting the buffer and writing it to an exception log.

[0010] In some embodiments, the offline calibration parameters further include a calibration compensation amount between vision and sound. Before synchronously outputting sound and visual stimuli, the method further includes: calibrating the display and audio player, performing multiple measurements during the calibration process to obtain the visual peak time, audio peak time, and calibration measurement time for each measurement; determining a calibration delay based on the visual peak time and audio peak time obtained for each measurement; determining the absolute value of the calibration delay, using the calibration delay as a calibration compensation amount when the absolute value is less than or equal to a calibration threshold, and adjusting the output time of the sound or visual stimulus based on the calibration delay when the absolute value is greater than the calibration threshold; and obtaining and storing the calibration result based on the device information of the display, the device information of the audio player, the calibration measurement time, and the calibration compensation amount.

[0011] In some embodiments, after synchronously outputting sound stimuli and visual stimuli, the method further includes: downsampling the sound stimuli and visual stimuli according to a preset time interval and a preset length, and obtaining the audio envelope of the output sound stimuli and the visual sequence of the corresponding visual stimuli; performing cross-correlation calculation based on the audio envelope and visual sequence to determine the delay index, wherein the delay index is the maximum value obtained by the cross-correlation calculation; updating the calibration compensation amount based on the delay index and the downsampling sampling rate, using the calibration compensation amount as the initial value, to obtain the updated calibration compensation amount; and adjusting the playback position of the sound stimuli in the buffer based on the updated calibration compensation amount, wherein the buffer stores the sound stimuli and the buffer determines the playback position in the form of a pointer.

[0012] In some embodiments, the task configuration parameters include a modulation mode and modulation parameters corresponding to the modulation mode, and a sound mode and sound parameters corresponding to the sound mode; based on the task configuration parameters, generating sound stimuli and visual stimuli corresponding to the cognitive training task includes: generating visual stimuli corresponding to the cognitive training task based on the modulation mode and modulation parameters, wherein the modulation mode includes at least one of brightness modulation, grayscale modulation, and transparent masking modulation; generating sound stimuli corresponding to the cognitive training task based on the sound mode and sound parameters, wherein the sound mode includes any one of click sound, modulation sound, and gated noise; and / or the cognitive training task includes multiple tasks, each corresponding to a different cognitive domain; adjusting the next cognitive training task based on the cognitive training results includes: determining the user's impaired rating vector for multiple cognitive domains based on the cognitive training results, and obtaining the training weight vector for each cognitive training task; performing a weighted calculation based on the impaired rating vector, training weight vector, and pre-configured priority weight vector to obtain a similarity result; performing preliminary screening processing on the cognitive training tasks based on the impaired rating vector, training weight vector, and priority weight vector to obtain preliminary screened cognitive training tasks; and performing sorting and screening processing on the preliminary screened cognitive training tasks based on the similarity result to adjust the next cognitive training task.

[0013] In some embodiments, adjusting the next cognitive training task based on the cognitive training results includes: determining accuracy data, reaction time data, completion rate data, and penalty item data based on the cognitive training results; performing weighted processing on the accuracy data, reaction time data, completion rate data, penalty item data, first weight, second weight, third weight, and fourth weight to obtain comprehensive performance score data; and adjusting the next cognitive training task based on the comprehensive performance score data, target score data, safety upper limit data, and safety lower limit data.

[0014] In some embodiments, adjusting the next cognitive training task based on comprehensive performance score data, target score data, safety upper limit data, and safety lower limit data includes: when the comprehensive performance score data is greater than the target score data, increasing the training difficulty or extending the training time of the next cognitive training task based on the next task configuration parameters, wherein the increased training difficulty or extended training time meets the safety upper limit data; when the comprehensive performance score data is less than or equal to the target score data, decreasing the training difficulty, increasing the rest time, and decreasing the parameters corresponding to visual or auditory stimuli based on the next task configuration parameters, wherein the decreased training difficulty, increased rest time, and decreased parameters meet the safety lower limit data; and / or the method further includes: obtaining abnormal state data of the user performing the cognitive training task; and adjusting the next cognitive training task based on the abnormal state data.

[0015] In some embodiments, the cognitive training result includes a quality assessment result. Adjusting the next cognitive training task based on the cognitive training result includes: performing a quality assessment on the cognitive training result to obtain a quality assessment result; determining the cognitive assessment result as valid data when the quality assessment result is greater than or equal to a first quality assessment threshold; determining the cognitive assessment result as partially valid data and marking it as low-quality data when the quality assessment result is greater than or equal to a second quality assessment threshold and less than or equal to a third quality assessment threshold; determining the cognitive assessment result as invalid data and generating a prompt message when the quality assessment result is less than the second quality assessment threshold; and adjusting the next cognitive training task based on the valid data and / or partially valid data.

[0016] This disclosure also provides a multimodal cognitive training device, comprising: a first acquisition module for acquiring task configuration parameters of a cognitive training task, the task configuration parameters including at least a target frequency; a generation module for generating sound stimuli and visual stimuli corresponding to the cognitive training task based on the task configuration parameters, and synchronously outputting sound stimuli and visual stimuli according to the target frequency during the execution of the cognitive training task; a second acquisition module for acquiring cognitive training results obtained by the user from performing the cognitive training task; and an adjustment module for adjusting the next cognitive training task based on the cognitive training results.

[0017] This disclosure also provides an edge computing device, including a memory and a processor. The memory stores a computer program, which, when executed by the processor, implements the steps of the method of any of the preceding claims.

[0018] This disclosure also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method of any of the preceding claims.

[0019] Through the above technical solutions, the multimodal cognitive training method provided in this disclosure, by setting up a multimodal cognitive training task configuration interface that supports task enhancement mode, can activate the task enhancement mode of synchronous output of sound and visual stimuli at the target frequency in response to user selection operations, and carry out training by matching cognitive tasks corresponding to different cognitive domains. It can achieve multi-brain area collaborative activation by relying on multi-sensory synchronous rhythmic stimulation, which can significantly improve the training effect of cognitive rehabilitation training. At the same time, the precise audiovisual synchronous output method ensures the stability of training rhythm and the standardization of stimulus timing, effectively avoiding the problem of poor training effect caused by audiovisual asynchrony and rhythm disorder, and significantly improving the standardization and accuracy of multimodal cognitive training. It can also fully collect the user's cognitive training results and adaptively adjust the agent parameters based on the training results. It can dynamically adapt the training strategy according to the differences in cognitive impairment of different users and real-time training performance, realize personalized closed-loop optimization of training parameters and training plan, and automatically generate cognitive training reports and the next cognitive training task through iteratively updated agents. It realizes an intelligent rehabilitation mode with quantitative recording of training process, traceable training effect, and intelligent iteration of training plan, which greatly improves the personalization, accuracy and intelligence of cognitive rehabilitation training. Attached Figure Description

[0020] To more clearly illustrate the technical solutions in the embodiments of this disclosure or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this disclosure. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0021] Figure 1 This is a flowchart of the multimodal human cognitive training method disclosed in this embodiment; Figure 2 This is a schematic diagram of a layered display disclosed in an embodiment of this disclosure; Figure 3 This is a schematic diagram of the structure of the multimodal human cognitive training system disclosed in this embodiment; Figure 4 This is a schematic diagram of the multimodal personnel cognitive training system disclosed in this embodiment; Figure 5 This is a schematic diagram of the audio-visual synchronous stimulation disclosed in the embodiments of this disclosure; Figure 6 This is a block diagram of the multimodal human cognitive training device disclosed in this embodiment; Figure 7 This is a block diagram of an edge computing device disclosed in an embodiment of this disclosure. Detailed Implementation

[0022] The embodiments of this disclosure will be further described in detail below with reference to the accompanying drawings and examples. The detailed description of the embodiments and the accompanying drawings are used to illustrate the principles of this disclosure by way of example, but should not be used to limit the scope of this disclosure. This disclosure can be implemented in many different forms and is not limited to the specific embodiments disclosed herein, but includes all technical solutions falling within the scope of the claims.

[0023] These embodiments are provided to make the disclosure thorough and complete, and to fully express the scope of the disclosure to those skilled in the art. It should be noted that, unless otherwise specifically stated, the relative arrangement of components and steps, material composition, numerical expressions, and values ​​set forth in these embodiments should be interpreted as exemplary only and not as limiting.

[0024] All terms used in this disclosure have the same meaning as understood by one of ordinary skill in the art to which this disclosure pertains, unless otherwise specifically defined. It should also be understood that terms defined in general dictionaries should be interpreted as having meanings consistent with their meanings in the context of the relevant art, and not as idealized or highly formalized, unless expressly defined herein.

[0025] Techniques, methods, and equipment known to those skilled in the art may not be discussed in detail, but where appropriate, they should be considered part of the specification.

[0026] The data acquisition involved in this disclosure includes physiological data, which are various measurable data signals in the human body, including but not limited to electrocardiogram (ECG) signals, skin temperature (SKT) signals, photoplethysmogram (PPG) signals, electrodermal activity (EDA) signals, heart rate (HR) signals, electromyogram (EMG) signals, electroencephalogram (EEG) signals, and peripheral capillary oxygen saturation (SPO2) signals.

[0027] Patients with various types of brain dysfunction, neurological abnormalities, and cognitive deficits (such as mild cognitive impairment (MCI)) commonly experience a decline in multiple dimensions of cognitive abilities, including attention, working memory, executive function, sensory integration, and processing speed. This severely impacts their daily information processing, perceptual coordination, and behavioral regulation abilities. Therefore, achieving high-precision, personalized, and adaptive multimodal cognitive rehabilitation training is a pressing technical challenge that cognitive rehabilitation technology needs to address.

[0028] Cognitive rehabilitation training often focuses on single cognitive tasks, paper-and-pencil exercises, or general computerized tasks, making it difficult to simultaneously provide stable neural rhythm stimulation and structured task training. Patient participation, compliance, and training effectiveness vary among individuals. Audiovisual stimulation programs typically rely on dedicated lighting, external sound sources, or independent stimulation devices, separating visual and auditory stimulation from cognitive tasks. These programs have the following shortcomings: first, auditory and visual stimulation are difficult to control uniformly using the same software master clock; second, key data such as stimulation start time, phase, duration, missed frames, or audio delay lack consistent recording; and third, the audiovisual stimulation process cannot be precisely aligned with specific cognitive task events, making it difficult to track the "stimulus-task performance-feedback" relationship.

[0029] In one embodiment, while a display or ordinary computer can present images and sounds by playing audio, it typically lacks dedicated frame-level rendering, audio buffering, phase compensation, and session logging mechanisms designed for 40Hz stimuli. If flickering and sound are achieved solely through ordinary animation or audio playback, issues such as display refresh rate mismatch, audio buffering delay, task interface stuttering, and stimulus timing drift can easily occur, affecting training standardization and traceability.

[0030] In another embodiment, cognitive rehabilitation software focuses primarily on question presentation, score statistics, and training records, lacking an intelligent closed loop for aspects such as training plan generation, stimulus parameter constraints, handling of patient discomfort feedback, adaptive training difficulty, and report interpretation. Directly introducing a general artificial intelligence model may lead to problems such as uncontrollable output, unauthorized medical conclusions, parameters exceeding safe ranges, and lack of medical confirmation.

[0031] Therefore, there is an urgent need for a software system for cognitive rehabilitation scenarios in MCI: that can output stable, synchronous, and recordable 40Hz audio-visual stimulation using common displays and headphones, and can run cognitive training tasks during the stimulation process; at the same time, it should introduce an intelligent agent constrained by safety rules for training program generation, task difficulty adjustment, synchronous quality monitoring, safety prompts, and structured report generation, thereby improving the standardization, individualization, and clinical use efficiency of training.

[0032] The technical problems to be solved by this invention include: 1. How to control the display and headphones to output visual and auditory stimuli with a target frequency of 40Hz respectively, without relying on a dedicated audio-visual stimulator, and to ensure that both types of stimuli are uniformly scheduled by the same master clock. 2. How to make 40Hz audio-visual stimulation an optional enhancement mode in rehabilitation cognitive training tasks, rather than an independent entry point separate from existing cognitive rehabilitation software, so that the doctor can complete task selection and plan confirmation on the doctor's end, and the actual presentation can be achieved after the patient enters the assigned training. 3. How to achieve interface and acoustic coordination of "40Hz visual stimulation in non-task white areas, stable presentation in the central cognitive training window, and 40Hz auditory stimulation from headphones" during cognitive training, avoiding occlusion or interference of the stimulation layer with the training content. 4. How to solve problems such as stimulation frequency error, start delay, phase drift, and frame loss caused by factors such as the refresh rate of ordinary displays, audio playback buffering, and operating system scheduling. 5. How to generate training plans, adjust task difficulty and stimulation experience parameters within safe boundaries according to individual differences of MCI patients, and avoid the agent overstepping its authority to output medical diagnoses or uncontrolled parameter tuning. 6. How to generate traceable training reports so that healthcare professionals can see the stimulus parameters, cognitive performance, synchronization quality, safety incidents, and recommendations for the next training session.

[0033] In view of this, this disclosure provides a multimodal cognitive training method and system, which adopts 40Hz audio-visual synchronous stimulation, cognitive training tasks and intelligent agent closed-loop control. In the same training session, the software master clock uniformly schedules the 40Hz visual flicker output by the display, the 40Hz sound stimulation output by the headset, cognitive training task events, patient reaction events, safety feedback events and intelligent agent assisted decision-making events, so that "audio-visual stimulation - cognitive training - behavioral performance - safety feedback - report suggestions" form a recordable, traceable and verifiable closed loop.

[0034] Figure 1 This is a flowchart of the multimodal human cognitive training method disclosed in this embodiment, such as... Figure 1 As shown, the multimodal personnel cognitive training method 100 includes steps S110 to S140.

[0035] Step S110: Obtain the task configuration parameters for the cognitive training task. The task configuration parameters shall include at least the target frequency.

[0036] Specifically, the doctor logs into the multimodal cognitive training system and performs selection operations. For example, by selecting the corresponding cognitive training task in the cognitive training task configuration interface and checking the task enhancement mode (the task enhancement mode includes simultaneous sound and visual stimulation at the target frequency), training duration, training difficulty, visual stimulation switching order, sound mode, modulation mode, etc., the task configuration parameters are obtained. The task configuration parameters include sound mode (such as click sound, modulation sound, etc.), sound parameters corresponding to the sound mode (such as sound parameters obtained by selecting the corresponding parameters according to the sound mode, such as amplitude, amplitude, pulse shape, etc.), modulation mode (such as brightness modulation, grayscale modulation, etc.), modulation parameters corresponding to the modulation mode (such as modulation parameters obtained by selecting the corresponding parameters according to the modulation mode, such as modulation frequency, modulation amplitude, modulation waveform, duty cycle, etc.), target frequency, stimulation period, training duration, training difficulty, etc. The target frequency is, for example, 40Hz, which is not intended to limit this application.

[0037] Step S120: Based on the task configuration parameters, generate sound stimuli and visual stimuli corresponding to the cognitive training task, and output sound stimuli and visual stimuli synchronously according to the target frequency during the execution of the cognitive training task.

[0038] Specifically, cognitive training tasks, such as attention maintenance tasks, visual search tasks, working memory tasks, short-term memory encoding tasks, executive function tasks, response inhibition tasks, processing speed tasks, spatial localization tasks, and multi-task switching tasks, can all be presented during 40Hz audio-visual stimulation. Each task corresponds to a different cognitive domain. For example, the working memory task corresponds to the working memory cognitive domain, which is responsible for short-term online information storage and real-time operation processing.

[0039] After the user obtains the cognitive training task configured and assigned by the doctor, synchronous sound and visual stimuli are output at the target frequency in task enhancement mode. The cognitive training task is displayed in a specific position on the monitor (such as the central rectangular box of the display interface, which the user can manually adjust the size). The monitor displays visual stimuli according to a preset visual stimulus switching order, such as full screen, edge area, corner area, and custom area. At the same time, the audio player (such as headphones) outputs sound stimuli. While the cognitive task content is presented on the display interface of the cognitive training task, 40Hz rhythmic audiovisual synchronous stimulation is superimposed, so that multimodal stimulation is synchronized with the cognitive training tasks of each cognitive domain.

[0040] Step S130: Obtain the cognitive training results obtained by the user from performing the cognitive training task.

[0041] Specifically, during the cognitive training task performed by the user (patient) with superimposed audio-visual synchronous stimulation, the timestamp data of the cognitive training task execution, the user's feedback data, and the actual visual stimulus presentation time and actual audio stimulus playback time of the synchronous output of audio and visual stimulus instructions under the timestamp data are collected in real time. The feedback data includes reaction time, answer accuracy, task completion status, training pause behavior, and subjective discomfort feedback data, etc. The multi-dimensional data are aligned and processed to obtain the cognitive training result corresponding to the user's training.

[0042] Step S140: Adjust the next cognitive training task based on the cognitive training results.

[0043] Specifically, the next cognitive training task includes specific cognitive training tasks (task type), training duration, training difficulty, etc. Based on the results of this cognitive training, the agent calculates the user's comprehensive performance score data using weighted averages. This score data is then combined with preset target score data to adjust the agent's task configuration parameters, dynamically adjusting the next stage's cognitive training difficulty, 40Hz audio-visual stimulation experience parameters, training duration, and rest interval duration, generating a combination of cognitive training tasks suitable for the user's recovery level. Alternatively, based on the results of this cognitive training, the agent determines the cognitive domain impairment score vector, performs a weighted average calculation with the training weight vector of each cognitive training task and a pre-configured priority weight vector to obtain a similarity result, and then filters based on the similarity result to obtain the next cognitive training task. The agent generates a quantitative cognitive training report based on the cognitive training results of this cognitive training task and the next cognitive training task.

[0044] The embodiments of this disclosure, by setting up a multimodal cognitive training task configuration interface that supports task enhancement mode, can activate the task enhancement mode of synchronous output of sound and visual stimuli at the target frequency in response to user selection. It matches cognitive tasks corresponding to different cognitive domains for training, and can achieve multi-brain region collaborative activation through multi-sensory synchronous rhythmic stimulation, significantly improving the training effect of cognitive rehabilitation training. Simultaneously, the precise audiovisual synchronous output method ensures stable training rhythm and standardized stimulus timing, effectively avoiding poor training effects caused by audiovisual asynchrony and rhythm disorder, significantly improving the standardization and accuracy of multimodal cognitive training. Furthermore, it can completely collect the user's cognitive training results and adaptively adjust the agent parameters based on the training results. It can dynamically adapt training strategies according to the differences in cognitive impairment and real-time training performance of different users, achieving personalized closed-loop optimization of training parameters and training plans. Moreover, the iteratively updated agent automatically generates cognitive training reports and the next cognitive training task, realizing an intelligent rehabilitation mode with quantitative recording of the training process, traceable training effects, and intelligent iteration of training plans, greatly improving the personalization, accuracy, and intelligence of cognitive rehabilitation training.

[0045] In some embodiments of this disclosure, obtaining the cognitive training results obtained by a user performing a cognitive training task includes: obtaining timestamp data of the user performing the cognitive training task and the user's feedback data, wherein, under the timestamp data, sound stimuli and visual stimuli are output synchronously, and the timestamp data and feedback data are based on the master clock data; correcting the timestamp data based on pre-calibrated display link compensation and audio link compensation to obtain the actual visual stimulus presentation time and the actual sound stimulus playback time; determining visual phase data and sound phase data based on the target frequency, the actual visual stimulus presentation time, and the actual sound stimulus playback time; and performing multimodal event alignment processing on the timestamp data, visual phase data, sound phase data, and user feedback data to obtain the cognitive training results.

[0046] Specifically, when cognitive training tasks or patient behaviors occur, a unified master clock timestamp data is recorded. After considering the visual link (display link compensation) and audio link compensation, the visual phase data and audio phase data at this moment are calculated according to the following formulas (1) and (2), respectively: (1) (2) in, and These are the visual link compensation amount and the audio link compensation amount, respectively, and T0 is the target period determined based on the target frequency. The actual visual stimulus presentation time. For the actual sound stimulus playback time, the timestamp data, visual phase data, and sound phase data corresponding to the task event are bound to the same event record.

[0047] When the stimulus for the cognitive training task is presented at time Patient response time is When, reaction time (feedback data) As shown in formula (3): (3) Through the aforementioned time-binding relationships, task events are linked to patient responses, audio-visual phases to behavioral performance, and audio-visual phases to the presentation time of key tasks. The event log includes at least a timestamp, event type, event value, visual phase, audio phase, reaction result, and reaction time.

[0048] The logical flow is as follows: Task event or behavior occurs → Read unified master clock timestamp → Deduct corresponding link compensation amount → Calculate visual phase and sound phase → Calculate reaction time → Form structured event record → Write to session log.

[0049] In some embodiments of this disclosure, visual stimuli are output through a display, and auditory stimuli are output through an audio player. The display link compensation amount and audio link compensation amount are obtained in the following ways: obtaining offline calibration parameters and scheduling delay data, wherein the offline calibration parameters include the display link delay amount corresponding to the display and the audio link delay amount corresponding to the audio player; determining joint delay data based on the display link delay amount, audio link delay amount and scheduling delay data; and determining the display link compensation amount and audio link compensation amount based on the joint delay data.

[0050] Specifically, the display link delay, audio link delay, and common offset caused by operating system scheduling are measured separately. Under the notation conventions of this embodiment, the joint delay data is calculated as shown in formula (4): (4) in, The delay between the submission of rendering commands and the actual change in screen brightness. The latency from writing audio data to the actual output at the headphones. This refers to the common scheduling offset that can be identified during the measurement process.

[0051] Read the monitor refresh rate and the audio sampling rate of the audio player; measure the visual link latency using a photodiode; measure the audio link latency using a sound card loopback or hardware loopback; calculate the combined latency data and determine the output link that needs to be advanced or delayed; limit the compensation amount to the range of -10ms to +30ms; if the measured value exceeds this range, stop automatic compensation and prompt for device replacement or device configuration adjustment.

[0052] In some embodiments of this disclosure, sound stimuli are output through an audio player, which includes multiple channels. Before outputting sound stimuli through the audio player, the multimodal human cognitive training method further includes: determining the delayed channel among the multiple channels and the delay compensation amount corresponding to the delayed channel; determining the number of offset sampling points in the buffer based on a preset audio sampling rate and delay compensation amount, wherein the buffer stores sound stimuli; and adjusting the playback position of the delayed channel in the buffer based on the number of offset sampling points.

[0053] Specifically, the multi-channel audio player adopts a "single-instance generation, multi-channel distribution" architecture. All channels share the same waveform generation engine (used to generate sound stimuli), phase accumulator, and master clock timestamp data, and calculate the current sampling point of each channel in the same audio callback.

[0054] Let the delay compensation amount of the i-th channel (delay channel) be... The unit is microseconds, and the preset audio sampling rate is f. s To convert the time-domain delay compensation into the discrete-domain offset, the number of offset sampling points of the channel in the circular buffer is calculated according to formula (5). : (5) set up Let be the amplitude coefficient of the i-th channel. Let be the number of offset sampling points for that channel, then the output of the i-th channel... As shown in formula (6): (6) In some embodiments of this disclosure, after outputting sound stimuli, the method further includes: when the user performs a cognitive training task, obtaining the playback position of the buffer corresponding to the multi-channel, wherein the buffer determines the playback position in the form of a pointer; when the difference between the playback positions of any two channels is greater than a preset threshold and less than or equal to a reset threshold, adjusting the pointer of the phase accumulator or the buffer; when the difference between the playback positions of any two channels is less than or equal to the preset threshold, continuing to output sound stimuli based on the current playback position; when the difference between the playback positions of any two channels is greater than the reset threshold, resetting the buffer and writing it to an exception log.

[0055] Specifically, the playback position of each channel is read every 100ms. Taking the left and right channels (2 channels) as an example, the difference in sampling position between the channels is recorded. The definition is shown in formula (7), where, This indicates the number of offset sampling points corresponding to the left channel. This indicates the number of offset sampling points corresponding to the right channel: (7) Playback continues as long as the difference in sampling position does not exceed 5 (preset threshold) sampling points. When the difference exceeds 5 sampling points but is still within the range that can be gradually corrected (not exceeding the reset threshold), the playback position difference between channels is gradually reduced by slightly changing the increment of the phase accumulator or moving the current read position of the buffer forward or backward by the corresponding number of sampling points. When the difference exceeds the reset threshold or increases for multiple consecutive detection cycles, the audio stream is reinitialized and an exception log is written. The buffer read pointer indicates the sampling point position to be read from and output from the circular buffer next.

[0056] The logical flow is as follows: generate a common waveform (sound stimulus) → read the amplitude and delay parameters of each channel → calculate the number of offset sampling points → read the corresponding samples from the circular buffer → output the data of each channel → compare the playback position → perform maintenance, progressive compensation or audio stream reset.

[0057] In some embodiments of this disclosure, the offline calibration parameters further include a calibration compensation amount between vision and sound. Before synchronously outputting sound and visual stimuli, the method further includes: calibrating the display and audio player, performing multiple measurements during the calibration process to obtain the visual peak time, audio peak time, and calibration measurement time for each measurement; determining a calibration delay based on the visual peak time and audio peak time obtained for each measurement; determining the absolute value of the calibration delay, using the calibration delay as a calibration compensation amount when the absolute value is less than or equal to a calibration threshold, and adjusting the output time of the sound or visual stimulus based on the calibration delay when the absolute value is greater than the calibration threshold; and obtaining and storing the calibration result based on the device information of the display, the device information of the audio player, the calibration measurement time, and the calibration compensation amount.

[0058] Specifically, offline calibration is performed upon first run, upon change of display device, upon change of audio device, or upon change of driver configuration. The time corresponding to the start of offline calibration is the calibration measurement time. During calibration, visual white frames and audio pulses are output synchronously, and the actual output time is detected using a light sensor and a microphone or sound card loopback channel, respectively.

[0059] Let the visual peak time obtained from the i-th measurement be . The audio envelope peak time (audio peak time) is Then the calibrated acoustic-optical delay As shown in formula (8): (8) Measure continuously N times (corresponding to N cycles), preferably N equals 10, and take the average value as the calibration delay. As shown in formula (9): (9) This embodiment specifies A positive value indicates that sound occurs after sight. A negative value indicates that sound precedes vision. To compensate for calibration delay, a phase compensation amount is written in the opposite direction. As shown in formula (10): (10) When the absolute value of the calibration delay does not exceed the default threshold (calibration threshold) of 5ms, the current parameters can be maintained and the measurement results recorded, and the calibration delay can be used as the calibration compensation amount; when its absolute value exceeds 5ms, the corresponding output link should be advanced or the other output link should be delayed according to the phase compensation amount.

[0060] For example, play synchronized visual white frames and 1ms audio click pulses for 10 cycles of 40Hz; detect the peak visual brightness and peak audio envelope in each cycle; calculate the calibration acoustic-optical delay cycle by cycle; average the results of multiple measurements to obtain the calibration delay; write the phase compensation parameters according to the delay direction; and save the device combination (device information), calibration measurement time, and calibration results.

[0061] During the user's cognitive training task, the actual playback position of the audio driver report is compared with the theoretical phase corresponding to the unified master clock every 30 seconds to obtain the time drift between the two. When the absolute value of the drift exceeds 2ms, the number of sampling points corresponding to 0.1ms is used as a correction step, and the reading position of the audio buffer is moved forward or backward according to the drift direction; the cumulative movement within a 30-second detection cycle does not exceed 1ms, in order to gradually correct the playback position and avoid perceptible sound discontinuity.

[0062] In some embodiments of this disclosure, after synchronously outputting sound stimuli and visual stimuli, the method further includes: downsampling the sound stimuli and visual stimuli according to a preset time interval and a preset length, and obtaining the audio envelope of the output sound stimuli and the visual sequence of the corresponding visual stimuli; performing cross-correlation calculation based on the audio envelope and visual sequence to determine the delay index, wherein the delay index is the maximum value obtained by the cross-correlation calculation; updating the calibration compensation amount based on the delay index and the downsampling sampling rate, using the calibration compensation amount as the initial value, to obtain the updated calibration compensation amount; and adjusting the playback position of the sound stimuli in the buffer based on the updated calibration compensation amount, wherein the buffer stores the sound stimuli and the buffer determines the playback position in the form of a pointer.

[0063] Specifically, an audio envelope of 2 seconds (preset length) is extracted every 10 seconds (preset time interval) and downsampled to 1kHz; simultaneously, a visual theoretical modulation sequence (visual sequence) within the same time window is generated. Let the audio envelope be x[n] and the visual sequence be y[n], then their cross-correlation function is... As shown in formula (11): (11) Find the maximum value of the cross-correlation function within the allowed search delay range, and its corresponding delay index. As shown in formula (12): (12) Let the sampling rate after downsampling be The real-time audio-visual delay obtained in this real-time detection As shown in formula (13): (13) To reduce compensation jumps caused by noise in a single detection, the system uses an exponentially weighted moving average to update the current compensation value. (Calibration compensation amount), as shown in formula (14): (14) in, To calibrate the compensation amount, j represents the j-th real-time verification, and α is preferably taken as 0.3. When the real-time audio-visual delay... When the absolute value of the synchronization time is no more than 5ms, it is considered to be strictly synchronized; when it is greater than 5ms but no more than 10ms, training can continue, but the synchronization quality will be marked as decreased; when it is greater than 10ms, a warning will be triggered and the stimulus data segment will be marked as low quality.

[0064] The real-time synchronization verification logic is as follows: extract the audio envelope → generate a visual sequence → calculate the cross-correlation → find the correlation peak → convert the time delay → exponentially smooth the compensation value → determine whether the buffer needs to be adjusted or low quality needs to be marked.

[0065] Figure 2 This is a schematic diagram of a layered display disclosed in an embodiment of this disclosure.

[0066] like Figure 2 As shown, the display content is divided into a stimulus layer 20, a task layer 21, a feedback layer 201, and a safety control layer 202. The stimulus layer 20 is mainly located in the white or non-task display area outside the central training window (task layer 21). This area is modulated for brightness, color, grayscale, or transparent masking with a target frequency of 40Hz. The task layer 21 is located in the central training content window and is used to stably present cognitive training questions, target objects, prompts, response buttons, or interactive objects. The feedback layer 201 is used to display information such as correct / incorrect, progress, score, and rest prompts. The safety control layer 202 is used for pausing, exiting, discomfort feedback, and abnormal reminders, and has the highest interaction priority. Normally, the feedback layer and the safety control layer are located within the task layer, and their positions can be manually adjusted according to task requirements.

[0067] The white peripheral area of ​​the training interface (task layer 21) receives 40Hz visual flicker, while the central training window displays the cognitive training content. This design ensures that patients receive continuous 40Hz visual rhythmic stimulation during training while preventing the cognitive training questions from being obscured or difficult to recognize by high-frequency flicker. Simultaneously, the headset (audio player) outputs 40Hz click sounds, 40Hz modulated sounds, 40Hz gated noise, or other rhythmic sounds, achieving synchronized visual and auditory stimulation.

[0068] The stimulation areas of stimulation layer 20 are divided into full-screen, edge, corner, and custom areas, as shown in Table 1: Table 1. Stimulation Area Types Corresponding to the Display Screen

[0069] Within a training segment, every T_switch seconds (default 60 seconds, configurable from 30 to 300 seconds), the stimulation area automatically switches to the next preset type. The switching order is, for example: full screen → edge area → corner area → edge area → full screen... (the order can be configured by the doctor). The switching is completed smoothly within 1-2 seconds, gradually reducing the stimulation intensity of the old area and increasing the stimulation intensity of the new area to avoid abrupt stimulation.

[0070] Depending on the task and stage, the task is set to trigger (initially configured by the doctor, and subsequently automatically generated by the agent for cognitive training tasks, and confirmed by the doctor): In task stages that require high concentration, the stimulation area is automatically reduced from "full screen" to "edge area" or "corner area" to reduce interference; in simple maintenance tasks or rest stages, it can be expanded to "full screen" to enhance rhythm induction. The time of each switch, the original area, the new area, and the triggering reason are all written to the stimulation log.

[0071] In some embodiments of this disclosure, the task configuration parameters include a modulation mode and modulation parameters corresponding to the modulation mode, as well as a sound mode and sound parameters corresponding to the sound mode; based on the task configuration parameters, generating sound stimuli and visual stimuli corresponding to the cognitive training task includes: generating visual stimuli corresponding to the cognitive training task based on the modulation mode and modulation parameters, wherein the modulation mode includes at least one of brightness modulation, grayscale modulation, and transparent mask modulation; generating sound stimuli corresponding to the cognitive training task based on the sound mode and sound parameters, wherein the sound mode includes any one of click sound, modulation sound, and gated noise.

[0072] Specifically, based on the modulation mode and parameters selected by the doctor, visual stimuli can be generated. Modulation modes include, but are not limited to, brightness modulation, grayscale modulation, and transparent mask modulation. Modulation parameters include, for example, modulation frequency, modulation amplitude, modulation waveform, and duty cycle. Based on the sound mode and parameters selected by the doctor, sound stimuli can be generated. Sound modes include, but are not limited to, click sounds, modulation sounds, gated noise, pink noise, or other audible rhythmic sounds. Sound parameters include, for example, amplitude, and pulse shape. Audio buffer data is generated before or during the cognitive training task. This audio buffer data forms an amplitude envelope or pulse sequence with a 40Hz period. When an output command is received, the amplitude envelope or pulse sequence is output as the sound stimulus.

[0073] The doctor's input can be pre-configured with a modulation mode identifier to select brightness modulation, grayscale modulation, or transparent mask modulation. At the start of each display frame, the stimulus phase and current stimulus area are recalculated based on a unified master clock. Then, the corresponding modulation algorithm is invoked. After the current display frame is completed, the system records the frame number, timestamp, stimulus area, modulation mode, duty cycle, and actual output status to ensure unified time alignment with sound stimuli, cognitive training task events, and patient responses, as well as subsequent quality verification.

[0074] Using a high-precision clock as the master clock for the session, a visual stimulus schedule, an auditory stimulus schedule, and a cognitive training task event schedule are generated at the start of training. The period of the 40Hz target rhythm is 25ms. The stimulus phase at any given moment is calculated based on the master clock for the session, and the visual brightness state and auditory amplitude envelope are synchronized to that phase.

[0075] Using the unified master clock data of the training session as the time reference, the pixels to be rendered in the display interface are processed frame by frame. Let t be the current time, f be the target frequency, and T be a stimulation period; when the target frequency is 40Hz, a stimulation period is 25ms, and the stimulation period is calculated as shown in formula (15): (15) Convert the current moment into a normalized phase within a stimulus cycle. As shown in formula (16), The duty cycle threshold d of the current modulation mode is compared to determine whether the display frame uses high-brightness output, low-brightness output, first mask output, or second mask output.

[0076] (16) The overall logical process is: read task configuration parameters (such as target frequency, stimulation period, etc.) → obtain the unified main clock time → calculate the current 40Hz phase → determine the current stimulation area → judge whether the pixel is located in the stimulation area → select luminance modulation, grayscale modulation or transparent mask modulation → calculate the output luminance or color → generate the current display frame → record the current modulation state and timestamp.

[0077] (1) For luminance modulation: the luminance modulation is used to make the stimulation area alternately present a high-luminance state and a low-luminance state according to the set duty cycle in each 40Hz period. I_max is the target luminance coefficient in the high-luminance phase, and I_min is the target luminance coefficient in the low-luminance phase. The specific content is as follows: Read the current time t, and calculate the normalized phase . Judge whether the current pixel point p is located in the stimulation area Ω(t). If it is not located in the stimulation area, the original luminance is maintained, as shown in formula (17), where I_base(p) represents the original luminance of the pixel point p without stimulation modulation, d represents the duty cycle, represents the output corresponding to this pixel point: (17) If the current pixel point is located in the stimulation area, the current phase is compared with the duty cycle d. When <d, the current moment is in the high-luminance phase of the current period, and the maximum luminance coefficient is output, as shown in formula (18): (18) When φ(t)≥d, the current moment is in the low-luminance phase of the current period, and the minimum luminance coefficient is output, as shown in formula (19): (19) The duration of the high-luminance state and the low-luminance state in one period is shown in formula (20): (20) The preferred parameter ranges are: I_max is 0.8 to 1.0, I_min is 0 to 0.3, and the duty cycle d is 0.3 to 0.7; in a preferred embodiment, d is 0.5, at this time, the high-luminance state and the low-luminance state each last for 12.5ms within the 40Hz period.

[0078] The logical process of luminance modulation is: calculate the current phase → judge whether the pixel point is located in the stimulation area. When the judgment result is "No", the original luminance is output; when the judgment result is "Yes", the current phase φ(t) is further compared with the duty cycle d: when φ(t)<d, the high luminance is output, and when φ(t)≥d, the low luminance is output.

[0079] (2) For grayscale modulation: This is used to dynamically mix the original image within the stimulation area with a preset grayscale value. The mixing coefficient α(t), which changes periodically with time, controls the output image to be closer to the original image or closer to the target grayscale value. To avoid confusion with brightness modulation parameters, α_min and α_max represent the minimum and maximum values ​​of the grayscale mixing coefficient, respectively, and G represents the target grayscale value. The specific details are as follows: Read the original brightness I_base(p) of the current pixel and determine whether the pixel is located within the stimulation region. If it is not located within the stimulation region, maintain the original brightness.

[0080] If sinusoidal grayscale modulation is used, the grayscale mixing coefficient α(t) is calculated according to the current time, as shown in formula (21), so that the grayscale change smoothly alternates between the minimum and maximum values. Here, φ_0 is an optional initial phase; when no initial phase is set, φ_0 is 0.

[0081] (twenty one) If triangular wave modulation is used, the normalized triangular wave is first calculated based on the current phase. Then, the grayscale mixing coefficient is determined based on this, as shown in formulas (22) and (23): (twenty two) (twenty three) If square wave modulation is used, the modulation is switched between α_max and α_min according to the duty cycle. The maximum mixing coefficient is used when the current phase is less than the duty cycle, and the minimum grayscale mixing coefficient is used when the current phase is greater than or equal to the duty cycle, as shown in formula (24): (twenty four) By weighting and mixing the original brightness with the target grayscale value according to the grayscale mixing coefficient, the output corresponding to pixel p can be obtained. As shown in formula (25): (25) When α(t) is small, the output is closer to the original image; when α(t) is large, the output is closer to the target grayscale value. For pixels outside the stimulus region, I_out(p,t) = I_base(p).

[0082] The preferred parameter range is as follows: α_min and α_max are set within the range of 0.2 to 0.8 according to the stimulus intensity, and α_min < α_max; the modulation waveform can be selected as sine wave, triangle wave or square wave; the target gray level value G is a normalized gray level value in the range of 0 to 1.

[0083] The grayscale modulation logic flow is as follows: determine whether the pixel is located in the stimulation area. If the result is "no", directly output the original brightness. If the result is "yes", read the modulation waveform and calculate the current grayscale mixing coefficient α(t). After weighted mixing of the original brightness and the target grayscale value, output the modulated grayscale result.

[0084] (3) For transparent mask modulation: A transparent color layer is superimposed on the cognitive training task layer, and the mask color or mask transparency changes according to a 40Hz rhythm. This method can apply periodic visual stimulation to a specified area while preserving the visibility of the task content, as follows: A transparent overlay layer is created above the task interface, and the actual coverage area of ​​the overlay layer is limited according to the region configuration; the pause, exit, and discomfort feedback controls maintain the highest interaction priority, and are selected between the first overlay color C_1 and the second overlay color C_2 according to the current phase and duty cycle, as shown in formula (26), where C_overlay(t) is the current overlay color: (26) The first mask color can be set to semi-transparent white, and the second mask color can be set to semi-transparent black; the above colors are only preferred embodiments, and other light colors, dark colors, or combinations of the same color with different transparency can also be used according to the training scheme.

[0085] For pixels within the stimulation area, perform transparency blending according to the current mask opacity 'a' to obtain the color output of the current pixel. As shown in formula (27): (27) Where C_base(p) is the original pixel color, C_overlay(t) is the current mask color, and a is the current mask opacity. This represents the color output of the current pixel. For pixels outside the stimulus region, C_out(p,t) = C_base(p).

[0086] Will Write the current display frame and record the mask color, transparency, stimulation area, duty cycle, and corresponding timestamp.

[0087] The preferred parameter range is as follows: the opacity 'a' of the first mask and the second mask is 0.05 to 0.30; the first mask is preferably white or light-colored, and the second mask is preferably black or dark-colored; the duty cycle 'd' is 0.3 to 0.7.

[0088] The transparent mask modulation logic flow is as follows: calculate the current phase → determine whether the pixel is located in the stimulation area. If the result is "no", output the original color; if the result is "yes", select the first mask color or the second mask color according to the comparison result of the current phase and the duty cycle, read the opacity of the selected mask, perform transparency blending and output the composite color.

[0089] It's important to note that monitor refresh rates must be integer multiples of 80Hz, such as 80Hz, 160Hz, and 240Hz. Periodic bright / dark or strong / weak modulation sequences are generated in integer frames. For example, on a 240Hz monitor, one 40Hz cycle corresponds to 6 frames, which can be achieved by using 3 frames of high brightness and 3 frames of low brightness to form a 50% duty cycle. If the refresh rate is not an integer multiple of 40Hz, a phase accumulator is used to generate a brightness sequence close to 40Hz, and the period error is calculated. When the error exceeds a threshold, the system prompts the user to adjust the display refresh rate or replace the display device.

[0090] The sound generation methods in this application are illustrated using click sounds, modulation sounds, and gated noise as examples, and are not intended to limit this application. The details are as follows: (1) Click tone generation: First, calculate the single stimulus period based on the target frequency, and convert the current time of the unified master clock into the normalized phase within the period. Let the target frequency be f0, the period be T0, and the normalized phase be θ(t). The calculation method is shown in formulas (28) and (29): (28) (29) Let A be the overall amplitude, ρ be the proportion of the pulse width to one cycle, E(t) be the conversation-level fade-in / fade-out envelope, h(θ) be the pulse shaping function, and click tone output. Determine using the following formula (30): (30) Current phase When the pulse is in the pulse-on interval, the output is an audio sample value that has been shaped and envelope-controlled; when the current phase is in other intervals, the output is zero. The pulse shaping function h(θ) can be in rectangular, Gaussian, or raised cosine form, as shown in formulas (31), (32), and (33): (31) (32) (33) Where σ is the Gaussian pulse width parameter, rectangular pulses are simple to implement but have more high-frequency components; Gaussian pulses and raised cosine pulses have smoother start and end edges, which can reduce transient pops.

[0091] The specific execution process is as follows: read the target frequency, amplitude, pulse width ratio and pulse shape; calculate the current phase according to the unified master clock; determine whether the current phase is in the pulse opening interval; if it is in the opening interval, calculate the pulse shaping value and multiply it by the amplitude and fade-in / fade-out envelope, otherwise output zero value; write the sampled value into the audio buffer and record the corresponding timestamp.

[0092] The logic flow is as follows: Read parameters → Calculate the 40Hz period and the current phase → Determine whether the current phase is in the pulse opening interval. If the result is "yes", select the pulse shaping function and calculate the pulse sampling value; if the result is "no", set the current sampling value to zero → Write the current sampling value to the audio buffer.

[0093] (2) Modulation tone generation: First, a carrier signal within the audible frequency range is generated, assuming... For carrier frequency, Given the initial phase of the carrier, the carrier signal... As shown in formula (34): (34) A 40Hz modulation envelope with values ​​ranging from 0 to 1 is then generated. The modulation envelope can be a sine wave, a square wave, or a triangle wave, as shown in formulas (35), (36), and (37), respectively: (35) (36) (37) Where d is the square wave modulation duty cycle, and φ0 is the initial modulation phase. Let D be the modulation depth, ranging from 0 to 1, then the final amplitude modulation tone... The output is shown in formula (38): (38) When D equals 1, the carrier amplitude is fully modulated between zero and maximum values; when D equals 0, no amplitude modulation occurs; when D is between 0 and 1, the modulation depth varies proportionally.

[0094] The specific execution process is as follows: read the carrier frequency, carrier type, modulation waveform, modulation depth, and duty cycle; generate the carrier value at the current sampling time; calculate the 40Hz modulation envelope based on the unified master clock and the selected waveform; multiply the carrier, modulation envelope, amplitude, and fade-in / fade-out envelope; and write the result to the audio buffer.

[0095] The logical flow is as follows: generate carrier → calculate 40Hz phase → generate modulation envelope → apply modulation depth → superimpose fade-in and fade-out envelope → output audio sample value.

[0096] (3) Gated noise generation: First, random noise with a value range of -1 to 1 is generated, and the noise spectrum can be limited to the set lower limit frequency through a bandpass filter. With upper limit frequency between: (39) in, The noise before filtering is represented by U(-1,1), which indicates a uniform distribution in the interval from -1 to 1. This is the noise signal after bandpass filtering. Indicates the lower limit frequency is The upper limit frequency is Bandpass filtering is applied; when bandpass filtering is not enabled, n(t) = .

[0097] A hard-gated envelope is generated based on the current 40Hz phase and duty cycle. : (40) To reduce the popping noise generated at the gate edges, the duration can be set at the opening and closing edges. The linear rising and falling edges. Let r be the proportion of the rising or falling edge to a single 40Hz cycle, then the soft-gated envelope can be expressed as: (41) The final output of the gated noise is: (42) The specific execution process is as follows: generate white noise, pink noise, or brown noise; perform bandpass filtering according to the configuration; calculate the current 40Hz phase according to the unified master clock; generate a gated envelope according to the duty cycle, and apply rising and falling edges at the gated edges; multiply the gated envelope, noise, amplitude, and session envelope and write them into the audio buffer.

[0098] The logic flow is as follows: generate noise → perform optional bandpass filtering → calculate 40Hz phase → determine gating status. When the result is "on", calculate the soft gating envelope based on the current phase and multiply the envelope with the noise, amplitude and session envelope to obtain the output sample value. When the result is "off", set the output sample value to zero → write to the audio buffer.

[0099] Volume, envelope, duty cycle, fade-in / fade-out, and rest interval can be set. Table 2 defines the different parameter types for sound stimuli, and Table 3 defines the fade-in / fade-out time parameters. Table 2 Definitions of Different Parameter Types of Sound Stimulation

[0100] Table 3. Fade-in / Fade-out Time Parameters

[0101] Patients can rest according to a preset cycle: After the training segment begins, the `rest_timer` is activated. When `rest_timer >= rest_interval_min`, a rest prompt is displayed in the center of the screen: "A short rest is about to begin, please relax." After a countdown of `rest_tip_advance` seconds, the following actions are executed: Audio: 300ms linear fade-out; Visual: 40Hz flicker pauses, and the area returns to static white / gray; Cognitive task: Pauses the appearance of new questions; After a rest duration of `rest_duration` seconds, the following automatic recovery occurs: Audio: 300ms linear fade-in; Visual: 40Hz flicker resumes; Cognitive task: Continues.

[0102] Patients can actively request a rest: Patients can request a rest at any time through the "I want to rest" button on the interface. The rest will immediately execute fade_out_duration_pause (200ms fade-out), pausing the audio-visual stimulation and tasks. There is no upper limit to the rest duration. After the patient clicks "Continue Training", a 200ms fade-in will be executed to resume training. The log will record: patient_request_rest + duration.

[0103] In some embodiments of this disclosure, the cognitive training tasks include multiple tasks, each corresponding to a different cognitive domain. The multimodal cognitive training method further includes: determining the user's impairment rating vector for multiple cognitive domains based on the cognitive training results, and obtaining the training weight vector for each cognitive training task; performing a weighted calculation based on the impairment rating vector, the training weight vector, and the pre-configured priority weight vector to obtain a similarity result; performing a preliminary screening process on the cognitive training tasks based on the impairment rating vector, the training weight vector, and the priority weight vector to obtain a preliminary screening of cognitive training tasks; and performing a sorting and screening process on the preliminary screening of cognitive training tasks based on the similarity result to adjust the next cognitive training task.

[0104] Specifically, the intelligent agent is constrained by safety rules. It is not an independent diagnosis and treatment system, nor does it replace medical staff in diagnosis. Instead, it is embedded in the training program, quality monitoring and reporting assistance modules of the cognitive rehabilitation training software to improve the efficiency of training program generation, individualized adjustment, risk control, synchronous quality monitoring and report generation.

[0105] Input data includes: patient basic information, cognitive assessment results, training goals set by medical staff, previous training records, task accuracy, reaction time, completion rate, subjective fatigue score, feedback on dizziness / eye discomfort / hearing discomfort, training interruption records, monitor refresh rate, audio sampling rate, missing frame log, synchronization quality evaluation, and training constraints set by medical staff.

[0106] Output data includes: recommended training task combinations, training interval duration, task difficulty, stimulus intensity range, volume range, rest interval, next training suggestion, abnormal risk warning, synchronization quality warning, and training report summary. Outputs involving treatment plans, stimulus intensity boundaries, and treatment course adjustments require confirmation from medical staff before taking effect.

[0107] The solution-generating agent selects candidate tasks from the task library based on the patient's cognitive domain impairment and historical training performance, thus obtaining the next cognitive training task.

[0108] Cognitive training tasks include, but are not limited to, attention maintenance tasks, visual search tasks, working memory tasks, short-term memory encoding tasks, executive function tasks, response inhibition tasks, processing speed tasks, spatial localization tasks, and multi-task switching tasks, each corresponding to a different cognitive domain. Each task can be presented during 40Hz audio-visual stimulation. The cognitive training tasks are shown in Table 4. Table 4 Cognitive Training Task List

[0109] Let the patient's impairment score vector across m cognitive domains be . The training weight vector for the corresponding cognitive domain is: , Assigning weights (priority weight vectors) to the clinical priority or medical care of the i-th cognitive domain, the weighted cosine similarity between the patient and the task is calculated as shown in formula (43): (43) Similarity results The higher the priority weight vector, the more aligned the task training focus is with the patient's impaired cognitive domain. When no separate priority weight vector is configured, each... It can be uniformly set to 1.

[0110] The screening process is as follows: Read the patient's cognitive domain impairment scores and task library feature vectors; for cognitive domains with impairment scores (scores corresponding to the impairment score vector) not lower than 0.5, prioritize tasks with training weights (weights corresponding to the training weight vector) not lower than 0.6; reduce the matching weight for cognitive domains with impairment scores lower than 0.3; if a task has a weight higher than 0.7 in a cognitive domain with impairment scores lower than 0.2, exclude that task; calculate and rank the weighted cosine similarity of the remaining tasks (cognitive training tasks after initial screening); under the constraint of covering at least 3 different impaired cognitive domains, select the top 3 to 5 tasks to form a draft plan. The task plan must be confirmed by medical staff before taking effect, resulting in the next cognitive training task. The agent only provides candidate task ranking and explanation information.

[0111] In some embodiments of this disclosure, adjusting the next cognitive training task based on the cognitive training results includes: determining accuracy data, reaction time data, completion rate data, and penalty item data based on the cognitive training results; performing weighted processing on the accuracy data, reaction time data, completion rate data, penalty item data, first weight, second weight, third weight, and fourth weight to obtain comprehensive performance score data; and adjusting the next cognitive training task based on the comprehensive performance score data, target score data, safety upper limit data, and safety lower limit data.

[0112] Specifically, the accuracy rate (accuracy data), reaction time improvement (reaction time data), task completion rate (completion data), and adverse feedback (penalty data) were normalized. The normalized accuracy rate value was... As shown in formula (44), where clip(x,0,1) restricts x to the range of 0 to 1: (44) Improved normalization value during reaction As shown in formula (45): (45) in, The average reaction time during the benchmark training phase or under preset baseline conditions. The average reaction time for this training exercise is represented by the same time unit for both; the constant 0.5 is the baseline shift for the reaction time improvement term, making the time before improvement equal to the average reaction time before improvement. Set to 0.5; clip(·,0,1) is used to limit the result to the range of 0 to 1.

[0113] Normalized value of task completion rate As shown in formula (46): (46) in, This represents the number of questions, trials, or subtasks completed in this training session. The total number of questions, trials, or subtasks that should be completed according to the training plan and using the same counting method, and >0; The value range is from 0 to 1.

[0114] Inappropriate penalty items The number of discomfort triggers is calculated based on the number of times this training is conducted, and an upper limit is set, as shown in formula (47): (47) in, This represents the number of discomfort events identified during this training session through patient-reported incidents or system security rules; a coefficient of 0.1 indicates a penalty of 0.1 for each discomfort event, and 0.5 is the upper limit for the penalty. The value range is from 0 to 0.5.

[0115] Overall performance rating data As shown in formula (48): (48) in, As the first weight, As the second weight, As the third weight, As the fourth weight, the default weight is set to: The higher the overall performance score, the better the patient performs in terms of accuracy, reaction speed, and completion rate, and the fewer adverse events.

[0116] Let θ be the parameter to be adjusted by the agent (as shown in Table 5). This is the current value (task configuration parameter). This is the maximum allowable single-step adjustment amount for this parameter. and Given the lower safety limit and upper safety limit data respectively, the recommended new parameter values ​​are... (The configuration parameters for the next task) are shown in formula (49): (49) Table 5. Parameters to be adjusted for the agent.

[0117] In some embodiments of this disclosure, the next cognitive training task is adjusted based on comprehensive performance score data, target score data, safety upper limit data, and safety lower limit data. This includes: when the comprehensive performance score data is greater than the target score data, increasing the training difficulty or extending the training time of the next cognitive training task based on the next task configuration parameters, wherein the increased training difficulty or extended training time meets the safety upper limit data; when the comprehensive performance score data is less than or equal to the target score data, decreasing the training difficulty of the next cognitive training task, increasing the rest time, and decreasing the parameters corresponding to visual or auditory stimuli based on the next task configuration parameters, wherein the decreased training difficulty, increased rest time, and decreased parameters meet the safety lower limit data.

[0118] Specifically, when the score is higher than 0.7 (target score data), the agent tends to increase the training difficulty or extend the training interval within the safety boundary; when the score is lower than 0.7, the agent tends to reduce the difficulty, reduce the stimulus experience parameters, or increase the rest. For parameters with opposite directions of change, such as rest interval, the same update logic is executed after setting the direction coefficient in the parameter definition.

[0119] The specific process of the intelligent agent adjusting the task configuration parameters is as follows: calculate each normalized index; calculate the comprehensive performance score data; calculate the candidate adjustment amount (such as the maximum single-step adjustment amount) based on the degree to which the comprehensive performance score data deviates from the target score data by 0.7; apply the upper and lower limits of parameter safety and the single-step change limit; check the discomfort locking conditions and medical staff permissions; output adjustment suggestions and wait for medical staff confirmation.

[0120] In some embodiments of this disclosure, adjusting the next cognitive training task further includes: obtaining abnormal state data of the user performing the cognitive training task; and adjusting the next cognitive training task based on the abnormal state data.

[0121] Specifically, the intelligent agent performs safety monitoring, monitoring abnormal state data such as dizziness, nausea, eye discomfort, auditory discomfort, excessive fatigue, and frequent pauses. Abnormal state judgment is based on multi-dimensional physiological and behavioral indicators, and adopts a threshold trigger + continuous confirmation recognition logic: visual discomfort is triggered by the patient actively clicking the "visual discomfort" button, blinking frequency >30 times / minute (if eye tracking is supported), or eye closure duration >2 seconds; auditory discomfort is triggered by the patient clicking the "auditory discomfort" button, frequently adjusting the headphone volume (≥3 times within 1 minute), or actively lowering the system volume; cognitive fatigue is triggered by reaction time >40% longer than baseline, 3 consecutive task errors, accuracy rate lower than random level (e.g., accuracy rate <50% in a two-choice task), or subjective fatigue score ≥7 / 10; device malfunction is triggered by frame loss rate ≥1%, sound-light phase difference >10ms, audio buffer underload, or display refresh rate fluctuation >5%; severe emergency state (dizziness, nausea, tinnitus) is triggered by the patient clicking the emergency button or reporting discomfort twice consecutively. The identification logic employs a dual confirmation process: a single trigger enters the observation state and records a log; if the same anomaly is triggered again within 60 seconds or a single trigger exceeds the threshold limit (e.g., closing eyes for more than 5 seconds), a genuine anomaly is confirmed, and corresponding actions are executed (reducing stimulus intensity / pausing training / emergency stop). All abnormal events are written to the safety log and marked with a timestamp.

[0122] In some embodiments of this disclosure, the cognitive training result includes a quality assessment result. Adjusting the next cognitive training task based on the cognitive training result includes: performing a quality assessment on the cognitive training result to obtain a quality assessment result; determining the cognitive assessment result as valid data when the quality assessment result is greater than or equal to a first quality assessment threshold; determining the cognitive assessment result as partially valid data and marking it as low-quality data when the quality assessment result is greater than or equal to a second quality assessment threshold and less than or equal to a third quality assessment threshold; determining the cognitive assessment result as invalid data and generating a prompt message when the quality assessment result is less than the second quality assessment threshold; and adjusting the next cognitive training task based on the valid data and / or partially valid data.

[0123] Specifically, training data quality is determined by four dimensions: synchronization quality, task participation, behavioral effectiveness, and device status. Each dimension is converted to a score from 0 to 100, and the overall quality score (quality assessment result) is calculated. The calculation method is shown in formula (50): (50) in, This indicates the sub-score of the acoustic-optical synchronization quality. The sub-score represents the degree of task participation. Sub-scores representing behavioral effectiveness This indicates the device status sub-rating. This is a deduction item for abnormal behavior.

[0124] Acoustic-optical synchronization quality sub-score According to the acoustic-optical phase difference (e.g., real-time audio-visual delay) The determination is as shown in formula (51): (51) Task participation rating Based on task completion rate As shown in formula (52): (52) Behavioral Validity Sub-Rating According to accuracy As shown in formula (53): (53) Equipment status sub-rating According to the frame rate As shown in formula (54): (54) Deductions for Abnormal Behavior This includes abnormally rapid responses with a reaction time of less than 200ms and answer patterns consistent with random guessing. Abnormally rapid responses will result in a deduction of 20 points, while random guessing patterns will result in a deduction of 30 points.

[0125] Data is considered valid when the overall score (quality assessment result) is not lower than 70 points (first quality assessment threshold); data is considered partially valid and marked as low quality when the overall score is between 50 (second quality assessment threshold) and 69 points (third quality assessment threshold), and is not used for agent adaptive parameter tuning; data is considered invalid and prompts for retraining when the overall score is lower than 50 points. If there is no interaction during the entire training session or the actual training time is less than 30% of the preset time, the quality assessment result does not need to be calculated further, and the data is directly considered invalid.

[0126] The logical flow is as follows: read synchronization, participation, behavior and equipment indicators → calculate four sub-scores → calculate abnormal deductions → obtain quality assessment results → perform effective, partially effective or ineffective classification → write to the quality log and decide whether to use it for efficacy analysis and agent parameter tuning.

[0127] In another embodiment, the device is not limited to a specific brand of display, headphones, or computer hardware, as long as the hardware can be controlled by software to output visual and auditory stimuli. Visual stimuli are not limited to pure black and white flickering, but can also be grayscale, color, brightness, contrast, transparent masks, patterns, or 40Hz modulation of local areas; preferably, the non-task white area is modulated at 40Hz while the central task window is stably displayed. Auditory stimuli are not limited to a single sine wave, but can also be gated tones, impulse sounds, white noise, pink noise, voice masking tones, or other audible signals modulated at 40Hz.

[0128] In another embodiment, the agent can be deployed locally, on a server, or in a hybrid architecture, and can employ rule-based models, machine learning models, large language models, knowledge base retrieval enhancement models, or combinations of the above models, but its output should be constrained by security rules, permissions, and logging mechanisms. Cognitive training tasks can be extended according to different diseases or rehabilitation goals, and are not limited to MCI; they can also be used for other subject groups requiring cognitive function training or research.

[0129] This disclosure also provides a multimodal cognitive training system, which includes a display, an audio player, and a processor. The display is used to show the configuration interface of the multimodal cognitive training task, the cognitive training task, and visual stimuli, and to receive user operation feedback during the execution of the cognitive training task. The audio player is used to output sound stimuli. The processor is used to execute the steps in the multimodal cognitive training method.

[0130] Figure 3 This is a schematic diagram of the structure of the multimodal human cognitive training system disclosed in this embodiment. Figure 4 This is a schematic diagram of the multimodal personnel cognitive training system disclosed in this embodiment; as shown... Figure 3 , Figure 4As shown, the input of the multimodal cognitive training system includes two parts: the doctor's end and the patient's end. The doctor's end performs tasks such as patient registration, application allocation, and training plan confirmation (next cognitive training task). The doctor selects a training task, for example, choosing a cognitive training task and checking the 40Hz audio-visual stimulation mode. The patient's end logs in, enters the assigned training task, and the patient's display interface configures and invokes the session based on the cognitive training task selected by the doctor (loading task parameters, 40Hz stimulation parameters, and training duration), invoking visual stimulation (controlling the white area of ​​the screen to flash at a frequency of 40Hz) and adjusting audio stimulation. The system uses a headset to output 40Hz rhythmic sound to present cognitive training content (the cognitive training task content is stably displayed in the middle window). Then, it performs synchronous scheduling and closed-loop control (synchronizing the sound and light stimulation, task events, and agent regulation with a unified clock) to form a joint dialogue of cognitive training and 40Hz sound and light stimulation (training execution output). The patient executes the cognitive training task to obtain the cognitive training results. The output process is recorded (stimulation parameters, task events, response data, and logs are recorded). The agent provides auxiliary output (generating risk warnings, difficulty suggestions, and training suggestions). Finally, a report is output (generating training results and rehabilitation reports).

[0131] For doctors: The existing modules from the manual, such as user login, information overview, application management, patient management, organization management, system management, and recycle bin, will be retained. The 40Hz function in this application is mainly integrated into the "Application Management" and "Patient Management" workflows: In Application Management, medical staff select or create rehabilitation applications and check or configure the 40Hz audio-visual stimulation enhancement mode in cognitive training tasks; in Patient Management, medical staff assign cognitive training tasks containing the 40Hz enhancement mode to specific patients through "Application Selection" and view the results and reports after training.

[0132] Patient-side: The patient-side retains the patient login and application details logic from the instruction manual. After logging in using their medical appointment number or ID card number, patients can view the applications they need to complete in the application details, and then click "Participate in Training" to enter the corresponding cognitive training task. Only after the patient enters the training task on the patient-side will the 40Hz visual stimulation layer, 40Hz sound stimulation, and task event recording be activated.

[0133] The relationship between the 40Hz function and cognitive training tasks: This solution does not design the 40Hz audio-visual stimulation as a separate treatment entry point independent of the product. Instead, it adds a "40Hz audio-visual stimulation enhancement mode" to the rehabilitation-type cognitive training tasks of the multimodal cognitive training software. Medical staff complete application management, patient management, training application selection, parameter confirmation, and result viewing on the existing doctor's end; patients log in on the existing patient end and enter the assigned cognitive training task in the application details. After entering the training, the software automatically overlays 40Hz visual stimulation and 40Hz sound stimulation. The 40Hz audio-visual stimulation is not a background stimulus played independently outside of cognitive training, but is invoked as an enhancement layer for rehabilitation-type cognitive training tasks. The doctor's end is responsible for "selection / confirmation," and the patient's end is responsible for "execution / feedback." The real technical implementation focuses on the synchronous scheduling of audio-visual stimulation within the training session (e.g., ...). Figure 5 As shown, a 40Hz cycle is 25ms, visual flicker can be 12.5ms bright and 12.5ms dark, sound pulse can generate a 1ms click sound at the beginning of each cycle, synchronously record phase, delay and quality indicators), interface layered presentation, task event alignment and agent security closed loop.

[0134] It should be noted that on the doctor's end, the selection of rehabilitation cognitive training tasks, configuration of the 40Hz enhancement mode, and patient assignment are completed; on the patient's end, after entering the assigned training in the application details, the 40Hz sound and light stimulation and cognitive training are performed.

[0135] A single training session (cognitive training task) may include baseline description, audiovisual comfort confirmation, audiovisual preview, training segments, rest periods between segments, training end feedback, and report generation. Each training segment can be configured with task type, segment duration, stimulus intensity, volume, duty cycle, task difficulty, and rest interval. Table 6 shows the switching between different states of the cognitive training task: Table 6. Switching between different states in the cognitive training task.

[0136] The specific data structures for the different data in this application are shown below: 1. Patient record data: patient number, age group, assessment results, main cognitive domains affected, audiovisual comfort record, precautions and medical notes.

[0137] 2. Training program data (cognitive training task): Session number, application name, task type, whether 40Hz enhancement mode is enabled, training segment duration, visual stimulus parameters, auditory stimulus parameters, rest interval, task difficulty, termination conditions, confirmation personnel and confirmation time.

[0138] 3. Stimulus log data: target frequency, period, visual refresh rate, audio sampling rate, visual phase table, audio envelope table, startup time, compensation amount, missed frame records, audio buffer records, and synchronization quality score.

[0139] 4. Task event data: Question number, stimulus presentation time, patient reaction time, reaction result, accuracy rate, reaction time, missed answers, cue usage, and changes in task difficulty.

[0140] 5. Safety incident data: pause, resume, exit, type of subjective discomfort, time of occurrence, handling measures, whether training was terminated, and whether medical review was required.

[0141] 6. Report data: training summary, trend statistics, synchronization quality description, agent-assisted suggestions, medical staff confirmation records, and subsequent training suggestions.

[0142] For ease of understanding, the following specific embodiments are provided in this application: Step 1: Medical staff log in to the doctor's terminal, select or create a rehabilitation cognitive training task in the application management, and enable the 40Hz sound and light stimulation enhancement mode.

[0143] Step 2: The software or intelligent agent generates a draft training plan (the next cognitive training task) based on patient information, past training performance, and medical goals set by the healthcare professionals. This plan includes the task type, training segment, 40Hz stimulus parameters, volume, rest interval, and safe termination conditions.

[0144] Step 3: Medical staff confirm or modify the plan on the doctor's end, and then assign the training task to the patient through the application in patient management.

[0145] Step 4: The patient logs in through the patient client and accesses the assigned cognitive training task in the application details. The software checks the display refresh rate, audio device, volume, and audiovisual comfort.

[0146] Step 5: After training begins, the white or non-task area flashes at a 40Hz rhythm, the central training window stably displays the cognitive task content, and the headphones output a 40Hz rhythmic sound synchronously.

[0147] Step Six: Record stimulus phase, task events, patient responses, pause / exit, discomfort feedback, missed frames, and audio buffer status, and have the agent generate suggestions such as maintaining, reducing difficulty, increasing rest, pausing, or terminating within safe boundaries.

[0148] Specifically, each task event is bound to a unified timestamp, including the appearance of the question, the appearance of the target, the patient's click or key press, correct / incorrect answer, the appearance of a prompt, and the start / end of a rest period. Task events are correlated with the phase of the audiovisual stimuli at the same moment for subsequent analysis of the patient's reaction speed, accuracy, missed answer rate, and fatigue trend under stimulus conditions. A high-precision master clock (accuracy ≤ 1ms) is activated at the start of the session, and all timestamps are the absolute offset of this clock (in ms). The visual flicker phase and the sound envelope phase are calculated in real time based on the master clock; therefore, the phase of the audiovisual stimuli at any given moment can be uniquely determined by the master clock.

[0149] Step 7: After training, the agent generates an execution report, which includes a training summary and suggestions for the next training session (cognitive training report). The report includes training duration, task type, stimulus parameters, completion rate, accuracy rate, reaction time changes, synchronization quality, patient feedback, abnormal events, agent-assisted suggestions, and medical confirmation records.

[0150] It is important to note that every suggestion from the agent (for the next cognitive training task), medical confirmation, parameter modification, automatic pause, and report generation is logged. The patient's side only provides training execution, pause, exit, and discomfort feedback; the therapist's side (doctor's side) can confirm the training plan; and the administrator's side can configure parameter boundaries, rule bases, and permission policies.

[0151] The multimodal cognitive training method and system proposed in this application adopts the following: (1) Using 40Hz audio-visual synchronous stimulation as an optional enhancement mode for rehabilitation cognitive training tasks in multimodal cognitive function testing and rehabilitation software, rather than an independent stimulation playback entry separate from the cognitive training software. This not only conforms to the existing doctor / patient application process, but also enables 40Hz stimulation to directly serve cognitive training tasks, enhancing the consistency between the patent solution and the actual software architecture of the product. (2) Using the software master clock to uniformly control the 40Hz visual stimulation of the display and the 40Hz sound stimulation of the headphones, and to synchronize the two. This achieves synchronous output of audio-visual stimulation without relying on dedicated audio-visual stimulation hardware, reducing deployment costs and improving the universality of training scenarios. (3) Proposing an interface layered structure of "40Hz flashing in the non-task white area + stable presentation of the central cognitive training window + 40Hz sound stimulation from the headphones". While ensuring the continuous existence of visual rhythmic stimulation, it avoids the training questions and interactive objects being obscured by flashing, improving task readability, patient compliance, and training safety. (4) The 40Hz audio-visual stimulation layer is superimposed on the cognitive training task layer, and the stimulation phase, task events and patient responses are recorded uniformly. This makes the "audio-visual stimulation - cognitive task - training performance" form a traceable closed loop, which is convenient for medical staff to review the training quality and changes in patients. (5) An audio-visual synchronization scheduling and quality monitoring mechanism is set up, including master clock, visual phase table, audio buffer, phase compensation, and missing frame / delay recording. This reduces the stimulation error caused by display refresh rate, audio buffer and system scheduling in ordinary computer environment and improves the standardization of training. (6) An intelligent agent constrained by safety rules is embedded in the cognitive rehabilitation training software. It is responsible for program generation, adaptive parameter tuning, synchronization quality analysis, safety prompts and report generation. This improves the efficiency of individualized training and clinical use, and at the same time reduces the risk of uncontrollable output of the intelligent agent through medical staff confirmation, permission control and log mechanism. (7) The intelligent agent takes patient cognitive assessment, task performance, discomfort feedback, equipment synchronization quality and historical training records as input. The training program no longer depends on a fixed template, but can be dynamically adjusted according to the patient status and equipment status, which enhances the continuity and interpretability of training. (8) Generate a structured training report that integrates stimulus parameters, synchronization quality, cognitive task performance, agent-assisted suggestions, and medical confirmation records. This facilitates clinical follow-up, treatment management, research statistics, and quality control.

[0152] Figure 6 This is a block diagram of the multimodal human cognitive training device disclosed in this embodiment.

[0153] like Figure 6 As shown, another embodiment of this application provides a multimodal cognitive training device 600, which includes: a first acquisition module 610, a generation module 620, a second acquisition module 630, and an adjustment module 640.

[0154] The first acquisition module 610 is used to acquire the task configuration parameters of the cognitive training task, and the task configuration parameters include at least the target frequency. The generation module 620 is used to generate sound stimuli and visual stimuli corresponding to the cognitive training task based on the task configuration parameters, and to output sound stimuli and visual stimuli synchronously according to the target frequency during the execution of the cognitive training task. The second acquisition module 630 is used to acquire the cognitive training results obtained by the user performing cognitive training tasks. Adjustment module 640 is used to adjust the next cognitive training task based on the cognitive training results.

[0155] It is understood that the multimodal cognitive training device 600 is used to implement the steps in the aforementioned multimodal cognitive training method, which will not be repeated here.

[0156] Figure 7 This is a block diagram of an edge computing device provided in an embodiment of this disclosure.

[0157] like Figure 7 As shown, the edge computing device in this embodiment includes a processor 701 and a memory 702, wherein the memory 702 stores programs or instructions that can run on the processor 701. When the program or instructions corresponding to the edge computing device are executed by the processor 701, the multimodal human cognitive training method described above can be implemented.

[0158] This disclosure also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the multimodal human cognitive training method described above.

[0159] The embodiments of this disclosure have now been described in detail. To avoid obscuring the concept of this disclosure, some details known in the art have not been described. Those skilled in the art can fully understand how to implement the technical solutions disclosed herein based on the above description.

[0160] While specific embodiments of this disclosure have been described in detail by way of examples, those skilled in the art should understand that the examples are for illustrative purposes only and not intended to limit the scope of this disclosure. Those skilled in the art should understand that modifications can be made to the above embodiments or equivalent substitutions can be made to some technical features without departing from the scope and spirit of this disclosure. In particular, as long as there is no structural conflict, the technical features mentioned in the various embodiments can be combined in any manner.

Claims

1. A multimodal cognitive training method, characterized in that, The method includes: Obtain task configuration parameters for a cognitive training task, wherein the task configuration parameters include at least the target frequency; Based on the task configuration parameters, sound stimuli and visual stimuli corresponding to the cognitive training task are generated, and the sound stimuli and visual stimuli are synchronously output according to the target frequency during the execution of the cognitive training task. Obtain the cognitive training results obtained by the user performing the cognitive training task; The next cognitive training task will be adjusted based on the cognitive training results.

2. The method according to claim 1, characterized in that, The step of obtaining the cognitive training results obtained by the user performing the cognitive training task includes: The timestamp data of the user performing the cognitive training task and the user's feedback data are obtained. Under the timestamp data, the sound stimulus and the visual stimulus are output synchronously. The timestamp data and the feedback data are based on the master clock data. The timestamp data is corrected based on the pre-calibrated display link compensation amount and audio link compensation amount to obtain the actual visual stimulus presentation time and the actual sound stimulus playback time. Based on the target frequency, the actual visual stimulus presentation time, and the actual sound stimulus playback time, visual phase data and sound phase data are determined. The timestamp data, visual phase data, sound phase data, and user feedback data are subjected to multimodal event alignment processing to obtain cognitive training results.

3. The method according to claim 2, characterized in that, The visual stimulus is output through a display, and the sound stimulus is output through an audio player. The display link compensation amount and the audio link compensation amount are obtained in the following ways: Obtain offline calibration parameters and scheduling delay data. The offline calibration parameters include the display link delay corresponding to the display and the audio link delay corresponding to the audio player. Based on the display link latency, the audio link latency, and the scheduling latency data, determine the joint latency data; Based on the joint delay data, the display link compensation amount and the audio link compensation amount are determined.

4. The method according to claim 1, characterized in that, The sound stimulus is output through an audio player, the audio player including multiple channels; before outputting the sound stimulus through the audio player, the method further includes: Determine the delay channel among the multiple channels and the delay compensation amount corresponding to the delay channel; Based on the preset audio sampling rate and the delay compensation amount, the number of offset sampling points in the buffer is determined, wherein the buffer stores the sound stimulus; Based on the number of offset sampling points, adjust the playback position of the delayed channel in the buffer.

5. The method according to claim 4, characterized in that, After outputting the sound stimulus, the method further includes: When the user performs the cognitive training task, obtain the playback position of the buffer corresponding to the multi-channel audio. When the difference between the playback positions of any two channels is greater than a preset threshold and less than or equal to a reset threshold, the phase accumulator or the pointer of the buffer is adjusted, and the buffer determines the playback position in the form of a pointer. When the difference between the playback positions of any two channels is less than or equal to a preset threshold, the sound stimulus continues to be output based on the current playback position; When the difference between the playback positions of any two channels is greater than the reset threshold, the buffer is reset and an exception log is written.

6. The method according to claim 3, characterized in that, The offline calibration parameters also include calibration compensation amounts between visual and auditory stimuli. Before synchronously outputting the auditory and visual stimuli, the method further includes: The display and the audio player are calibrated by performing multiple measurements during the calibration process to obtain the visual peak time, audio peak time and calibration measurement time for each measurement. The calibration delay is determined based on the visual peak time and the audio peak time obtained from each measurement; The absolute value of the calibration delay is determined. When the absolute value is less than or equal to the calibration threshold, the calibration delay is used as the calibration compensation amount. When the absolute value is greater than the calibration threshold, the output time of the sound stimulus or the visual stimulus is adjusted based on the calibration delay. Based on the device information of the display, the device information of the audio player, the calibration measurement time, and the calibration compensation amount, the calibration result is obtained and stored.

7. The method according to claim 6, characterized in that, After simultaneously outputting the auditory stimulus and the visual stimulus, the method further includes: The sound stimulus and the visual stimulus are downsampled according to a preset time interval and a preset length, and the audio envelope of the sound stimulus and the visual sequence of the visual stimulus at the corresponding time are obtained. Based on the audio envelope and the visual sequence, a cross-correlation calculation is performed to determine the delay index, wherein the delay index is the maximum value obtained from the cross-correlation calculation; Using the calibration compensation amount as the initial value, the calibration compensation amount is updated based on the delay index and the downsampling sampling rate to obtain the updated calibration compensation amount; The playback position of the sound stimulus in the buffer is adjusted based on the updated calibration compensation amount, wherein the buffer stores the sound stimulus and the playback position is determined by a pointer.

8. The method according to claim 1, characterized in that, The task configuration parameters include a modulation mode and corresponding modulation parameters, as well as a sound mode and corresponding sound parameters; the step of generating sound and visual stimuli corresponding to the cognitive training task based on the task configuration parameters includes: Visual stimuli corresponding to the cognitive training task are generated based on the modulation mode and the modulation parameters, wherein the modulation mode includes at least one of brightness modulation, grayscale modulation, and transparent masking modulation; Based on the sound pattern and the sound parameters, sound stimuli corresponding to the cognitive training task are generated, wherein the sound pattern includes any one of click sounds, modulation sounds, and gated noise; and / or The cognitive training tasks include multiple tasks, each corresponding to a different cognitive domain; adjusting the next cognitive training task based on the cognitive training results includes: Based on the cognitive training results, determine the user's impairment rating vector for multiple cognitive domains, and obtain the training weight vector for each cognitive training task. The similarity result is obtained by weighting the damaged score vector, the training weight vector, and the pre-configured priority weight vector. The cognitive training task is initially screened based on the damaged score vector, the training weight vector, and the priority weight vector to obtain the cognitive training task after initial screening. Based on the similarity results, the cognitive training tasks after the initial screening are sorted and filtered to adjust the next cognitive training task.

9. The method according to claim 1, characterized in that, Adjusting the next cognitive training task based on the cognitive training results includes: Based on the cognitive training results, accuracy data, reaction time data, completion rate data, and penalty item data are determined. The comprehensive performance score data is obtained by weighting the accuracy data, reaction time data, completion rate data, penalty item data, first weight, second weight, third weight, and fourth weight. Based on the comprehensive performance score data, target score data, safety upper limit data, and safety lower limit data, the next cognitive training task will be adjusted.

10. The method according to claim 9, characterized in that, The adjustment of the next cognitive training task based on the comprehensive performance score data, target score data, safety upper limit data, and safety lower limit data includes: When the overall performance score data is greater than the target score data, the training difficulty of the next cognitive training task is increased or the training time is extended, wherein the increased training difficulty or extended training time meets the safety upper limit data. When the overall performance score is less than or equal to the target score, the training difficulty of the next cognitive training task is reduced, the rest time is increased, and the parameters corresponding to visual or auditory stimuli are reduced, wherein the reduced training difficulty, increased rest time, and reduced parameters meet the safety lower limit data; and / or The method further includes: Obtain abnormal state data of the user performing the cognitive training task; Based on the abnormal state data, adjust the next cognitive training task.

11. The method according to claim 1, characterized in that, The cognitive training results include quality assessment results, and adjusting the next cognitive training task based on the cognitive training results includes: The cognitive training results are evaluated to obtain the quality evaluation results; When the quality assessment result is greater than or equal to the first quality assessment threshold, the cognitive assessment result is determined to be valid data. When the quality assessment result is greater than or equal to the second quality assessment threshold and less than or equal to the third quality assessment threshold, the cognitive assessment result is determined to be partially valid data and marked as low-quality data. When the quality assessment result is less than the second quality assessment threshold, the cognitive assessment result is determined to be invalid data and a prompt message is generated. Adjust the next cognitive training task based on the effective data and / or the partial effective data.

12. A multimodal human cognitive training device, characterized in that, The device includes: The first acquisition module is used to acquire task configuration parameters for a cognitive training task, wherein the task configuration parameters include at least the target frequency; The generation module is used to generate sound stimuli and visual stimuli corresponding to the cognitive training task based on the task configuration parameters, and to synchronously output the sound stimuli and visual stimuli according to the target frequency during the execution of the cognitive training task; The second acquisition module is used to acquire the cognitive training results obtained by the user when performing the cognitive training task. The adjustment module is used to adjust the next cognitive training task based on the cognitive training results.

13. An edge computing device having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method described in any one of claims 1-11.

14. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method described in any one of claims 1-11.

Images