Closed-loop visual stimulation system and method based on multi-modal synchronous sensing signal

Abstract

The embodiment of the invention discloses a closed-loop visual stimulation system and method based on a multi-modal synchronous sensing signal, and the method comprises the steps: collecting at least two types of multi-modal data, carrying out the processing and analysis of the multi-modal data, determining a stimulation position and a stimulation parameter, and outputting a stimulation signal according to the stimulation position and the stimulation parameter, the modal combination of the multi-modal data is determined according to the stimulation target, and the stimulation parameters comprise the stimulation amplitude and the stimulation frequency, so that the stimulation position and the stimulation parameters can be adjusted according to the multi-modal data, and the stimulation accuracy is improved.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and more specifically, to a closed-loop visual stimulation system and method based on multimodal synchronous sensing signals. Background Technology

[0002] External non-invasive electro / magnetic stimulation signals refer to specific waveform signals applied to target tissues through electrodes or coils. These signals regulate nerve excitability through electro / magnetic fields and are widely used in fields such as visual therapy and rehabilitation, functional enhancement, and human-computer interaction.

[0003] Traditional electromagnetic stimulation methods mainly employ open-loop control, stimulating the target area according to set stimulation parameters. However, these methods suffer from insufficient spatial positioning accuracy and a lack of individualized adaptive adjustment of stimulation parameters, resulting in poor stimulation effects. Summary of the Invention

[0004] In view of this, embodiments of the present invention provide a closed-loop visual stimulation system and method based on multimodal synchronous sensing signals to adjust the stimulation position and stimulation parameters of the stimulation signals, thereby improving the accuracy of stimulation.

[0005] In a first aspect, embodiments of the present invention provide a closed-loop visual stimulation system based on multimodal synchronous sensing signals, the system comprising: A data acquisition module is used to acquire multimodal data, which includes at least two types of data, and the modality combination of the multimodal data is determined according to the stimulus target. The processing module is used to process and analyze the multimodal data to determine the stimulus location and stimulus parameters, including stimulus amplitude and stimulus frequency. A stimulation device for outputting a stimulation signal based on the stimulation location and stimulation parameters.

[0006] Optionally, the multimodal data includes at least two of the following: functional near-infrared spectroscopy data, electroencephalogram data, eye-tracking data, and / or motion state data.

[0007] Optionally, the processing module includes a timing control submodule, which is used for: In response to the stimulation duration reaching a first predetermined duration, the stimulation device is controlled to stop outputting stimulation signals, and the data acquisition module is controlled to start acquiring multimodal data. In response to the data acquisition duration reaching a second predetermined duration, the data acquisition module is controlled to stop acquiring data, and the processing module is controlled to process and analyze the multimodal data to determine the stimulation location and stimulation parameters, so that the stimulation device outputs a stimulation signal.

[0008] Optionally, the processing module is further configured to: Feature extraction is performed on the multimodal data to obtain a quantized feature vector; Perform feature fusion on the quantized feature vectors to generate a fused feature vector; The fused feature vector is input into the decision model to obtain the stimulus location and stimulus parameters; The stimulation location and stimulation parameters are sent to the stimulation device.

[0009] Optionally, the processing module is deployed with a pre-trained deep learning model, the deep learning model comprising: At least one neural network sub-model is used to map input data into a quantized feature vector of a preset dimension, wherein each neural network sub-model corresponds to data of a different data type; The fusion network is used to fuse the various quantized feature vectors to generate a fused feature vector. A decision model is used to make decisions based on the fused feature vector, generating stimulus locations and stimulus parameters.

[0010] Optionally, the processing module is further configured to control the corresponding neural network sub-module to enter the activation state based on the modality combination of the multimodal data; The neural network sub-model is further used to map the input data into a quantized feature vector of a preset dimension in response to entering the activation state.

[0011] Optionally, the decision model is trained based on the following steps: Obtain labeled sample data, which includes historical multimodal data and corresponding classification labels and regression labels. The classification labels are the optimal stimulus positions corresponding to the historical multimodal data, and the regression labels are the optimal stimulus parameters corresponding to the historical multimodal data. The historical multimodal data is processed to obtain a historical fusion vector; Training for a classification task is performed based on the historical fusion vector and the classification label; Regression task training is performed based on the historical fusion vector and the regression label.

[0012] Optionally, the processing module is further configured to: In response to an abnormality in the multimodal data representation state, the stimulation device is controlled to suspend the output of stimulation signals and broadcast an alarm message.

[0013] Secondly, embodiments of the present invention also provide a closed-loop visual stimulation method based on multimodal synchronous sensing signals, the method comprising: Acquire multimodal data, which includes at least two types of data, and the modality combination of the multimodal data is determined according to the stimulus target; The multimodal data is processed and analyzed to determine the stimulus location and stimulus parameters, including stimulus amplitude and stimulus frequency. The stimulation location and stimulation parameters are sent to the stimulation device so that the stimulation device outputs a stimulation signal according to the stimulation location and stimulation parameters.

[0014] Thirdly, embodiments of the present invention also provide an electronic device, including a memory and a processor, wherein the memory is used to store one or more computer program instructions, wherein the one or more computer program instructions are executed by the processor to implement the method as described in the second aspect of the present invention.

[0015] This invention, through the acquisition of multimodal data including at least two types, processes and analyzes the multimodal data to determine the stimulus location and stimulus parameters, and outputs a stimulus signal based on the stimulus location and stimulus parameters. The modal combination of the multimodal data is determined according to the stimulus target, and the stimulus parameters include stimulus amplitude and stimulus frequency. Therefore, this invention can adjust the stimulus location and stimulus parameters based on the multimodal data. By establishing a closed-loop visual stimulation system architecture that integrates synchronously acquired multimodal data, and utilizing the synchronous acquisition of multimodal signals from multiple sites, the stimulation effect can be evaluated in multiple dimensions, improving the accuracy of the stimulation. Attached Figure Description

[0016] The above and other objects, features and advantages of the present invention will become clearer from the following description of embodiments of the invention with reference to the accompanying drawings, in which: Figure 1 This is a schematic diagram of a closed-loop visual stimulation system according to an embodiment of the present invention; Figure 2 This is a flowchart of the stimulation signal adjustment method according to an embodiment of the present invention; Figure 3 This is a schematic diagram illustrating the working process of the closed-loop visual stimulation system according to an embodiment of the present invention; Figure 4 This is a flowchart of a stimulation signal adjustment method according to an embodiment of the present invention; Figure 5 This is a schematic diagram of a deep learning model according to an embodiment of the present invention; Figure 6 This is a flowchart of the model training method according to an embodiment of the present invention; Figure 7 This is a flowchart of the closed-loop visual stimulation method according to an embodiment of the present invention; Figure 8 This is a schematic diagram of an electronic device according to an embodiment of the present invention. Detailed Implementation

[0017] The present application is described below based on embodiments, but it is not limited to these embodiments. In the detailed description of the present application below, certain specific details are described in detail. Those skilled in the art can fully understand the present application without these details. To avoid obscuring the substance of the present application, well-known methods, processes, flows, elements, and circuits are not described in detail.

[0018] Furthermore, those skilled in the art should understand that the accompanying drawings provided herein are for illustrative purposes only and are not necessarily drawn to scale.

[0019] Unless the context explicitly requires it, words such as "including" or "contains" throughout the application should be interpreted as including rather than exclusive or exhaustive; that is, meaning "including but not limited to".

[0020] In the description of this application, it should be understood that the terms "first," "second," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance. Furthermore, in the description of this application, unless otherwise stated, "a plurality of" means two or more.

[0021] Figure 1 This is a schematic diagram of a closed-loop visual stimulation system according to an embodiment of the present invention. Figure 1 As shown, the closed-loop visual stimulation system 1 of this embodiment includes a data acquisition module 11, a processing module 12, and a stimulation device 13.

[0022] The data acquisition module 11 is used to acquire multimodal data, which includes at least two types of data. For example, the data types of multimodal data may include at least two of the following: fNIRS (functional near-infrared spectroscopy) data, EEG (electroencephalogram) data, eye-tracking data, and motion state data (such as acceleration and angular velocity data of the target area measured by an inertial measurement unit, or motion trajectory of the target area acquired by a motion capture system). Depending on the different data types included in the multimodal data, the data acquisition module 11 may include various sensors corresponding to the data types to be acquired.

[0023] The modal combination of multimodal data is determined according to the stimulus target. Optionally, the data acquisition module 11 is further used to determine the region to be stimulated according to the stimulus target, and select the corresponding modal combination according to the region to be stimulated. For example, if the stimulus target is visual-motor coordination training, the region to be stimulated related to the stimulus target includes the parietal lobe eye movement area, the posterior parietal cortex, the premotor cortex, and the primary motor cortex. Since the stimulus target involves the integration of visual information and motor planning, a modal combination of fNIRS data, EEG data, and eye movement data can be selected. fNIRS data can provide spatial localization of parietal lobe and premotor cortex activation, EEG, with its high temporal resolution, can capture the dynamic process of neural information transmission within the sensorimotor network, and eye movement data is the final behavioral output of coordination ability. By combining the three, the state information of the "perception-decision-execution" chain of the stimulus object after receiving the stimulus can be comprehensively evaluated. Based on this, the stimulus signal can be adjusted, which can effectively improve the accuracy of the stimulus.

[0024] Different modalities of data each possess advantages in temporal and spatial resolution. Through targeted modal combinations, multimodal data can achieve information complementarity. For example, fNIRS data has high spatial resolution, enabling precise localization of stimulus regions, but low temporal resolution. In contrast, EEG data has high temporal resolution and low spatial resolution, accurately capturing the response of neural electrical signals after stimulation, while eye-tracking signals provide the final behavioral output. Through multimodal data fusion, a clear three-dimensional profile of when, where, and what kind of behavioral output can be constructed, compensating for the information blind spots of single-modal data.

[0025] Furthermore, multimodal data also serves as a cross-validation tool. For example, relying solely on EEG data for judgment may lead to misjudgments due to noise interference. However, when EEG data shows activity in a certain brain region, introducing fNIRS and eye-tracking data for auxiliary judgment—such as using fNIRS data to determine whether blood oxygen concentration in that region has increased and using eye-tracking signals to determine whether related behaviors have improved—can improve the accuracy of decision-making through cross-validation and reduce the risk of erroneous regulation due to noise in a single signal.

[0026] Meanwhile, the analysis of multimodal data based on different stimulation targets and corresponding modal combinations ensures the scalability of the system. Although this embodiment mainly uses transcranial electro / magnetic stimulation as an example, those skilled in the art will understand that by making adaptive adjustments to the data acquisition module 11 and the processing module 12, this embodiment can also be used in other similar scenarios.

[0027] The processing module 12 is used to process and analyze multimodal data to determine the stimulus location and stimulus parameters, wherein the stimulus parameters include stimulus amplitude and stimulus frequency. Optionally, the stimulus parameters may also include stimulus pulse width, various combinations of different stimulus frequency modulations, and duration of a single stimulus.

[0028] Depending on the target of stimulation, the stimulation device 13 can output stimulation signals through alternating current stimulation, direct current stimulation, and / or transcranial magnetic stimulation to provide non-invasive visual stimulation via transcranial and / or transorbital methods. The processing module 12 and the stimulation device 13 are connected wirelessly. After determining the stimulation location and parameters, the processing module 12 sends them to the stimulation device 13, which then outputs stimulation signals based on the received stimulation location and parameters.

[0029] Specifically, depending on the stimulation signal, the stimulation device 13 outputs a stimulation signal through electrodes or stimulation coils. In one optional implementation, the stimulation position can be roughly adjusted by adjusting the electrode position, and the stimulation coil can be roughly adjusted by using a fixing device. Both can also be finely adjusted using TI (Temporal Interference) technology. Therefore, the stimulation position can include both physical location and TI parameters. Optionally, the stimulation device 13 can include a display screen to show the stimulation position, instructing the operator to manually adjust the position of the electrodes or stimulation coils. The stimulation device 13 can also include a voice broadcast unit for voice navigation of the stimulation position to assist the operator in adjusting the stimulation position. The stimulation device can also include structures such as a robotic arm to adjust the position of the electrodes or stimulation coils based on the stimulation position, avoiding errors caused by manual operation. After the stimulation position is adjusted, the stimulation device can output a stimulation signal according to the TI parameters and stimulation parameters. In another optional implementation, the position of the electrodes or stimulation coils is fixed after installation, and the stimulation position can only be adjusted through the TI parameters; that is, the stimulation position includes the TI parameters.

[0030] Figure 2 This is a flowchart of the stimulation signal adjustment method according to an embodiment of the present invention. Figure 2 As shown, after the processing module 12 determines the stimulation location and stimulation parameters, it can send them to the stimulation device 13 based on the following steps.

[0031] Step S210: Determine whether the stimulus position needs to be adjusted. If adjustment is needed, proceed to step S220; otherwise, proceed to step S230.

[0032] Step S220: Adjust positioning settings.

[0033] Step S230: Determine whether the stimulation parameters need to be adjusted. If they need to be adjusted, proceed to step S240; otherwise, proceed to step S280.

[0034] Step S240: Determine whether the stimulus amplitude needs to be adjusted. If adjustment is needed, proceed to step S250; otherwise, proceed to step S260.

[0035] Step S250: Adjust the amplitude setting.

[0036] Step S260: Determine whether the stimulation frequency needs to be adjusted. If it needs to be adjusted, proceed to step S270; otherwise, proceed to step S280.

[0037] Step S270: Adjust the frequency setting.

[0038] In step S280, the updated settings are sent to the stimulation device 13.

[0039] It is understood that the above steps are explained using stimulation parameters including stimulation amplitude and stimulation frequency as examples. Depending on the stimulation parameters, other stimulation parameters can be judged after steps S260 and S270. Specific details will not be elaborated here.

[0040] Since the stimulation signal after each round of stimulation may only require adjustment of a small portion of its parameters, sending all the previously analyzed stimulation positions and parameters to the stimulation device 13 would increase processing resource overhead and slow down signal adjustment efficiency. This embodiment filters out the updated portions of the stimulation positions and parameters, sending only those updated portions to the stimulation device. This allows the stimulation device to selectively modify certain parameters, reducing the consumption of processing resources and increasing the adjustment speed of the stimulation signal.

[0041] Optionally, the stimulation device 13 also includes an alarm submodule. The processing module 12 monitors the multimodal data it receives, and in response to any abnormality in the multimodal data representation, controls the stimulation device 13 to pause the output of stimulation signals and broadcast an alarm message. Specifically, the processing module 12 can be pre-configured with thresholds corresponding to various data types. If a certain type of data exceeds the corresponding threshold, it is determined that the state of the stimulated object is abnormal.

[0042] This embodiment collects multimodal data including at least two types, processes and analyzes the multimodal data to determine the stimulus location and stimulus parameters, and outputs a stimulus signal based on the stimulus location and stimulus parameters. The modal combination of the multimodal data is determined according to the stimulus target, and the stimulus parameters include stimulus amplitude and stimulus frequency. Therefore, this embodiment can adjust the stimulus location and stimulus parameters based on the multimodal data to improve the accuracy of the stimulus.

[0043] Figure 3This is a schematic diagram illustrating the working process of the closed-loop visual stimulation system according to an embodiment of the present invention. Figure 3 As shown, in one optional implementation, the processing module 12 includes a timing control submodule. In response to the stimulation duration reaching a first predetermined duration, the timing control submodule controls the stimulation device 13 to stop outputting stimulation signals and controls the data acquisition module 11 to start acquiring multimodal data. In response to the data acquisition duration reaching a second predetermined duration, the timing control submodule controls the data acquisition module 11 to stop acquiring data and controls the processing module 12 to process and analyze the multimodal data to determine the stimulation location and stimulation parameters, so that the stimulation device 13 outputs stimulation signals. Then, after the stimulation duration reaches the first predetermined duration again, stimulation stops and data acquisition begins. The closed-loop visual stimulation system achieves continuous iterative updates of the stimulation signals by repeatedly executing the above steps, ensuring the degree of adaptation between the stimulation signals and the stimulation object.

[0044] This embodiment, by setting a timing control submodule, automatically stops stimulating the object for a period of time and collects data. Based on the collected data, it adjusts the stimulation position and stimulation parameters, adjusts the stimulation signal after a certain period of data collection, and then stimulates again based on the adjusted stimulation signal. Thus, this embodiment achieves closed-loop control of "outputting stimulation signal based on stimulation position and stimulation parameters - collecting data - adjusting stimulation position and stimulation parameters - outputting stimulation signal based on the adjusted stimulation position and stimulation parameters", thereby improving the accuracy of stimulation.

[0045] Figure 4 This is a flowchart of a stimulation signal adjustment method according to an embodiment of the present invention. Figure 4 As shown, in one alternative implementation, the processing module 12 determines the stimulus location and stimulus parameters by performing the following steps.

[0046] Step S410: Extract features from the multimodal data to obtain quantized feature vectors.

[0047] Specifically, before feature extraction, the processing module 12 can preprocess the multimodal data, which may include three steps: denoising, filtering, and time alignment. Taking fNIRS data and EEG data as examples of multimodal data, the processing module 12 uses wavelet transform and other methods to remove motion artifacts from the fNIRS data and power frequency interference from the EEG data. It then filters the fNIRS data and EEG data using bandpass filters of 0.01-0.3Hz and 1-40Hz, respectively. Next, using the stimulus start time as a reference, it uses an interpolation algorithm to process the sampling rate differences between the various modal data to ensure that the different modal data are accurately aligned on the time axis for subsequent synchronous analysis.

[0048] After data preprocessing, processing module 12 performs feature extraction on each modality of data. For example, for fNIRS data, it extracts temporal features (such as mean, slope, variance, etc.) of oxyhemoglobin and deoxyhemoglobin concentrations in the target region, frequency domain features (such as low-frequency oscillation power in the 0.01-0.1Hz range), and brain region functional connectivity strength based on Pearson correlation coefficient. For EEG data, it extracts the amplitude, latency, power spectral density of each frequency band, and network connectivity based on phase lock value of the P300 (positive wave appearing approximately 300ms after stimulus) in event-related potentials. For eye-tracking signals, it extracts fixation coordinates, saccade velocity, pupil diameter change rate, task accuracy, reaction time, etc. Then, each feature is converted into a quantized feature vector to provide input for subsequent feature fusion.

[0049] Step S420: Perform feature fusion on the quantized feature vector to generate a fused feature vector. Feature fusion can be performed using canonical correlation analysis or a deep autoencoder, or other methods; this embodiment does not impose any limitations on this approach.

[0050] Step S430: Input the fused feature vector into the decision model to obtain the stimulus location and stimulus parameters.

[0051] Step S440: The stimulation location and stimulation parameters are sent to the stimulation device.

[0052] In one alternative implementation, the processing module 12 is equipped with a pre-trained deep learning model, which obtains stimulus locations and stimulus parameters by inputting multimodal data into the deep learning model.

[0053] Figure 5 This is a schematic diagram of a deep learning model according to an embodiment of the present invention. Figure 5 As shown, the deep learning model 5 in this embodiment adopts a modular hybrid neural network architecture, which includes at least one neural network sub-model 51, a fusion network 52, and a decision model 53. Figure 5 Taking two neural network sub-models 51 as an example, it should be understood that their specific number can be determined according to actual needs. Each neural network sub-model 51 corresponds to data of different data types, used to map the input data into a quantized feature vector of a preset dimension. Specifically, each neural network sub-model 51 can correspond to one data type, and the number of neural network sub-models 51 can be determined according to the number of data modalities. Alternatively, each neural network sub-model 51 can correspond to data of one dimension. For example, a deep learning model includes two neural network sub-models 51, corresponding to spatial dimension data (i.e., data with high spatial resolution) and temporal dimension data (i.e., data with high temporal resolution), respectively.

[0054] In one example, the two neural network sub-models 51 are a convolutional neural network sub-model and a recurrent neural network sub-model, respectively. The convolutional neural network sub-model processes the spatial topology of the fNIRS data, taking an fNIRS channel*time point matrix as input and outputting a quantized feature vector in the spatial dimension. The recurrent neural network sub-model uses a long short-term memory network to process EEG and eye-tracking data, taking time-series data (such as the theta band time series of EEG) as input and outputting a quantized feature vector in the time dimension.

[0055] Optionally, each neural network sub-model 51 is normally in a dormant state. When the processing module 12 receives multimodal data sent by the data acquisition module 11, it controls the corresponding neural network sub-model 51 to enter the active state according to the modal combination of the multimodal data, while the other neural network sub-models 51 remain dormant. In response to entering the active state, the neural network sub-model 51 maps the input data into a quantized feature vector of a preset dimension. Therefore, this embodiment can call each neural network sub-model as needed, thereby effectively reducing computational resource consumption and improving the real-time performance of the system.

[0056] After processing the data, each neural network sub-model sends its output to the fusion network 52. The fusion network 52 fuses the quantized feature vectors to generate a fused feature vector, which is then sent to the decision model 53. The decision model 53 makes decisions based on the fused feature vector, generating stimulus locations and stimulus parameters.

[0057] Decision model 53 employs a multi-task learning framework, capable of performing classification and regression tasks based on the input fused feature vector to generate stimulus locations and stimulus parameters. The classification task determines the optimal stimulus location corresponding to the input parameters, while the regression task predicts the optimal stimulus parameters.

[0058] Figure 6 This is a flowchart of the model training method according to an embodiment of the present invention. Figure 6 As shown, the decision model in this embodiment is trained based on the following steps: Step S610: Obtain labeled sample data. The labeled sample data includes historical multimodal data and corresponding classification and regression labels. The classification and regression labels are determined by relevant technical personnel based on the real-time stimulus effect evaluation corresponding to the historical multimodal data. The classification label represents the optimal stimulus location corresponding to the historical multimodal data, and the regression label represents the optimal stimulus parameter corresponding to the historical multimodal data.

[0059] Step S620: Perform data processing on the historical multimodal data to obtain the historical fusion vector.

[0060] Step S630: Training for the classification task based on historical fusion vectors and classification labels.

[0061] Step S640: Train the regression task based on the historical fusion vector and regression labels.

[0062] The decision model trained through the above steps can perform regression and classification tasks based on the input multimodal data, thereby obtaining the optimal stimulus location and optimal stimulus parameters corresponding to the input data.

[0063] This embodiment deploys a deep learning model with a modular hybrid neural network architecture in the processing module. By selecting and fusing features from the corresponding neural network sub-models, it achieves adaptation to multimodal data with different modal combinations. Furthermore, by processing the input data through a multi-task learning model based on supervised learning, it obtains the optimal stimulus location and optimal stimulus parameters, thereby improving the accuracy of the stimulus.

[0064] Figure 7 This is a flowchart of a closed-loop visual stimulation method according to an embodiment of the present invention. Figure 7 As shown, the closed-loop visual stimulation method in this embodiment specifically includes the following steps: Step S710: Acquire multimodal data. The multimodal data includes at least two types of data, and the modal combination of the multimodal data is determined based on the stimulus target.

[0065] Step S720: Process and analyze the multimodal data to determine the stimulus location and stimulus parameters, including stimulus amplitude and stimulus frequency.

[0066] Step S730: Send the stimulation location and stimulation parameters to the stimulation device so that the stimulation device outputs a stimulation signal according to the stimulation location and stimulation parameters.

[0067] This embodiment collects multimodal data including at least two types, processes and analyzes the multimodal data to determine the stimulus location and stimulus parameters, and outputs a stimulus signal based on the stimulus location and stimulus parameters. The modal combination of the multimodal data is determined according to the stimulus target, and the stimulus parameters include stimulus amplitude and stimulus frequency. Therefore, this embodiment can adjust the stimulus location and stimulus parameters based on the multimodal data to improve the accuracy of the stimulus.

[0068] Figure 8 This is a schematic diagram of an electronic device according to an embodiment of the present invention. (For example...) Figure 8 As shown, Figure 8The illustrated electronic device is a general-purpose data processing device, comprising a general-purpose computer hardware architecture, including at least a processor 81 and a memory 82. The processor 81 and memory 82 are connected via a bus 83. The memory 82 is adapted to store instructions or programs executable by the processor 81. The processor 81 can be a standalone microprocessor or a collection of one or more microprocessors. Thus, the processor 81 executes the instructions stored in the memory 82, thereby performing the method flow of the embodiments of the present invention as described above to process data and control other devices. The bus 83 connects the aforementioned components together, and also connects these components to a display controller 84, a display device, and an input / output (I / O) device 85. The input / output (I / O) device 85 can be a mouse, keyboard, modem, network interface, touch input device, motion-sensing input device, printer, and other devices known in the art. Typically, the input / output device 85 is connected to the system via an input / output (I / O) controller 86.

[0069] Those skilled in the art will understand that embodiments of this application can be provided as methods, apparatus (devices), or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-readable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0070] This application is described with reference to flowchart illustrations of methods, apparatus (devices), and computer program products according to embodiments of this application. It should be understood that each step in the flowchart can be implemented by computer program instructions.

[0071] These computer program instructions may be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including an instruction means, the implementation process of which is described in the instruction means. Figure 1 The function specified in one or more processes.

[0072] These computer program instructions may also be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing device, produce instructions for implementing processes. Figure 1 A device for a function specified in one or more processes.

[0073] Another embodiment of the present invention relates to a non-volatile storage medium for storing a computer-readable program for use by a computer to execute some or all of the above-described method embodiments.

[0074] That is, those skilled in the art will understand that all or part of the steps in the methods of the above embodiments can be implemented by a program specifying the relevant hardware. This program is stored in a storage medium and includes several instructions to cause a device (which may be a microcontroller, chip, etc.) or processor to execute all or part of the steps of the methods described in the embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, a portable hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0075] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A closed-loop visual stimulation system based on multimodal synchronous sensing signals, characterized in that, The system includes: A data acquisition module is used to acquire multimodal data, which includes at least two types of data, and the modality combination of the multimodal data is determined according to the stimulus target. The processing module is used to process and analyze the multimodal data to determine the stimulus location and stimulus parameters, including stimulus amplitude and stimulus frequency. A stimulation device for outputting a stimulation signal based on the stimulation location and stimulation parameters.

2. The system according to claim 1, characterized in that, The multimodal data includes at least two of the following: functional near-infrared spectroscopy data, electroencephalogram (EEG) data, eye-tracking data, and motion state data.

3. The system according to claim 1, characterized in that, The processing module includes a timing control submodule, which is used for: In response to the stimulation duration reaching a first predetermined duration, the stimulation device is controlled to stop outputting stimulation signals, and the data acquisition module is controlled to start acquiring multimodal data. In response to the data acquisition duration reaching a second predetermined duration, the data acquisition module is controlled to stop acquiring data, and the processing module is controlled to process and analyze the multimodal data to determine the stimulation location and stimulation parameters, so that the stimulation device outputs a stimulation signal.

4. The system according to claim 1, characterized in that, The processing module is further used for: Feature extraction is performed on the multimodal data to obtain a quantized feature vector; Perform feature fusion on the quantized feature vectors to generate a fused feature vector; The fused feature vector is input into the decision model to obtain the stimulus location and stimulus parameters; The stimulation location and stimulation parameters are sent to the stimulation device.

5. The system according to claim 1, characterized in that, The processing module is equipped with a pre-trained deep learning model, which includes: At least one neural network sub-model is used to map input data into a quantized feature vector of a preset dimension, wherein each neural network sub-model corresponds to data of a different data type; The fusion network is used to fuse the various quantized feature vectors to generate a fused feature vector. A decision model is used to make decisions based on the fused feature vector, generating stimulus locations and stimulus parameters.

6. The system according to claim 5, characterized in that, The processing module is further configured to control the corresponding neural network sub-module to enter the activation state based on the modality combination of the multimodal data; The neural network sub-model is further used to map the input data into a quantized feature vector of a preset dimension in response to entering the activation state.

7. The system according to claim 5, characterized in that, The decision model is trained based on the following steps: Obtain labeled sample data, which includes historical multimodal data and corresponding classification labels and regression labels. The classification labels are the optimal stimulus positions corresponding to the historical multimodal data, and the regression labels are the optimal stimulus parameters corresponding to the historical multimodal data. The historical multimodal data is processed to obtain a historical fusion vector; Training for a classification task is performed based on the historical fusion vector and the classification label; Regression task training is performed based on the historical fusion vector and the regression label.

8. The system according to claim 1, characterized in that, The processing module is further used for: In response to an abnormality in the multimodal data representation state, the stimulation device is controlled to suspend the output of stimulation signals and broadcast an alarm message.

9. A closed-loop visual stimulation method based on multimodal synchronous sensing signals, characterized in that, The method includes: Acquire multimodal data, which includes at least two types of data, and the modality combination of the multimodal data is determined according to the stimulus target; The multimodal data is processed and analyzed to determine the stimulus location and stimulus parameters, including stimulus amplitude and stimulus frequency. The stimulation location and stimulation parameters are sent to the stimulation device so that the stimulation device outputs a stimulation signal according to the stimulation location and stimulation parameters.

10. An electronic device comprising a memory and a processor, characterized in that, The memory is used to store one or more computer program instructions, wherein the one or more computer program instructions are executed by the processor to implement the method as described in claim 9.

Images