Test method for multi-modal brain-computer interface device and device and computer readable storage medium thereof

Abstract

The application relates to the technical field of brain-computer interfaces, for example, to a test method for a multi-modal brain-computer interface device, a device thereof and a computer readable storage medium. The test method comprises the following steps: acquiring brain signals of at least two modes output by a to-be-tested multi-modal brain-computer interface device and unifying time stamps; pre-processing the multi-modal brain signals with time stamps to obtain synchronized and aligned multi-modal signal data; performing feature extraction and fusion on the synchronized and aligned multi-modal signal data by using a deep learning-based fusion model to generate a comprehensive feature vector; generating a test case set according to the comprehensive feature vector and parameter information of the to-be-tested multi-modal brain-computer interface device; interacting with the to-be-tested multi-modal brain-computer interface device according to the test case set, executing a test and recording test response data; and evaluating the performance of the brain-computer interface device based on the test response data and the comprehensive feature vector, thereby improving the accuracy of test results and test efficiency.

Description

Technical Field

[0001] This application relates to the field of brain-computer interface technology, such as a testing method and apparatus for a multimodal brain-computer interface device, and a computer-readable storage medium. Background Technology

[0002] Brain-computer interface (BCI) technology is a technology that enables direct information interaction between the brain and external devices without relying on peripheral nerves and muscle tissue. Its basic principle is to collect physiological signals generated by brain activity, such as electroencephalogram (EEG) signals and near-infrared signals, through sensors. These signals are then processed and converted into control commands that can be recognized by external devices, thereby enabling control of the device. To improve the stability of BCI devices, a single-modal testing system is used. However, single-modal testing systems can only test a single type of brain signal and cannot simulate the multimodal signal interaction scenarios encountered in the actual operation of multimodal BCI devices. This results in test results that do not fully reflect the true performance of the device.

[0003] In related technologies, in order to solve the testing problem for multimodal signals, a separate multimodal testing scheme is adopted to test brain signals of different modalities independently, and then the test results are manually integrated.

[0004] The implementation of this plan has at least the following problems: The tests of different modes are conducted independently, which makes it impossible to achieve synchronous acquisition of multimodal signals and test the synergistic effect of multimodal signals in the time dimension, resulting in low accuracy of test results. Furthermore, after the test is completed, the test results of different modes need to be manually integrated, which is time-consuming and labor-intensive, and is prone to errors due to human operation, resulting in low test efficiency.

[0005] It should be noted that the information disclosed in the background section above is only used to enhance the understanding of the background of this application, and therefore may include information that does not constitute prior art known to those skilled in the art. Summary of the Invention

[0006] To provide a basic understanding of some aspects of the disclosed embodiments, a brief summary is given below. This summary is not intended as a general commentary, nor is it intended to identify key / important components or describe the scope of protection of these embodiments, but rather as a prelude to the detailed description that follows.

[0007] This disclosure provides a testing method and apparatus for multimodal brain-computer interface devices, as well as a computer-readable storage medium, to solve the technical problems of low accuracy and low efficiency in testing results.

[0008] In some embodiments, a testing method for a multimodal brain-computer interface device is provided, comprising: acquiring brain signals of at least two modalities output by the multimodal brain-computer interface device under test and unifying their timestamps; preprocessing the timestamped multimodal brain signals to obtain synchronized and aligned multimodal signal data; using a deep learning-based fusion model to extract and fuse features from the synchronized and aligned multimodal signal data to generate a comprehensive feature vector; generating a test case set corresponding to the multimodal brain-computer interface device under test based on the comprehensive feature vector and parameter information of the multimodal brain-computer interface device under test; interacting with the multimodal brain-computer interface device under test according to the test case set, executing tests and recording test response data; and evaluating the performance of the multimodal brain-computer interface device under test based on the test response data and the comprehensive feature vector, and outputting a comprehensive evaluation result.

[0009] Optionally, the deep learning-based fusion model includes a multi-branch neural network with a cross-modal attention mechanism; the deep learning-based fusion model is used to extract and fuse features from synchronized multimodal signal data to generate a comprehensive feature vector, including: extracting high-dimensional features corresponding to brain signal data of at least two modalities through a multi-branch neural network; calculating the correlation weights between the high-dimensional features corresponding to brain signal data of at least two modalities through a cross-modal attention mechanism, and performing weighted fusion to generate a comprehensive feature vector.

[0010] Optionally, the association weights between high-dimensional features corresponding to brain signal data from at least two modalities are calculated using a cross-modal attention mechanism, and weighted fusion is performed to generate a comprehensive feature vector. This includes: mapping the high-dimensional features of each modal brain signal to query features, key features, and value features, respectively; calculating the attention score between the query features of the first modal brain signal and the key features of the second modal brain signal, and normalizing the attention score to obtain the attention weight of the first modal brain signal on the second modal brain signal; using the attention weight of the first modal brain signal on the second modal brain signal to perform a weighted summation of the value features of the second modal brain signal to obtain the cross-modal enhancement features of the first modal brain signal; similarly, calculating the cross-modal enhancement features of other modal brain signals in at least two modalities for each modal brain signal; and fusing the cross-modal enhancement features of all modal brain signals with the original high-dimensional features of each modal brain signal to generate the comprehensive feature vector.

[0011] Optionally, the cross-modal enhancement features of all modal brain signals are fused with the original high-dimensional features of each modality of brain signals to generate the comprehensive feature vector. This includes: concatenating the original high-dimensional features of each modality of brain signals with the corresponding cross-modal enhancement features to form the enhancement feature representation of each modality; inputting the enhancement feature representations of at least two modalities of brain signals into the fusion layer and performing nonlinear transformation and dimensionality reduction through a fully connected network; and outputting a low-dimensional dense vector containing cross-modal correlation information as the comprehensive feature vector.

[0012] Optionally, based on the comprehensive feature vector and the parameter information of the multimodal brain-computer interface device under test, a test case set corresponding to the multimodal brain-computer interface device under test is generated, including: obtaining basic test case templates and test rules from the test knowledge base; combining the real-time signal state reflected by the comprehensive feature vector and the parameter information of the multimodal brain-computer interface device under test, reasoning through the test rules, adjusting the test parameters and test sequences, and generating a test case set corresponding to the multimodal brain-computer interface device under test.

[0013] Optionally, by combining the real-time signal state reflected by the comprehensive feature vector with the parameter information of the multimodal brain-computer interface device under test, reasoning is performed through test rules to adjust test parameters and test sequences, generating a test case set corresponding to the multimodal brain-computer interface device under test. This includes: parsing the comprehensive feature vector to obtain the current signal quality index and brain activity state features; determining the supported brain-computer interface paradigm type and communication protocol based on the parameter information of the multimodal brain-computer interface device under test; matching the basic test case template corresponding to the brain-computer interface paradigm type from the test knowledge base; dynamically adjusting the stimulation parameters in the test case template, including stimulation duration, stimulation interval, and repetition count, through a rule engine based on the signal quality index; optimizing the test sequence through a reinforcement learning model based on brain activity state features, selecting the next test task that maximizes information gain, where information gain is calculated based on the uncertainty measure of the current test result; and combining the adjusted test parameters with the optimized test sequence to generate an executable test case script.

[0014] Optionally, based on the test response data and the comprehensive feature vector, the performance of the multimodal brain-computer interface device under test is evaluated, and a comprehensive evaluation result is output, including: extracting the recognition result and response time of the multimodal brain-computer interface device under test from the test response data; extracting the real-time quality indicators and fusion features of each modality of brain signals from the comprehensive feature vector; calculating the score of the multimodal brain-computer interface device under test on each indicator according to the preset evaluation indicator system, which includes recognition accuracy, response time, information transmission rate, and signal stability index; evaluating the comprehensive robustness of the multimodal brain-computer interface device under test in multimodal collaborative working scenarios based on the fusion features, and obtaining a cross-modal collaborative score; integrating the scores of each indicator and the cross-modal collaborative score to generate a comprehensive evaluation result that includes quantitative scores, performance analysis conclusions, and visualization charts.

[0015] Optionally, the timestamped multimodal brain signals are preprocessed to obtain synchronized and aligned multimodal signal data, including: filtering the acquired brain signals of at least two modalities to remove noise and interference signals; amplifying and standardizing the filtered brain signals to convert signals of different dimensions into a unified data format and magnitude; and interpolating and aligning the brain signals of each modality with different sampling rates based on the timestamps so that the points or segments of each modality signal are aligned on the time axis to obtain synchronized and aligned multimodal signal data.

[0016] In some embodiments, a testing apparatus for a multimodal brain-computer interface device is provided, including a processor and a memory storing program instructions, the processor being configured to execute, when running the program instructions, a testing method for a multimodal brain-computer interface device as described in any of the above embodiments.

[0017] In some embodiments, a computer-readable storage medium is provided storing program instructions that, when executed, cause a computer to perform a testing method for a multimodal brain-computer interface device as described in any of the above embodiments.

[0018] The testing method and apparatus for multimodal brain-computer interface devices, and the computer-readable storage medium provided in this disclosure can achieve the following technical effects: The testing method for multimodal brain-computer interface devices provided in this disclosure achieves synchronous acquisition of multimodal signals through unified timestamps and interpolation alignment. Preprocessing techniques such as filtering and standardization are combined to improve signal quality. A multi-branch neural network with a cross-modal attention mechanism is used to achieve deep fusion of multimodal features, capturing the complementary information between modalities. Simultaneously, the evaluation process considers both basic performance and cross-modal collaborative performance, more realistically simulating the actual working scenario of the multimodal brain-computer interface device, and the test results can comprehensively and accurately reflect the true performance of the product.

[0019] Furthermore, the testing method disclosed herein automates the entire process from signal acquisition, preprocessing, feature fusion, test case generation to test execution and result evaluation without human intervention. At the same time, it optimizes the test sequence through reinforcement learning models, reduces invalid tests, and effectively reduces testing time and manpower costs.

[0020] Furthermore, the testing method disclosed herein can adaptively generate personalized test case sets based on the parameter information and real-time signal status of the brain-computer interface device, dynamically adjust test parameters and sequences, adapt to multimodal brain-computer interface devices with different modal combinations, different interface protocols, and different paradigm types, and meet diverse testing needs.

[0021] The above general description and the description below are exemplary and illustrative only and are not intended to limit this application. Attached Figure Description

[0022] One or more embodiments are illustrated by way of example with reference to the accompanying drawings. These illustrations and drawings do not constitute a limitation on the embodiments. Elements having the same reference numerals in the drawings are shown as similar elements. The drawings are not to be scaled. And wherein: Figure 1 This is a flowchart of a testing method for a multimodal brain-computer interface device provided in an embodiment of this disclosure; Figure 2 This is a flowchart illustrating the process of generating a test case set based on a comprehensive feature vector and parameter information of the multimodal brain-computer interface device under test, as provided in this embodiment of the disclosure. Figure 3 This is a flowchart illustrating the intelligent optimization of test sequences based on a reinforcement learning model, as provided in an embodiment of this disclosure. Figure 4 This is a flowchart illustrating the process of evaluating the performance of a multimodal brain-computer interface device under test based on test response data and comprehensive feature vectors, and outputting a comprehensive evaluation result, according to an embodiment of this disclosure. Figure 5 This is a flowchart of a testing method for a multimodal brain-computer interface device provided in another embodiment of this disclosure; Figure 6 This is a schematic diagram of a testing system for a multimodal brain-computer interface device provided in an embodiment of this disclosure; Figure 7 This is a structural block diagram of a testing device for a multimodal brain-computer interface device provided in an embodiment of this disclosure. Detailed Implementation

[0023] To provide a more detailed understanding of the features and technical content of the embodiments of this disclosure, the implementation of the embodiments of this disclosure will be described in detail below with reference to the accompanying drawings. The accompanying drawings are for illustrative purposes only and are not intended to limit the embodiments of this disclosure. In the following technical description, for ease of explanation, several details are used to provide a full understanding of the disclosed embodiments. However, one or more embodiments may still be implemented without these details. In other cases, well-known structures and devices may be simplified in their depiction to simplify the drawings.

[0024] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this disclosure are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate for the embodiments of this disclosure described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion.

[0025] Unless otherwise stated, the term "multiple" means two or more.

[0026] In this embodiment of the disclosure, the character " / " indicates that the objects before and after it are in an "or" relationship. For example, A / B means: A or B.

[0027] The term "and / or" describes an association between objects, indicating that three relationships can exist. For example, A and / or B means: A or B, or A and B.

[0028] The term "correspondence" can refer to an association or binding relationship. The correspondence between A and B means that there is an association or binding relationship between A and B.

[0029] This invention provides a testing method for multimodal brain-computer interface devices, applicable to performance testing of multimodal brain-computer interface devices with at least two modalities of brain signals. It enables synchronous acquisition of multimodal signals, collaborative testing, automated test case generation, and comprehensive performance evaluation. The specific implementation of this invention will be described in detail below, using the testing scenario of a bimodal brain-computer interface device using EEG (Electroencephalogram) and fNIRS (functional near-infrared spectroscopy).

[0030] In some embodiments, combined with Figure 1 As shown, a testing method for a multimodal brain-computer interface device is provided, comprising: S101, acquire brain signals from at least two modalities output by the multimodal brain-computer interface device under test, and unify the timestamps.

[0031] In this step, the testing system establishes a data connection with the multimodal brain-computer interface device under test, acquiring multiple modalities of brain signals output by the device in real time. For example, it simultaneously acquires electroencephalogram (EEG) signals and functional near-infrared spectroscopy (FIR) signals. To ensure the synchronization of subsequent processing, the system adds a uniform timestamp to each frame of data acquired.

[0032] As a specific implementation, the system can be equipped with a high-precision GPS disciplined clock as a unified clock source, with all data acquisition channels synchronized with this clock source. When the acquisition submodule completes a sampling and generates a data frame, it immediately reads the current time from the clock source and writes it into the data frame header, forming a raw data packet with a timestamp.

[0033] S102, preprocess the timestamped multimodal brain signals to obtain synchronized and aligned multimodal signal data.

[0034] This step performs quality enhancement and spatiotemporal alignment on the acquired raw brain signals, laying the foundation for subsequent fusion. Preprocessing operations include at least filtering, amplification, normalization, and timestamp-based alignment.

[0035] As a specific implementation method: For EEG signals, a bandpass filter is used to retain the effective frequency band of 0.5-45Hz and remove power frequency interference and high frequency noise; For functional near-infrared spectral signals, a low-pass filter is used to remove physiological noise such as breathing and heartbeat. The filtered signal is amplified to adjust the weak signal to a suitable amplitude range for processing. The z-score normalization method is used to convert signals of different dimensions into a standard data format with a mean of 0 and a variance of 1. Based on timestamp information, a cubic spline interpolation method is used to interpolate multimodal signals with different sampling rates onto a unified time base, thereby achieving point alignment of the signals on the time axis.

[0036] S103 employs a deep learning-based fusion model to extract and fuse features from synchronized, aligned multimodal signal data, generating a comprehensive feature vector.

[0037] This step inputs the preprocessed multimodal signals into a deep learning-based fusion model. The model automatically extracts high-dimensional features of each modality and mines the correlation information between modalities, ultimately generating a comprehensive feature vector.

[0038] S104. Based on the comprehensive feature vector and the parameter information of the multimodal brain-computer interface device under test, generate a test case set corresponding to the multimodal brain-computer interface device under test.

[0039] This step utilizes the real-time signal state reflected by the comprehensive feature vector, combined with the inherent parameters of the brain-computer interface device, to generate a test plan. The test case set includes a series of test tasks and their parameter settings, such as stimulus type, stimulus duration, stimulus interval, number of repetitions, and task sequence.

[0040] As a specific implementation method, the system has a built-in test knowledge base that stores basic test case templates and test rules for different brain-computer interface paradigms. For example, templates for the P300 paradigm include the "monster ball" paradigm stimulus sequence, while templates for the motor imagery paradigm include left and right hand motor imagery task sequences. The system analyzes the current signal quality, such as signal-to-noise ratio and artifact level, and the user's brain activity state, such as resting state and task state, based on the comprehensive feature vector analysis. Combining this with the paradigm types supported by the brain-computer interface device, the system matches the corresponding template from the test knowledge base and dynamically adjusts the template parameters.

[0041] S105: Interact with the multimodal brain-computer interface device under test according to the test case set, execute the test, and record the test response data.

[0042] This step involves sending control commands to the device under test according to the generated test case set and receiving the device's response output. During test execution, brain signals output by the brain-computer interface device are continuously collected and correlated with the timestamps of the test commands to form a complete test data record.

[0043] As a specific implementation method, it connects to the device under test (DUT) through a standardized communication interface. For example, for devices supporting the LSL protocol, event markers are sent and data is received in real time via LSL streams; for devices with custom protocols, communication is conducted via serial port or network socket. The test execution module precisely controls the timing of command transmission according to the timing requirements of the test case script, and records the timestamp of each command transmission and the received device response data.

[0044] S106 evaluates the performance of the multimodal brain-computer interface device under test based on test response data and comprehensive feature vectors, and outputs a comprehensive evaluation result.

[0045] The testing method for multimodal brain-computer interface devices provided in this disclosure achieves synchronous acquisition of multimodal brain signals using a unified timestamp, completes signal synchronization alignment through preprocessing, and then mines the intrinsic correlation of multimodal signals through a deep learning fusion model. Based on signal features and product parameters, it adaptively generates test case sets and finally completes automated testing and comprehensive evaluation. This method realizes collaborative testing of multimodal signals, significantly improves the accuracy and comprehensiveness of test results, automates the entire testing process, effectively improves testing efficiency, adapts to multimodal brain-computer interface devices with different parameters, and enhances testing compatibility and flexibility.

[0046] Optionally, the deep learning-based fusion model includes a multi-branch neural network with a cross-modal attention mechanism; the deep learning-based fusion model is used to extract and fuse features from synchronized, aligned multimodal signal data to generate a comprehensive feature vector, including: High-dimensional features corresponding to brain signal data of at least two modalities are extracted using a multi-branch neural network. The association weights between high-dimensional features corresponding to brain signal data from at least two modalities are calculated using a cross-modal attention mechanism, and then weighted and fused to generate a comprehensive feature vector.

[0047] In this embodiment, a multi-branch neural network with a cross-modal attention mechanism is used. The multi-branch structure is used to extract high-dimensional features of different modal brain signals in a targeted manner, ensuring the integrity of the features of each modality signal. At the same time, the cross-modal attention mechanism is used to accurately capture the correlation between features of different modalities and assign corresponding weights to achieve intelligent weighted fusion of features. The generated comprehensive feature vector can more comprehensively and accurately represent the overall features of multimodal brain signals, providing a more reliable basis for subsequent test case generation and performance evaluation, and further improving the accuracy of testing.

[0048] The fusion model used in this embodiment is a multi-branch neural network that includes a cross-modal attention mechanism. The network structure includes: (1) Multi-branch feature extraction sub-network: For brain signals of different modalities, feature extraction branches are set up. Each branch consists of multiple convolutional layers, pooling layers, and recurrent neural network layers, which are used to extract the time domain, frequency domain, and spatial domain features of the signal of that modality.

[0049] As a specific implementation method: For the EEG signal branch, a three-dimensional convolutional neural network is used to extract spatiotemporal features. The input data shape is: number of channels × time point. After passing through multiple Conv3D layers and MaxPooling3D layers, the output dimension is a high-dimensional feature map with the dimensions of feature map height × feature map width × number of feature channels. Then, global average pooling is used to obtain a fixed-length feature vector.

[0050] For the functional near-infrared spectral signal branch, a long short-term memory (LSTM) network is used to extract temporal features. The input data is in the form of time point × number of channels. After passing through two layers of LSTM, the hidden state of the last time step is taken as the feature vector.

[0051] The high-dimensional feature vectors output by both branches are unified to 256 dimensions.

[0052] (2) Cross-modal attention fusion layer: This layer is used to calculate the correlation weights between different modal features and perform weighted fusion. The calculation process will be described in detail in the following examples.

[0053] (3) Fusion output layer: The enhanced features output from the cross-modal attention fusion layer are further fused with the original features. A fully connected network is then used for nonlinear transformation and dimensionality reduction to finally output a comprehensive feature vector. The specific implementation will be described in detail in the following examples.

[0054] In this embodiment, the multi-branch network structure allows for the setting of corresponding extraction networks for different modal signals, fully mining the features of each modality. The cross-modal attention mechanism can dynamically capture the correlation between modalities, so that the fused features contain both the feature information of each modality and the interaction information between modalities, providing a more comprehensive basis for subsequent test case generation and performance evaluation.

[0055] Optionally, the association weights between high-dimensional features corresponding to brain signal data from at least two modalities are calculated using a cross-modal attention mechanism, and then weighted and fused to generate a comprehensive feature vector, including: The high-dimensional features of each modality of brain signal are mapped to query features, key features, and value features, respectively; Based on the query features of the first modality brain signal, the attention score between the first modality brain signal and the key features of the second modality brain signal is calculated, and the attention score is normalized to obtain the attention weight of the first modality brain signal to the second modality brain signal. The cross-modal enhancement features of the first modality brain signal are obtained by weighting and summing the value features of the second modality brain signal using the attention weights of the first modality brain signal to the second modality brain signal; Similarly, calculate the cross-modal enhancement features of each modality of brain signal and the other modalities of brain signals in at least two modalities; The cross-modal enhancement features of all modal brain signals are fused with the original high-dimensional features of each modality brain signal to generate the comprehensive feature vector.

[0056] In this embodiment, by mapping high-dimensional features to query features, key features, and value features, the quantitative calculation of the correlation between features of different modalities is realized. The normalized attention weight can accurately reflect the importance of each modal feature to the target modality. The cross-modal enhancement feature generated based on this weight effectively integrates the effective information of other modalities. Combined with the original high-dimensional features, a comprehensive feature vector is generated, which not only retains the core features of each modality itself, but also integrates the correlation and complementary information between modalities, further improving the representation ability of the comprehensive feature vector and providing a more accurate basis for signal features in subsequent tests.

[0057] Optionally, the cross-modal enhancement features of all modal brain signals are fused with the original high-dimensional features of each modality of brain signal to generate the comprehensive feature vector, including: The original high-dimensional features of each modality brain signal are spliced ​​with the corresponding cross-modal enhancement features to form the enhancement feature representation of each modality; The enhanced feature representations of at least two modalities of brain signals are input into the fusion layer, and nonlinear transformation and dimensionality reduction are performed through a fully connected network; The output low-dimensional dense vector containing cross-modal correlation information is used as the comprehensive feature vector.

[0058] In this embodiment, the splicing operation completely preserves all the information of the original high-dimensional features and cross-modal enhanced features. The resulting modal enhanced feature representations have both single-modal core features and cross-modal correlation features. Then, the nonlinear transformation of the fully connected network is used to achieve deep fusion of multimodal enhanced features, while dimensionality reduction is completed to obtain a low-dimensional dense vector. This makes the comprehensive feature vector contain rich cross-modal correlation information, reducing the complexity of subsequent calculations and improving the computational efficiency of test case generation and performance evaluation.

[0059] Optionally, based on the comprehensive feature vector and the parameter information of the multimodal brain-computer interface device under test, a test case set corresponding to the multimodal brain-computer interface device under test is generated, including: Obtain basic test case templates and test rules from the test knowledge base; By combining the real-time signal state reflected by the comprehensive feature vector with the parameter information of the multimodal brain-computer interface device under test, reasoning is performed through test rules to adjust test parameters and test sequences, thereby generating a test case set corresponding to the multimodal brain-computer interface device under test.

[0060] In this embodiment, based on the basic templates and test rules of the test knowledge base, and combined with real-time signal status and product parameters, reasoning is performed and test parameters and sequences are dynamically adjusted to achieve adaptive generation of test cases. This makes the test cases highly matched with the characteristics and real-time signal status of the device under test, solving the problem of fixed test cases in the prior art. This greatly improves the relevance and flexibility of the test. At the same time, the modular design of the test knowledge base also facilitates the subsequent updating and expansion of templates and rules.

[0061] In some examples, combined Figure 2 As shown, the specific execution steps for generating a test case set based on the comprehensive feature vector and the parameter information of the multimodal brain-computer interface device under test include: S201, Build a test knowledge base.

[0062] The test knowledge base includes: a basic test case template library, a test rule library, and a historical test database.

[0063] The basic test case template library includes test templates for different brain-computer interface paradigms, such as the P300 template, steady-state visual evoked potential template, motor imagery template, and slow cortical potential template. Each template includes: a test task description, default values ​​for stimulus parameters, expected response type, and pass / fail criteria.

[0064] The test rule base includes mapping rules between signal quality and test parameters. For example, "If the signal-to-noise ratio is lower than the threshold T, the stimulation duration is increased by 20%" and "If the frequency of artifacts is higher than the threshold F, the number of trials is increased."

[0065] The test rule base also includes matching rules between product parameters and test templates. For example, "If the product supports the motion imagery paradigm, then the motion imagery template will be selected first" and "If the product sampling rate is lower than 100Hz, then high-frequency stimulation tasks will be disabled."

[0066] The historical test database includes successful and failed cases from historical tests, which are used for rule optimization and machine learning model training.

[0067] S202: Obtain input information.

[0068] The input information includes a comprehensive feature vector and parameter information of the device under test (DUT). The comprehensive feature vector reflects the current signal quality and brain activity state. The DUT parameter information includes model number, supported modal types, supported paradigm types, sampling rate range, and communication protocol. The DUT parameter information can be entered by the tester before testing or obtained through automatic identification.

[0069] S203: Based on the acquired input information, reason through the rule engine and match a suitable basic test template from the knowledge base.

[0070] One specific implementation approach is to use a rule engine based on the Rete algorithm. The rule engine loads all rules from the test rule base, performs pattern matching based on the comprehensive feature vector and device parameters, and triggers rules that meet the conditions.

[0071] For example, when the rule "IF device supports P300 AND current signal-to-noise ratio > 3 THEN select P300 standard template" is triggered, the system loads the P300 standard template from the template library.

[0072] S204: After selecting the basic test template, the test parameters in the template are dynamically adjusted according to the real-time signal status reflected by the comprehensive feature vector.

[0073] For example, for the P300 template, the default stimulus duration is 100ms and the stimulus interval is 200ms. If the signal-to-noise ratio (SNR) in the integrated feature vector is low, indicating poor signal quality, the stimulus duration is adjusted to 150ms according to the rule "if SNR < threshold, then increase stimulus duration by 50%". If the brain activity state features show that the user's attention level is low, the system adjusts the stimulus interval to 300ms according to the rule "if attention level < threshold, then increase stimulus interval by 50%".

[0074] In this embodiment, intelligent generation of test cases and adaptive parameter adjustment are achieved through a test knowledge base and test rule reasoning. The introduction of the test knowledge base makes test case generation based on evidence, ensuring the standardization and repeatability of the testing process; parameter adjustment based on real-time signal states enables the testing process to adapt to signal differences between different users and at different times, improving the robustness and effectiveness of the test.

[0075] Optionally, by combining the real-time signal state reflected by the comprehensive feature vector with the parameter information of the multimodal brain-computer interface device under test, reasoning is performed through test rules to adjust test parameters and test sequences, generating a test case set corresponding to the multimodal brain-computer interface device under test, including: Analyze the comprehensive feature vector to obtain the current signal quality index and brain activity state characteristics; Based on the parameter information of the multimodal brain-computer interface device under test, determine the supported brain-computer interface paradigm type and communication protocol; Match basic test case templates corresponding to the brain-computer interface paradigm type from the test knowledge base; Based on signal quality metrics, the stimulation parameters in the test case template are dynamically adjusted through a rule engine, including stimulation duration, stimulation interval, and number of repetitions. Based on brain activity state characteristics, a reinforcement learning model is used to optimize the test sequence and select the next test task that maximizes information gain. The information gain is calculated based on the uncertainty measure of the current test result. The adjusted test parameters are combined with the optimized test sequences to generate executable test case scripts.

[0076] In this embodiment, real-time signal quality and brain activity state are obtained by parsing the comprehensive feature vector. Combined with the product parameter matching and adaptation of the basic test case template, the stimulus parameters are dynamically adjusted for signal quality by the rule engine, and the test sequence is optimized based on brain activity state by the reinforcement learning model. This ensures that the parameters and sequence of the test cases are highly adapted to the real-time state and inherent characteristics of the device under test, maximizing the information gain of the test. At the same time, an executable test case script is generated, realizing the intelligent and personalized generation of test cases, further improving the relevance and effectiveness of the test, reducing invalid tests, and improving test efficiency.

[0077] In some examples, combined Figure 3 As shown, a reinforcement learning model is introduced to intelligently optimize the test sequence. The specific optimization process includes: S301 extracts signal quality indicators and brain activity state features from the comprehensive feature vector.

[0078] Signal quality metrics include the signal-to-noise ratio (SNR) of each mode, artifact frequency, and signal stability index. These can be obtained through the values ​​of the corresponding dimensions in the feature vector or through a dedicated signal quality assessment network.

[0079] Brain activity state characteristics include the current brain state, such as resting state, task state, fatigue state, attention level, cognitive load, etc. These can be extracted from the feature vector using a pre-trained state classifier.

[0080] S302, based on the parameter information of the multimodal brain-computer interface device under test, determine the supported brain-computer interface paradigm type and communication protocol.

[0081] Supported brain-computer interface paradigms include P300 and motor imagery, which determines the types of test tasks that can be selected.

[0082] The communication protocol and interface type determine the interaction method for test execution.

[0083] S303: Match the basic test case template corresponding to the brain-computer interface paradigm type from the test knowledge base.

[0084] The test knowledge base is used to match the basic test case templates that correspond to the device's supported paradigms, serving as the starting point for generating test sequences.

[0085] S304, based on signal quality indicators, dynamically adjusts the stimulus parameters in the test case template through a rule engine. The stimulus parameters include stimulus duration, stimulus interval, and number of repetitions.

[0086] Based on the signal quality indicators, the stimulus parameters in the test case template are dynamically adjusted using a rule engine. The specific adjustment method is the same as step S304 in the above embodiment.

[0087] S305, based on brain activity state characteristics, optimizes the test sequence through a reinforcement learning model and selects the next test task that maximizes information gain.

[0088] This step optimizes the test sequence using a reinforcement learning model and dynamically selects the next test task.

[0089] The design of the reinforcement learning model is as follows: State space: The current test state, including a list of completed test tasks, the test results of each task, and the current comprehensive feature vector.

[0090] Motion space: The next test task that can be selected, including the task type and difficulty level. Task types include P300 and motion imagery.

[0091] Reward function: Designed based on the information gain of the test task. Information gain is calculated based on the uncertainty measure of the current test results. Specifically, if the uncertainty of a certain performance indicator of the brain-computer interface device is high, and a certain test task can effectively reduce this uncertainty, then the expected information gain of that task is large.

[0092] Information gain is calculated as follows: Information Gain = H(Current) - H(Current | Expected Result After Task Execution); where H represents information entropy, used to measure uncertainty. H(Current) represents the uncertainty (entropy) of the system's perception of device performance in the current state. H(Current | Expected Result After Task Execution) is the conditional entropy, representing the remaining uncertainty of the system's perception of device performance after assuming the execution of a specific test task and obtaining a certain expected result. The system maintains a probability distribution regarding device performance, updating this distribution based on the results after each test task; the information gain is the entropy reduction before and after the update. H(Current) reflects the degree of ambiguity in understanding various performance indicators of the device, such as recognition accuracy and response time, based on the currently obtained test results. Higher uncertainty indicates that more tests are needed to clarify the true capabilities of the device. H(Current | Expected Result After Task Execution) is the system's expected estimate of the remaining uncertainty after the completion of a test task, based on current knowledge, before executing that task. It is the subtrahend in the information gain formula; the larger the information gain, the greater the expected knowledge increment from the task, making it more worthwhile to prioritize its execution.

[0093] Decision-making process: At each decision point, the reinforcement learning model selects the next test task that maximizes the expected information gain based on the current state. As testing progresses, the reinforcement learning model continuously adjusts the subsequent test sequence based on the results obtained.

[0094] S306 generates an executable test script.

[0095] The adjusted test parameters are combined with the optimized test sequences to generate structured, executable test case scripts. The scripts include a complete description of the test process, stimulus parameters for each task, expected response types, pass / fail criteria, and other information.

[0096] In this embodiment, by introducing a reinforcement learning model, the generation of test sequences is no longer fixed or driven by simple rules, but becomes a dynamic programming problem. The model can intelligently select the most valuable next test based on the information obtained during the testing process, avoiding wasting testing resources on already defined performance dimensions and concentrating testing resources on performance dimensions with higher uncertainty. This information gain maximization strategy can obtain the most comprehensive performance evaluation with the fewest number of tests, greatly improving testing efficiency. Simultaneously, adjusting stimulus parameters based on signal quality improves the executability of the testing task and the validity of the data.

[0097] Optionally, combined Figure 4 As shown, the steps for evaluating the performance of the multimodal brain-computer interface device under test and outputting the comprehensive evaluation results based on test response data and integrated feature vectors include: S401, extract the recognition results and response time of the multimodal brain-computer interface device under test from the test response data.

[0098] In this step, the identification results and response time are extracted from the test response data. The identification results include the output of the device under test for each test task, such as classification labels and confidence scores. The response time includes the time delay between stimulus presentation and the device's output of the identification result.

[0099] S402 extracts real-time quality indicators and fusion features of brain signals from various modalities from the comprehensive feature vector.

[0100] In this step, the real-time quality index of each modality of brain signal refers to the signal quality score obtained at each moment during the test by parsing specific dimensions of the integrated feature vector or calling a dedicated signal quality evaluation function. The fusion feature refers to the integrated feature vector itself, which serves as the input for subsequent cross-modal collaborative scoring.

[0101] S403 calculates the scores of the multimodal brain-computer interface device under test on various indicators according to the preset evaluation index system. The evaluation index system includes recognition accuracy, response time, information transmission rate and signal stability index.

[0102] In this step, recognition accuracy refers to the proportion of correctly classified trials out of the total number of trials for a classification task. For tasks of different difficulty levels, accuracy can be calculated separately or a weighted average can be used. Response time is calculated as the average, median, and standard deviation of response times across all valid trials; these can be calculated separately for different task types. Information transmission rate = (logarithmic function of recognition accuracy × number of classifications) / average response time. This indicator comprehensively reflects the device's recognition performance and response speed. Signal stability index refers to the real-time quality index of each modal signal during testing, calculating the change in signal quality over time. Signal stability index = 1 - (signal quality standard deviation / signal quality mean). The higher the index, the more stable the signal output by the device.

[0103] S404, based on fusion features, evaluates the comprehensive robustness of the multimodal brain-computer interface device under test in multimodal collaborative working scenarios and obtains a cross-modal collaborative score.

[0104] This step is a unique evaluation dimension of the present invention, used to evaluate the overall robustness of the device in multimodal collaborative working scenarios.

[0105] As a specific implementation method, the following calculation can be performed: In the test data, identify time periods with poor single-modal signal quality, such as periods with more EEG signal artifacts. Compare whether the device's recognition accuracy significantly decreases during these time periods. If the decrease is not significant, it indicates that the device has good cross-modal collaborative capabilities; if the decrease is significant, it indicates that the device is highly dependent on specific modalities.

[0106] Cross-modal collaborative score = 1 - (accuracy decrease during poor signal periods / accuracy during good signal periods).

[0107] S405 integrates the scores of various indicators and cross-modal collaborative scoring to generate a comprehensive evaluation result that includes quantitative scores, performance analysis conclusions, and visualization charts.

[0108] The scores of each indicator and the cross-modal collaborative score are integrated to generate a comprehensive evaluation report. The report includes: (1) The quantitative scoring section includes, but is not limited to: numerical scores and grade ratings for each indicator, such as excellent / good / qualified / unqualified. Cross-modal collaborative scoring and grade rating. The comprehensive performance score refers to the weighted sum of each indicator.

[0109] (2) The performance analysis conclusions include, but are not limited to: the device's superior performance dimensions, such as "the P300 paradigm recognition accuracy is excellent". The dimensions of the device that need improvement, such as "the response time of the motor imagery paradigm is relatively long". Multimodal collaborative performance analysis, such as "when the EEG signal is interfered with, the functional near-infrared spectrum can effectively compensate, and the collaborative performance is good".

[0110] (3) The visualization charts include, but are not limited to: a curve of accuracy as a function of the test task; a histogram of response time distribution; a trend chart of signal quality as a function of time; a radar chart of cross-modal cooperative performance; and a bar chart comparing with other reference devices.

[0111] The multi-dimensional evaluation index system comprehensively reflects the performance of the device under test. In particular, the introduction of cross-modal collaborative scoring specifically assesses the multi-modal collaborative capabilities of multi-modal brain-computer interface devices, a unique evaluation dimension not provided by existing testing methods. Visualized charts make the evaluation results intuitive and easy to understand, allowing testers to quickly grasp the device's performance characteristics and areas for improvement. The automatic generation of comprehensive evaluation reports eliminates the tedious work of manual data processing, improving testing efficiency.

[0112] In this embodiment, test response data and comprehensive feature vectors are combined to extract multi-dimensional evaluation criteria. The preset multi-dimensional evaluation index system can more comprehensively measure the basic performance of the device. The cross-modal collaborative scoring based on fused features specifically evaluates the core collaborative performance of the multimodal device. Finally, a comprehensive evaluation result including quantitative scores, analysis conclusions and visualization charts is generated, realizing a comprehensive, quantitative and visual evaluation of the performance of the multimodal brain-computer interface device. The test results are more intuitive and have more reference value, solving the problem of single test results and inability to evaluate collaborative performance in the existing technology.

[0113] Optionally, the timestamped multimodal brain signals are preprocessed to obtain synchronized and aligned multimodal signal data, including: filtering the acquired brain signals of at least two modalities to remove noise and interference signals; amplifying and standardizing the filtered brain signals to convert signals of different dimensions into a unified data format and magnitude; and interpolating and aligning the brain signals of each modality with different sampling rates based on the timestamps so that the points or segments of each modality signal are aligned on the time axis to obtain synchronized and aligned multimodal signal data.

[0114] In this embodiment, filtering effectively removes noise and interference from the signal, improving signal quality. Amplification and normalization processes unify multimodal signals with different dimensions and amplitudes. Furthermore, interpolation alignment based on timestamps solves the time synchronization problem for modal signals with different sampling rates, achieving point or segment alignment. This ultimately yields high-quality, synchronously aligned multimodal signal data, providing a reliable data foundation for subsequent feature fusion and testing, and effectively avoiding the impact of signal noise, dimensional differences, and time asynchrony on the accuracy of test results.

[0115] In some embodiments, combined with Figure 5 As shown, a testing method for a multimodal brain-computer interface device is provided, comprising: S501, acquire at least two modalities of brain signals output by the multimodal brain-computer interface device under test, and unify the timestamps.

[0116] S502 preprocesses the timestamped multimodal brain signals to obtain synchronized and aligned multimodal signal data.

[0117] S503 extracts high-dimensional features corresponding to brain signal data of at least two modalities through a multi-branch neural network.

[0118] S504 calculates the correlation weights between high-dimensional features corresponding to brain signal data from at least two modalities through a cross-modal attention mechanism, and performs weighted fusion to generate a comprehensive feature vector.

[0119] S505 retrieves basic test case templates and test rules from the test knowledge base.

[0120] S506 analyzes the comprehensive feature vector to obtain the current signal quality index and brain activity state characteristics.

[0121] S507, based on the parameter information of the multimodal brain-computer interface device under test, determines the supported brain-computer interface paradigm type and communication protocol.

[0122] S508 matches the basic test case template corresponding to the brain-computer interface paradigm type from the test knowledge base.

[0123] S509, based on signal quality metrics, dynamically adjusts the stimulus parameters in the test case template through a rule engine, including stimulus duration, stimulus interval, and number of repetitions.

[0124] S510 optimizes the test sequence through a reinforcement learning model based on brain activity state characteristics, and selects the next test task that maximizes information gain. The information gain is calculated based on the uncertainty measure of the current test result.

[0125] S511 combines the adjusted test parameters with the optimized test sequence to generate an executable test case script.

[0126] S512 interacts with the multimodal brain-computer interface device under test according to the test case set, executes the test, and records the test response data.

[0127] S513 extracts the recognition results and response time of the multimodal brain-computer interface device under test from the test response data.

[0128] S514 extracts real-time quality indicators and fusion features of brain signals from various modalities from the comprehensive feature vector.

[0129] S515 calculates the scores of the multimodal brain-computer interface device under test on various indicators according to the preset evaluation index system. The evaluation index system includes recognition accuracy, response time, information transmission rate and signal stability index.

[0130] S516, based on fusion features, evaluates the comprehensive robustness of the multimodal brain-computer interface device under test in multimodal collaborative working scenarios and obtains a cross-modal collaborative score.

[0131] S517 integrates the scores of various indicators and cross-modal collaborative scoring to generate a comprehensive evaluation result that includes quantitative scores, performance analysis conclusions, and visualization charts.

[0132] In some embodiments of this invention, combined with Figure 6 The aforementioned provides a testing system 60 for a multimodal brain-computer interface device, comprising: The data acquisition module 601 is used to acquire brain signals from at least two modalities output by the multimodal brain-computer interface device under test, and to unify the timestamps. Optionally, the data acquisition module includes a unified clock source and multiple acquisition sub-modules, each acquisition sub-module being used to acquire brain signals of one type of modality.

[0133] The preprocessing module 602 is used to preprocess the timestamped multimodal brain signals to obtain synchronized and aligned multimodal signal data. Optionally, the preprocessing module includes a filtering unit, an amplification unit, a normalization unit, and a signal alignment unit.

[0134] The multimodal fusion module 603 is used to extract and fuse features from synchronously aligned multimodal signal data using a deep learning-based fusion model to generate a comprehensive feature vector. Optionally, the multimodal fusion module includes a multi-branch feature extraction sub-network, a cross-modal attention fusion layer, and a fusion output layer.

[0135] The test case generation module 604 is used to generate a set of test cases corresponding to the multimodal brain-computer interface device under test based on the comprehensive feature vector and the parameter information of the device. Optionally, the test case generation module includes a test knowledge base, a rule engine, and a reinforcement learning model.

[0136] The test execution module 605 is used to interact with the multimodal brain-computer interface device under test according to the test case set, execute tests, and record test response data. Optionally, the test execution module includes an instruction sending unit, a data receiving unit, and a process recording unit.

[0137] Evaluation module 606 is used to evaluate the performance of the multimodal brain-computer interface device under test based on test response data and comprehensive feature vectors, and output a comprehensive evaluation result. Optionally, the evaluation module includes an index calculation unit, a cross-modal collaborative scoring unit, and a report generation unit.

[0138] The testing system for multimodal brain-computer interface devices provided in this disclosure achieves synchronous acquisition of multimodal brain signals using a unified timestamp, completes signal synchronization alignment through preprocessing, and then mines the intrinsic correlation of multimodal signals through a deep learning fusion model. Based on signal features and product parameters, it adaptively generates test case sets and finally completes automated testing and comprehensive evaluation. This system realizes collaborative testing of multimodal signals, significantly improves the accuracy and comprehensiveness of test results, automates the entire testing process, effectively improves testing efficiency, adapts to multimodal brain-computer interface devices with different parameters, and enhances testing compatibility and flexibility.

[0139] Combination Figure 7 As shown, this disclosure provides a testing apparatus 70 for a multimodal brain-computer interface device, including a processor 700 and a memory 701. Optionally, the apparatus 70 may further include a communication interface 702 and a bus 703. The processor 700, communication interface 702, and memory 701 can communicate with each other via the bus 703. The communication interface 702 can be used for information transmission. The processor 700 can call logical instructions in the memory 701 to execute the testing method for the multimodal brain-computer interface device described in the above embodiment.

[0140] Furthermore, the logic instructions in the aforementioned memory 701 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium.

[0141] The memory 701, as a computer-readable storage medium, can be used to store software programs and computer-executable programs, such as program instructions / modules corresponding to the methods in the embodiments of this disclosure. The processor 100 executes functional applications and data processing by running the program instructions / modules stored in the memory 701, that is, it implements the testing method for the multimodal brain-computer interface device in the above embodiments.

[0142] The memory 701 may include a program storage area and a data storage area. The program storage area may store the operating system and application programs required for at least one function; the data storage area may store data created based on the use of the terminal device. Furthermore, the memory 701 may include high-speed random access memory and may also include non-volatile memory.

[0143] In some embodiments, a computer-readable storage medium is provided storing program instructions that, when executed, cause a computer to perform a testing method for a multimodal brain-computer interface device as described in any of the above embodiments.

[0144] The technical solutions of this disclosure can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes one or more instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the method described in this disclosure. The aforementioned storage medium can be a non-transitory storage medium, such as a USB flash drive, external hard drive, read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk, etc., and other media capable of storing program code.

[0145] The foregoing description and accompanying drawings fully illustrate embodiments of this disclosure to enable those skilled in the art to practice them. Other embodiments may include structural, logical, electrical, procedural, and other changes. The embodiments represent only possible variations. Individual components and functions are optional unless explicitly required, and the order of operation may vary. Parts and features of some embodiments may be included in or replace parts and features of other embodiments. Moreover, the terminology used in this application is for describing embodiments only and is not intended to limit the claims. As used in the description of embodiments and claims, the singular forms “a,” “an,” and “the” are intended to equally include the plural forms unless the context clearly indicates otherwise. Similarly, the term “and / or” as used in this application means including one or more of the associated listed items and all possible combinations thereof. Additionally, when used in this application, the term "comprise" and its variations "comprises" and / or "comprising" refer to the presence of stated features, integrals, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components, and / or groups thereof. Without further limitations, an element defined by the phrase "comprises a..." does not exclude the presence of other identical elements in the process, method, or apparatus that includes said element. In this document, each embodiment may focus on the differences from other embodiments, and similar or identical parts between embodiments can be referred to mutually. For methods, products, etc., disclosed in the embodiments, if they correspond to the method section disclosed in the embodiments, the relevant parts can be referred to the description of the method section.

[0146] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the embodiments of this disclosure. Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0147] The methods and products disclosed in the embodiments herein (including but not limited to devices and equipment) can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For instance, the division of units may be merely a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces, and the indirect coupling or communication connection of devices or units may be electrical, mechanical, or other forms. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to implement this embodiment according to actual needs. In addition, the functional units in the embodiments of this disclosure may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0148] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than that shown in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. In the descriptions corresponding to the flowcharts and block diagrams in the accompanying drawings, the operations or steps corresponding to different blocks may also occur in a different order than disclosed in the description, and sometimes there is no specific order between different operations or steps. For example, two consecutive operations or steps may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. Each block in a block diagram and / or flowchart, and combinations of blocks in a block diagram and / or flowchart, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

Claims

1. A testing method for a multimodal brain-computer interface device, characterized in that, include: Acquire brain signals from at least two modalities output by the multimodal brain-computer interface device under test, and unify the timestamps; Preprocessing of timestamped multimodal brain signals yields synchronized and aligned multimodal signal data; A deep learning-based fusion model is used to extract and fuse features from synchronized, aligned multimodal signal data to generate a comprehensive feature vector. Based on the comprehensive feature vector and the parameter information of the multimodal brain-computer interface device under test, a test case set corresponding to the multimodal brain-computer interface device under test is generated. Interact with the multimodal brain-computer interface device under test according to the test case set, execute the test, and record the test response data; Based on test response data and comprehensive feature vectors, the performance of the multimodal brain-computer interface device under test is evaluated, and a comprehensive evaluation result is output.

2. The test method according to claim 1, characterized in that, The deep learning-based fusion model includes a multi-branch neural network with a cross-modal attention mechanism; it employs a deep learning-based fusion model to extract and fuse features from synchronized, aligned multimodal signal data, generating a comprehensive feature vector, including: High-dimensional features corresponding to brain signal data of at least two modalities are extracted using a multi-branch neural network. The association weights between high-dimensional features corresponding to brain signal data from at least two modalities are calculated using a cross-modal attention mechanism, and then weighted and fused to generate a comprehensive feature vector.

3. The test method according to claim 2, characterized in that, The association weights between high-dimensional features corresponding to brain signal data from at least two modalities are calculated using a cross-modal attention mechanism, and then weighted and fused to generate a comprehensive feature vector, including: The high-dimensional features of each modality of brain signal are mapped to query features, key features, and value features, respectively; Based on the query features of the first modality brain signal, the attention score between the first modality brain signal and the key features of the second modality brain signal is calculated, and the attention score is normalized to obtain the attention weight of the first modality brain signal to the second modality brain signal. The cross-modal enhancement features of the first modality brain signal are obtained by weighting and summing the value features of the second modality brain signal using the attention weights of the first modality brain signal to the second modality brain signal; Similarly, calculate the cross-modal enhancement features of each modality of brain signal and the other modalities of brain signals in at least two modalities; The cross-modal enhancement features of all modal brain signals are fused with the original high-dimensional features of each modality brain signal to generate the comprehensive feature vector.

4. The test method according to claim 3, characterized in that, The cross-modal enhancement features of all modal brain signals are fused with the original high-dimensional features of each modality brain signal to generate the comprehensive feature vector, including: The original high-dimensional features of each modality brain signal are spliced ​​with the corresponding cross-modal enhancement features to form the enhancement feature representation of each modality; The enhanced feature representations of at least two modalities of brain signals are input into the fusion layer, and nonlinear transformation and dimensionality reduction are performed through a fully connected network; The output low-dimensional dense vector containing cross-modal correlation information is used as the comprehensive feature vector.

5. The test method according to claim 1, characterized in that, Based on the comprehensive feature vector and the parameter information of the multimodal brain-computer interface device under test, a test case set corresponding to the multimodal brain-computer interface device under test is generated, including: Obtain basic test case templates and test rules from the test knowledge base; By combining the real-time signal state reflected by the comprehensive feature vector with the parameter information of the multimodal brain-computer interface device under test, reasoning is performed through test rules to adjust test parameters and test sequences, thereby generating a test case set corresponding to the multimodal brain-computer interface device under test.

6. The test method according to claim 5, characterized in that, By combining the real-time signal state reflected in the comprehensive feature vector with the parameter information of the multimodal brain-computer interface device under test, reasoning is performed through test rules to adjust test parameters and test sequences, generating a test case set corresponding to the multimodal brain-computer interface device under test, including: Analyze the comprehensive feature vector to obtain the current signal quality index and brain activity state characteristics; Based on the parameter information of the multimodal brain-computer interface device under test, determine the supported brain-computer interface paradigm type and communication protocol; Match basic test case templates corresponding to the brain-computer interface paradigm type from the test knowledge base; Based on signal quality metrics, the stimulation parameters in the test case template are dynamically adjusted through a rule engine, including stimulation duration, stimulation interval, and number of repetitions. Based on brain activity state characteristics, a reinforcement learning model is used to optimize the test sequence and select the next test task that maximizes information gain. The information gain is calculated based on the uncertainty measure of the current test result. The adjusted test parameters are combined with the optimized test sequences to generate executable test case scripts.

7. The test method according to any one of claims 1 to 6, characterized in that, Based on test response data and comprehensive feature vectors, the performance of the tested multimodal brain-computer interface device is evaluated, and a comprehensive evaluation result is output, including: Extract the recognition results and response time of the multimodal brain-computer interface device under test from the test response data; Real-time quality indicators and fusion features of brain signals from each modality are extracted from the comprehensive feature vector; Based on the preset evaluation index system, the scores of the multimodal brain-computer interface device under test on various indicators are calculated. The evaluation index system includes recognition accuracy, response time, information transmission rate and signal stability index. Based on fusion features, the comprehensive robustness of the multimodal brain-computer interface device under test in multimodal collaborative working scenarios is evaluated, and a cross-modal collaborative score is obtained. By integrating the scores of various indicators and cross-modal collaborative scoring, a comprehensive evaluation result is generated, which includes quantitative scores, performance analysis conclusions, and visualization charts.

8. The test method according to any one of claims 1 to 6, characterized in that, Preprocessing of timestamped multimodal brain signals yields synchronized and aligned multimodal signal data, including: The acquired brain signals of at least two modalities were filtered to remove noise and interference signals. The filtered brain signals are amplified and standardized to convert signals of different dimensions into a unified data format and magnitude. Based on timestamps, interpolation and alignment are performed on brain signals of different sampling rates to align points or segments of each modality on the time axis, resulting in synchronously aligned multimodal signal data.

9. A testing device for a multimodal brain-computer interface device, comprising a processor and a memory storing program instructions, characterized in that, The processor is configured to execute, when running the program instructions, the test method for a multimodal brain-computer interface device as described in any one of claims 1 to 8.

10. A computer-readable storage medium storing program instructions, characterized in that, When the program instructions are executed, they cause the computer to perform the testing method for a multimodal brain-computer interface device as described in any one of claims 1 to 8.

Images