Electroencephalogram action fusion interaction method and system

Abstract

The invention provides an electroencephalogram action fusion interaction method and device, and relates to the technical field of human-computer interaction. The electroencephalogram motion fusion interaction method comprises the following steps: acquiring a user input signal, a user electroencephalogram signal and a user head posture signal; inputting the user input signal and the user electroencephalogram signal into a pre-trained intention recognition model to obtain a user interaction intention; performing noise reduction and signal enhancement processing on the electroencephalogram signal of the user according to the head posture signal of the user to obtain a target electroencephalogram signal; inputting the target electroencephalogram signal and the user head posture signal into a pre-trained intention error correction model to obtain a user feedback intention; and determining interaction content according to the user feedback intention and the user interaction intention, and obtaining an interaction result according to the interaction content. According to the technical scheme, through multi-mode signal fusion, the intention capturing ability of the electroencephalogram signals and the physical operation advantages of the user input signals are effectively combined, and the recognition precision and the response speed are high.

Description

Technical Field

[0001] The present application relates to the field of human-computer interaction technology, and particularly to an electroencephalogram-action fusion interaction method and system. Background Art

[0002] Currently, single interaction systems are generally divided into an electroencephalogram (EEG) signal-based control system and a gesture-action recognition-based control system. The EEG signal-based control system triggers preset operation instructions by collecting EEG signals and analyzing the user's attention, relaxation level, or specific brain wave patterns. The gesture-action recognition-based control system relies on devices such as cameras and inertial sensors to recognize the movements of the user's hands or body for interactive operations. However, both of the above single interaction systems have problems of low accuracy and slow response speed.

[0003] Specifically, for the EEG signal-based control system, EEG signals are extremely vulnerable to external interference, resulting in a decline in signal quality, especially electromyogram (EMG) signals and environmental noise, which may lead to a decrease in the accuracy of intention recognition. Moreover, when relying solely on EEG signals for complex control, there are certain limitations in terms of accuracy and response time. In addition, traditional 64-lead wet electrode EEG caps are difficult to wear and require too much preparation time, resulting in a decrease in the user's willingness to wear them. For the gesture-action recognition-based control system, gesture recognition is easily restricted by problems such as light, occlusion, or drift caused by long-term use of IMU sensors, resulting in a poor interactive experience and also affecting accuracy and response time.

[0004] On this basis, some studies have proposed an interaction system that combines electroencephalogram (EEG) signals and gesture-action recognition. Such systems combine the perception ability of EEG signals and the intuitiveness of gesture recognition, aiming to provide users with a natural and contactless control method. Common applications include fields such as medical rehabilitation, virtual reality (VR), augmented reality (AR), game interaction, and smart home. Current technologies usually use multi-electrode EEG head-mounted devices to collect EEG signals, and at the same time use cameras, 9-axis inertial measurement units (IMUs), or other wearable devices to recognize gesture actions, ultimately achieving the ability to interact with devices. However, existing interaction systems that combine electroencephalogram (EEG) signals and gesture-action recognition usually have problems in coordinating EEG signals and gesture-action recognition. Summary of the Invention

[0005] Based on this, the present application provides an electroencephalogram-action fusion interaction method and system to achieve electroencephalogram-action fusion interaction that coordinates EEG signals and gesture-action recognition.

[0006] According to one aspect of the present application, a method for electroencephalogram (EEG) - motion fusion interaction is proposed, including: obtaining a user input signal, a user EEG signal, and a user head pose signal, wherein the user input signal is a voice signal or a gesture signal; inputting the user input signal and the user EEG signal into a pre - trained intention recognition model to obtain a user interaction intention; performing noise reduction and signal enhancement processing on the user EEG signal according to the user head pose signal to obtain a target EEG signal; inputting the target EEG signal and the user head pose signal into a pre - trained intention error - correction model to obtain a user feedback intention; determining interaction content according to the user feedback intention and the user interaction intention, and obtaining an interaction result according to the interaction content.

[0007] According to some embodiments, the method further includes: constructing an error - correction training data set and an identification training data set according to preset interaction task samples; training an intention recognition model based on the identification training data set, and training an intention error - correction model based on the error - correction training data set; collecting noisy EEG signal samples and noise - free EEG signal samples of a user for the same stimulus source, performing denoising processing on the noisy EEG signal samples to obtain denoised EEG signal samples, and constructing a noise training data set according to the denoised EEG signal samples and the noise - free EEG signal samples; training an EEG signal enhancement model based on the noise training data set.

[0008] According to some embodiments, performing noise reduction and signal enhancement processing on the user EEG signal according to the user head pose signal to obtain a target EEG signal includes: decomposing the user EEG signal using an empirical mode decomposition algorithm to obtain multiple decomposition signals in different frequency bands, and removing the decomposition signals in a preset frequency band to obtain an initial denoised EEG signal; simulating a noise signal according to the user head pose signal and the user EEG signal to obtain motion artifacts; filtering the initial denoised EEG signal using the motion artifacts to obtain a denoising result; inputting the denoising result into the EEG signal enhancement model to output the target EEG signal.

[0009] According to some embodiments, constructing an error - correction training data set according to preset interaction task samples includes:

[0010] In response to the user performing an error correction task sample in a preset interaction task sample, obtain the interaction recognition result output by a preset interaction result selection algorithm or interaction result recognition model; collect user feedback, and record the first electroencephalogram (EEG) signal sample and head pose signal sample generated by the user for the interaction recognition result, where the user feedback is the user's satisfaction with the interaction recognition result; in the case where the user feedback is dissatisfaction, record the interaction intention option selected by the user, and obtain the current EEG signal as the second EEG signal sample; update the first EEG signal sample according to the second EEG signal sample, and update the interaction recognition result according to the interaction intention option selected by the user; use the first EEG signal sample and the corresponding head pose signal sample as the first training data, use the user feedback as the first training label of the first training data, and construct an error correction training data set according to the first training data with the first training label.

[0011] According to some embodiments, construct a recognition training data set according to a preset interaction task sample, including: in response to the user performing an EEG recognition task sample in a preset interaction task sample, collect the user's interaction EEG signal sample; in response to the user performing an active recognition task sample in a preset interaction task sample, collect the user input signal sample of the user, where the user input signal sample is a voice signal sample or a gesture signal sample; use the interaction EEG signal sample as the second training data, use the EEG recognition task sample as the second training label of the second training data, and construct an EEG recognition training data set according to the second training data with the second training label; use the user input signal sample as the third training data, use the active recognition task sample as the third training label of the third training data, and construct an action recognition training data set according to the third training data with the third training label; where the EEG recognition training data set and the action recognition training data set constitute the recognition training data set.

[0012] According to some embodiments, train an intention error correction model based on the error correction training data set, including: preprocess and extract features from the first EEG signal sample in the error correction training data set to obtain an EEG signal feature sample; train an error judgment model based on the EEG signal feature sample and its corresponding first training label; input the target EEG signal feature sample into the error judgment model to obtain a judgment result, where the target EEG signal feature sample is one of the EEG signal feature samples; input the target head pose signal sample into a pre-trained head motion recognition model to obtain a head motion recognition result, where the target head pose signal sample is the head pose signal sample corresponding to the first EEG signal sample associated with the target EEG signal feature sample; optimize the error judgment model according to the judgment result and the head motion recognition result to obtain the intention error correction model.

[0013] According to some embodiments, obtaining the interaction recognition result output by a preset interaction result selection algorithm includes: when a user executes a preset interaction task sample, collecting the target test EEG signal of the user for a target interaction option, where the target interaction option is one of the interaction intention options; calculating the correlation coefficient between the target test EEG signal and each of a preset number of pre-constructed EEG signal templates according to the target test EEG signal and the preset number of pre-constructed EEG signal templates; and selecting the interaction intention option corresponding to the EEG signal template with the largest correlation coefficient as the interaction recognition result.

[0014] According to some embodiments, inputting a user input signal and a user EEG signal into a pre-trained intention recognition model to obtain a user interaction intention includes: inputting the user input signal into an action intention recognition model to obtain a first recognition result; inputting the user EEG signal into an EEG intention recognition model to obtain a second recognition result; calculating the user interaction intention according to the first recognition result, the second recognition result, and a preset first weight and second weight; where the EEG intention recognition model is trained based on an EEG recognition training data set, and the action intention recognition model is trained based on an action recognition training data set.

[0015] According to some embodiments, calculating the correlation coefficient between the target test EEG signal and each of a preset number of pre-constructed EEG signal templates according to the target test EEG signal and the preset number of pre-constructed EEG signal templates includes: inputting the target test EEG signal and the preset number of EEG signal templates into a pre-constructed spatial filter bank to calculate the correlation coefficient between the target test EEG signal and each of the EEG signal templates; or solving a first preset formula according to the target test EEG signal and the preset number of EEG signal templates to obtain the correlation coefficient between the target EEG signal and each of the EEG signal templates.

[0016] According to some embodiments, the first preset formula is:

[0017]

[0018] where E represents expectation; X k represents the kth EEG signal template, Y represents the target test EEG signal, r k represents the correlation coefficient between the EEG signal template X k with a stimulation frequency of f k and the EEG signal Y, x k represents a first intermediate variable, x k =X k T W X , y represents a second intermediate variable, y = Y T W Y , W X represents a first weighting coefficient, W YRepresents a second weighting coefficient.

[0019] According to some embodiments, the spatial filter bank is constructed through the following steps: acquiring multi-trial multi-channel EEG signals based on a target interaction option; using the TRCA algorithm, calculating a spatial filter for the target interaction option based on the multi-channel EEG signals, and thus obtaining the spatial filter bank.

[0020] According to some embodiments, the EEG signal template is constructed through the following steps: performing superposition averaging on the multi-trial single-channel EEG signals of the target interaction option to obtain the EEG signal template; or constructing an EEG signal template with the same data length as the single-trial EEG signal and including the stimulation frequency and harmonic frequencies according to the single-trial EEG signal of the target interaction option according to a second preset formula, where the stimulation frequency is the frequency at which the target interaction option visually stimulates the user.

[0021] According to some embodiments, the second preset formula is:

[0022]

[0023] where X k (t) represents the k-th EEG signal template, f k represents the k-th stimulation frequency, k = 1, 2, …, N f , N f is the number of frequency encoding targets, and N h represents the harmonic order, and t represents time.

[0024] According to one aspect of the present application, an EEG-action fusion interaction system includes: a signal acquisition module for acquiring a user input signal, a user EEG signal, and a user head pose signal, where the user input signal is a voice signal or a gesture signal; an intention recognition module for inputting the user input signal and the user EEG signal into a pre-trained intention recognition model to obtain a user interaction intention; a noise reduction and enhancement module for performing noise reduction and signal enhancement processing on the user EEG signal according to the user head pose signal to obtain a target EEG signal; an intention error correction module for inputting the target EEG signal and the user head pose signal into a pre-trained intention error correction model to obtain a user feedback intention; and an interaction result module for determining interaction content according to the user feedback intention and the user interaction intention, and obtaining an interaction result according to the interaction content.

[0025] According to one aspect of the present application, an electronic device is provided, which includes: one or more processors; a storage device for storing one or more programs; when the one or more programs are executed by the one or more processors, the one or more processors implement the method as described above.

[0026] According to one aspect of the present application, a computer-readable medium is provided, on which a computer program is stored, and when the program is executed by a processor, the method described above is implemented.

[0027] Through the above-mentioned embodiments provided by the present application, in the actual application process, the trained intent recognition model is used to fuse multi-modal signals, and the interaction intent is recognized by combining the intent capture ability of the electroencephalogram (EEG) signal and the advantage of physical operation recognition of user input, so as to obtain the user interaction intent; through noise reduction and signal enhancement processing, the effective signals and noise signals in the EEG signal are effectively distinguished; the trained intent correction model is used to perform error correction intent recognition on the target EEG signal obtained by noise reduction and signal enhancement, so as to obtain the user feedback intent; according to the recognized interaction intent and feedback intent, the interaction content is determined and the interaction is performed, thereby improving the recognition accuracy and response speed. Description of the Drawings

[0028] It should be understood that the above general description and the following detailed description are only exemplary and do not limit the present application.

[0029] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for the description of the embodiments will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present application, and those of ordinary skill in the art can also obtain other drawings according to these drawings without exceeding the scope of protection required by the present application.

[0030] Figure 1 It is a flowchart of the EEG-action fusion interaction method provided by the embodiment of the present application;

[0031] Figure 2 It is a framework schematic diagram of the EEG-action fusion interaction method provided by the embodiment of the present application;

[0032] Figure 3 It is a flowchart of performing noise reduction and signal enhancement processing on the user's EEG signal according to the user's head pose signal to obtain a target EEG signal provided by the embodiment of the present application;

[0033] Figure 4 It is a flowchart of constructing an error correction training data set and an identification training data set according to a preset interaction task sample provided by the embodiment of the present application;

[0034] Figure 5 It is a flowchart of training an intent correction model based on an error correction training data set provided by the embodiment of the present application;

[0035] Figure 6 It is a flowchart of obtaining an interaction recognition result output by a preset interaction result selection algorithm provided by the embodiment of the present application;

[0036] Figure 7 Flowchart for constructing the spatial filter bank provided by the embodiment of the present application;

[0037] Figure 8 Flowchart for constructing the EEG signal template provided by the embodiment of the present application;

[0038] Figure 9 Block diagram of the open-set fine-grained recognition device provided by the embodiment of the present application;

[0039] Figure 10 Schematic structural diagram of the electronic device provided by the embodiment of the present application. Detailed implementation manners

[0040] Next, the technical solutions in the embodiments of the present application will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative efforts fall within the protection scope of the present application.

[0041] In addition, the described features, structures or characteristics can be combined in any suitable manner in one or more embodiments. In the following description, many specific details are provided to give a full understanding of the embodiments of the present application. However, those skilled in the art will realize that the technical solutions of the present application can be practiced without one or more of the specific details, or other methods, components, devices, steps, etc. can be adopted. In other cases, well-known methods, devices, implementations or operations are not shown or described in detail to avoid obscuring various aspects of the present application.

[0042] The block diagrams shown in the accompanying drawings are only functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities can be implemented in software form, or implemented in one or more hardware modules or integrated circuits, or implemented in different networks and / or processor devices and / or microcontroller devices.

[0043] The flowcharts shown in the accompanying drawings are only exemplary illustrations, not necessarily including all the contents and operations / steps, nor necessarily executed in the described order. For example, some operations / steps can be decomposed, while some operations / steps can be combined or partially combined. Therefore, the actual execution order may change according to the actual situation.

[0044] It should be understood that although terms such as first, second, and third may be used herein to describe various components, these components should not be limited by these terms. These terms are used to distinguish one component from another. Thus, the first component discussed below may be referred to as the second component without departing from the teachings of the concepts of this application. As used herein, the term "and / or" includes any one of the associated listed items and all combinations of one or more of them.

[0045] Specific implementation manners can refer to the following embodiments.

[0046] Figure 1 It is a flowchart of the electroencephalogram-action fusion interaction method provided by the embodiments of this application. As Figure 1 shown, this method includes step S110-step S150.

[0047] In step S110, a user input signal, a user electroencephalogram signal, and a user head pose signal are acquired, where the user input signal is a voice signal or a gesture action signal.

[0048] The user input signal refers to a signal actively input by the user, including a voice signal or a gesture action signal. In the actual acquisition process, according to the exemplary embodiments, a hardware device (such as a data glove or a camera) is used to pick up gesture actions, and the gesture action data picked up by the hardware is used as the user input signal; or

[0049] According to the exemplary embodiments, a hardware device (such as a microphone) is used to pick up voice signals, and the voice signal data picked up by the hardware is used as the user input signal.

[0050] The user electroencephalogram signal refers to the electroencephalogram signal of the user. In the actual acquisition process, according to the exemplary embodiments, a wearable electroencephalograph (such as an electroencephalogram device based on wearable dry electrodes / semi-dry electrodes) is selected to pick up electroencephalogram signals, and the electroencephalogram signal data picked up by the hardware is used as the user electroencephalogram signal. Compared with the 64-lead wet electrode electroencephalogram cap in the prior art, it is more convenient to wear and has a better user experience.

[0051] The user head pose signal refers to a signal that describes the rotation of the user's head in the vertical, horizontal, and lateral directions. In the actual acquisition process, according to the exemplary embodiments, an inertial sensor is used to collect the head acceleration and angular velocity signals of the user, and the collected head acceleration and angular velocity signals are used as the user head pose signal.

[0052] In step S120, the user input signal and the user electroencephalogram signal are input into a pre-trained intention recognition model to obtain a user interaction intention.

[0053] This application trains an intention recognition model to fuse electroencephalogram (EEG) and user input signals for interactive intention recognition, greatly improving the recognition accuracy and response speed of the interactive system, and effectively solving the problems of difficult coordination between EEG signals and gesture action recognition, as well as the low accuracy and slow response speed of a single interactive system.

[0054] In step S130, the user's EEG signals are denoised and signal-enhanced according to the user's head pose signals to obtain target EEG signals.

[0055] It should be noted that while wearable electroencephalographs bring convenience, the signal quality and signal-to-noise ratio of the user's EEG signals obtained can also be further improved.

[0056] Among them, the noise of wearable electroencephalographs usually comes from blinking, head movement, and muscle movement. Therefore, this application models and eliminates the noise of the obtained user EEG signals.

[0057] Since denoising will cause loss of EEG information in the EEG signals, this application also enhances the denoised EEG signals. Finally, the target EEG signals after denoising and signal enhancement are output.

[0058] In step S140, the target EEG signals and the user's head pose signals are input into a pre-trained intention error correction model to obtain the user's feedback intention.

[0059] This application trains an intention error correction model to fuse target EEG signals and user head pose signals for anti-misoperation analysis and processing to improve recognition accuracy. The final algorithm will combine the analysis results of the user's head movement trajectory and the feedback results of the user's EEG signals to obtain the final user feedback intention.

[0060] In step S150, the interaction content is determined according to the user's feedback intention and the user's interaction intention, and the interaction result is obtained according to the interaction content.

[0061] It is determined whether intention error correction is required according to the user's feedback intention. If so, the execution of the interaction content corresponding to the user's interaction intention is revoked; if not, the interaction content corresponding to the user's interaction intention is executed, and finally the interaction result is obtained, as Figure 2 shown.

[0062] This application uses the trained intent recognition model to fuse multimodal signals, combines the intent capture ability of electroencephalogram (EEG) signals and the physical operation advantages of user input recognition for interactive intent recognition, and obtains the user's interactive intent; through noise reduction and signal enhancement processing, it effectively distinguishes the valid signals and noise signals in the EEG signals; uses the trained intent correction model to perform error correction intent recognition on the target EEG signals obtained by noise reduction and signal enhancement, and obtains the user's feedback intent; determines the interaction content based on the recognized interactive intent and feedback intent and conducts interactions, improving the recognition accuracy and response speed. Through multimodal signal fusion, this application effectively combines the intent capture ability of EEG signals and the physical operation advantages of user input recognition. Compared with a single-signal system, users can interact with the device more naturally and intuitively, reducing the problem of misoperations caused by inaccurate signals.

[0063] According to some embodiments, the method further includes steps S160 - S190.

[0064] In step S160, according to the preset interactive task samples, an error correction training data set and a recognition training data set are constructed.

[0065] To train the intent recognition model and the intent correction model, an error correction training data set and a recognition training data set need to be constructed.

[0066] In the actual construction process, this application will provide several preset interactive task samples for the user to complete, and construct the error correction training data set and the recognition training data set during the process of the user completing the preset interactive task samples.

[0067] According to the exemplary embodiments, the preset interactive task samples are simple interactive tasks, such as

[0068] In step S170, an intent recognition model is trained based on the recognition training data set, and an intent correction model is trained based on the error correction training data set.

[0069] The intent recognition model is trained using the recognition training data set, and the intent correction model is trained using the error correction training data set.

[0070] In step S180, noisy EEG signal samples and noise-free EEG signal samples of the user for the same stimulus source are collected, the noisy EEG signal samples are denoised to obtain denoised EEG signal samples, and a noise training data set is constructed based on the denoised EEG signal samples and the noise-free EEG signal samples.

[0071] In the EEG signal enhancement task, a noise training data set is constructed to train the EEG signal enhancement model.

[0072] In the process of constructing the noisy training data set, noisy electroencephalogram (EEG) signal samples and noise-free EEG signal samples of the same stimulus source from a user are collected. Among them, according to the exemplary embodiment, the noisy EEG signal samples can be collected using a dry electrode / semi-dry electrode EEG instrument, and the noise-free EEG signal samples can be collected using a wet electrode EEG instrument.

[0073] Based on the above embodiment, further, in order to collect different sample data of the same stimulus source from the user, the user is required to wear an easily wearable dry electrode / semi-dry electrode EEG instrument and a not easily wearable wet electrode EEG instrument simultaneously, so as to collect noisy EEG signal samples and noise-free EEG signal samples respectively.

[0074] The noisy EEG signal samples are denoised to obtain denoised EEG signal samples. A noisy training data set is constructed based on the denoised EEG signal samples and the noise-free EEG signal samples.

[0075] The embodiments of the present application do not limit the specific denoising method. According to the exemplary embodiment, the noisy EEG signal samples can be preliminarily denoised by using the empirical mode decomposition and reconstruction algorithm in cooperation with a filter. Then, the inertial signals of the user during the collection process are used to model the noise signals. Finally, an adaptive filter is used to further denoise the sample data. The specific implementation steps can refer to the subsequent embodiments.

[0076] In step S190, an EEG signal enhancement model is trained based on the noisy training data set.

[0077] The denoised EEG signal samples and the noise-free EEG signal samples obtained in step S180 are used to train the EEG signal enhancement model to perform a signal enhancement task.

[0078] Specifically, the EEG signal enhancement model includes an autoencoder and a decoder. During the training process, the relatively clean data (i.e., the noise-free EEG signal samples) collected by the wet electrode EEG instrument will be used as the training labels of the decoder. And the EEG signals collected by the dry electrode / semi-dry electrode EEG instrument and denoised (i.e., the denoised EEG signal samples) will be used as the input data of the autoencoder. Finally, through the trained autoencoder and decoder, the denoised signals can be enhanced, providing available data for subsequent intention error correction and modification.

[0079] According to some embodiments, referring to Figure 3 , in step S130, the user's EEG signals are denoised and signal-enhanced according to the user's head pose signals to obtain target EEG signals, which can be specifically implemented through steps S310 - S340.

[0080] In step S310, the empirical mode decomposition algorithm is used to decompose the user's EEG signal, obtaining multiple decomposed signals in different frequency bands, and removing the decomposed signals in a preset frequency band to obtain an initial noise-reduced EEG signal.

[0081] Specifically, the empirical mode decomposition and reconstruction algorithm and a filter are used to preliminarily reduce the noise of the EEG signal containing noise. In the actual operation process, first, the empirical mode decomposition algorithm is used to decompose the user's EEG signal containing noise. The empirical mode decomposition algorithm will further divide each frequency band of the EEG signal from low frequency to high frequency into decomposed signals in several frequency bands. Thus, noise reduction processing can be performed according to different types of noise. In this step, the decomposed signals in the preset frequency band can be eliminated.

[0082] According to the exemplary embodiment, the decomposed signal in the preset frequency band can be the 50 / 60 Hz power line noise.

[0083] After elimination, the empirical mode reconstruction algorithm is used to splice the decomposed signals to obtain an initial noise-reduced EEG signal.

[0084] In step S320, a motion artifact is obtained by simulating a noise signal based on the user's head pose signal and the user's EEG signal.

[0085] In this step, the user's EEG signal and inertial signal are used to model the noise signal. In this stage, the user's EEG signal and the user's head pose signal collected by the inertial sensor are used to simulate the common noise signals in the electroencephalogram to obtain motion artifacts.

[0086] In order to determine the characteristics of the common noise signals, during the process of constructing the noise training dataset in step S180, relevant noise acquisition instruments were also worn, and noise signals and electroencephalogram signals were collected simultaneously. And through the collected noise signals, research was carried out on how to model and eliminate the noise.

[0087] During the noise simulation, in some embodiments, the acceleration and angular velocity changes during head movement can be obtained according to the user's head pose signal, and the muscle activity noise can be simulated based on the acceleration and angular velocity changes. In some embodiments, the head pose changes can be compared with the time series of the user's EEG signal to find the changes in the user's EEG signal caused by head movement, such as the artifacts generated by electrode displacement. Further, if the user's head pose signal contains environmental noise (such as vibration), it can be used as a noise source.

[0088] The above-simulated noise is superimposed on the clean user's EEG signal according to a certain ratio to obtain a motion artifact.

[0089] According to the exemplary embodiment, the motion artifact can be a head muscle movement artifact, an eye movement artifact, and a head movement artifact.

[0090] In step S330, the initial noise-reduced EEG signal is filtered using motion artifacts to obtain a noise reduction result.

[0091] The EEG signal is further denoised using an adaptive filter. According to the exemplary embodiment, the initial noise-reduced EEG signal output in step S310 is used as the primary input of a least mean square (LMS) adaptive filter, and the motion artifacts output in step S320 are used as the reference input of the LMS filter. Finally, a clean EEG signal is output as the noise reduction result.

[0092] In step S340, the noise reduction result is input into an EEG signal enhancement model to output a target EEG signal.

[0093] In this step, the denoised signal is enhanced. Since the noise reduction algorithm does not perfectly remove all the noise in the signal and retain all the EEG signals. Instead, the EEG signal after the noise reduction stage will lose some EEG information. Therefore, it is necessary to reconstruct and enhance the existing EEG signal.

[0094] This application uses a trained EEG signal enhancement model to enhance the noise reduction result, and the model finally outputs a target EEG signal.

[0095] This application can effectively distinguish the valid signals and noise signals in the EEG signal, extract the valid components of the EEG signal and apply them.

[0096] According to some embodiments, referring to Figure 4 , in step S160, an error correction training data set is constructed according to a preset interaction task sample, which can be specifically implemented through steps S410 - S450.

[0097] In step S410, in response to the user executing an error correction task sample in the preset interaction task sample, an interaction recognition result output by a preset interaction result selection algorithm or an interaction result recognition model is obtained.

[0098] When the user executes an error correction task sample in the preset interaction task sample, the interaction result selection algorithm or the interaction result recognition model will output an interaction recognition result according to the user's execution situation.

[0099] Specifically, the error correction task samples can be gesture interaction task samples, voice interaction task samples, or visual stimulus interaction task samples. For gesture interaction task samples or voice interaction task samples, the interaction result recognition model is the corresponding gesture action recognition deep learning model or voice recognition deep learning model, which will return the user's interaction intention as the interaction recognition result by recognizing the user's gesture actions or voice inputs.

[0100] For visual stimulus interaction task samples, in some embodiments, a high refresh rate display is used to draw high-frequency visual stimuli, and multiple interaction instructions are encoded by a frequency-phase joint method to implement a multi-instruction interaction interface with a weak flicker sensation and high comfort, and the multi-instruction interaction interface is used to display visual stimulus interaction task samples.

[0101] In addition, the SSVEP-BCI technology is applied in combination with an interaction result selection algorithm to enable the user to select different interaction options on the screen. Among them, the SSVEP-BCI (Steady-State Visual Evoked Potential-Brain-Computer Interface) technology is a brain-computer interface system that utilizes the brain's response to visual stimuli. In this technology, the user generates specific electroencephalogram signals by gazing at one or more flickering visual stimuli, and these signals are called Steady-State Visual Evoked Potentials (SSVEP for short).

[0102] On the interaction screen, different options can generate different electroencephalogram frequency stimuli for the user. By analyzing the differences in frequencies through the interaction result selection algorithm, the degree of the user's attention to each interaction intention can be further analyzed, enabling the user to make a quick and seamless selection and obtain the interaction recognition result.

[0103] It should be noted that the interaction result selection algorithm is actually a classification model task, and the SSVEP signals can be modeled using trained and untrained methods. The former can achieve a faster and more accurate interaction method through short-term training, and the latter can be used without training and can be used immediately after wearing, thus adapting to the actual needs of different scenarios. Among them, the trained method refers to a deep learning training model, and the untrained method refers to a mathematical model.

[0104] In step S420, user feedback is collected, and the first electroencephalogram signal sample and head pose signal sample generated by the user in response to the interaction recognition result are recorded. Among them, the user feedback is the user's satisfaction with the interaction recognition result.

[0105] After the user executes the preset interaction task sample and obtains the interaction recognition result, it is also necessary to collect the user's explicit feedback on the interaction recognition result, including satisfaction or dissatisfaction, which is recorded as user feedback.

[0106] While collecting user feedback, record the implicit feedback of the user on the interaction recognition result, including electroencephalogram signals (denoted as the first electroencephalogram signal sample) and head inertial attitude data (denoted as the head attitude signal sample).

[0107] In step S430, in the case where the user feedback is dissatisfied, record the interaction intention option selected by the user, and obtain the current electroencephalogram signal as the second electroencephalogram signal sample.

[0108] After the user gives feedback on dissatisfaction, the interaction result selection algorithm or the interaction result recognition model will provide several other interaction intention options for the user to choose, and record the interaction intention option selected by the user and the current electroencephalogram signal data (denoted as the second electroencephalogram signal sample).

[0109] During the user's selection process, make a selection through active input, such as inputting a voice signal or a gesture action for selection.

[0110] In step S440, update the first electroencephalogram signal sample according to the second electroencephalogram signal sample, and update the interaction recognition result according to the interaction intention option selected by the user.

[0111] In step S450, take the first electroencephalogram signal sample and the corresponding head attitude signal sample as the first training data, take the user feedback as the first training label of the first training data, and construct an error correction training data set according to the first training data with the first training label.

[0112] It should be emphasized that in the process of constructing the error correction training data set, in order to further improve the accuracy of the model, for a specific application scenario, design multi-type task paradigms to construct preset interaction task samples, such as different task difficulties, task scenarios, etc., to induce different types of error-related potential components, and synchronously collect behavior data related to task performance to better understand the cognitive mechanism of errors.

[0113] Furthermore, based on the error correction training data set, the following research tasks can also be completed to better model and classify electroencephalogram signals: (1) Observe the time-domain waveform to better distinguish and understand the error-related potential components related to different cognitive processes. (2) Use frequency analysis methods to capture the frequency-domain features in the error-related potential. (3) Plot the electroencephalogram topographic map to understand the activation pattern of the brain at different frequencies. (4) Establish an individual difference model to consider the variation of error-related potentials among different individual characteristics (such as age, gender, cognitive ability).

[0114] According to some embodiments, in step S160, construct an identification training data set according to the preset interaction task sample, which can be specifically implemented through steps S460 - S490.

[0115] In step S460, in response to the user performing the EEG recognition task sample in the preset interaction task sample, an interactive EEG signal sample of the user is collected.

[0116] When the user performs the EEG recognition task sample in the preset interaction task sample, the EEG signal of the user within a preset time range is collected and recorded as the interactive EEG signal sample.

[0117] According to the exemplary embodiment, the preset time range is 4 seconds.

[0118] In addition, the EEG recognition task sample is a visual stimulus interaction task sample. For specific content, refer to the above embodiments, and details are not described herein again in this application.

[0119] In step S470, in response to the user performing the active recognition task sample in the preset interaction task sample, a user input signal sample of the user is collected, where the user input signal sample is a voice signal sample or a gesture action signal sample.

[0120] When the user performs the active recognition task sample in the preset interaction task sample, the active input of the user within a preset time range is collected and recorded as the user input signal sample.

[0121] The active recognition task sample can be a gesture interaction task sample or a voice interaction task sample. For specific content, refer to the above embodiments, and details are not described herein again in this application.

[0122] In step S480, the interactive EEG signal sample is used as the second training data, and the EEG recognition task sample is used as the second training label of the second training data. An EEG recognition training dataset is constructed according to the second training data with the second training label.

[0123] In step S490, the user input signal sample is used as the third training data, and the active recognition task sample is used as the third training label of the third training data. An action recognition training dataset is constructed according to the third training data with the third training label; among them, the EEG recognition training dataset and the action recognition training dataset constitute the recognition training dataset.

[0124] It should be explained that the EEG recognition training dataset is used to train the EEG intention recognition model, and the action recognition training dataset is used to train the action intention recognition model.

[0125] According to some embodiments, in step S120, the user input signal and the user EEG signal are input into a pre-trained intention recognition model to obtain the user interaction intention, which can be specifically realized through steps S121 - S123.

[0126] In step S121, the user input signal is input into the action intention recognition model to obtain a first recognition result.

[0127] In step S122, the user's electroencephalogram (EEG) signal is input into the EEG intention recognition model to obtain a second recognition result.

[0128] In step S123, based on the first recognition result and the second recognition result, as well as the preset first weight and second weight, the user interaction intention is calculated; wherein, the EEG intention recognition model is trained based on an EEG recognition training dataset, and the action intention recognition model is trained based on an action recognition training dataset.

[0129] During the application process, the EEG intention recognition model and the action intention recognition model respectively recognize and predict the user's EEG signal and the user input signal input by the user, and respectively output the second recognition result and the first recognition result. Based on the first recognition result and the second recognition result, weighted calculation is performed using preset rules (including the weights of the prediction results of the action intention recognition model and the EEG intention recognition model, that is, the first weight and the second weight) to obtain the user interaction intention.

[0130] In addition, during the hardware integration and R & D, the outputs of the signals of all wearable devices are connected to the server and synchronized in the time axis. The synchronized gesture actions or voice inputs will be used as input data and input into the intelligent interaction system to complete various tasks in the system.

[0131] According to the exemplary embodiment, the entire interaction system is implemented using Unreal Engine 5 or other open-source software libraries for development.

[0132] According to some embodiments, referring to Figure 5 , in step S170, an intention error correction model is trained based on an error correction training dataset, which can be specifically implemented through steps S510 - S550.

[0133] In step S510, the first EEG signal samples in the error correction training dataset are preprocessed and feature extraction is performed to obtain EEG signal feature samples.

[0134] In the research of error correction algorithms, a multi-modal data fusion method is proposed to improve the accuracy and robustness of the algorithms.

[0135] During the actual operation process, before classification, the first EEG signal samples in the error correction training dataset are preprocessed to reduce the feature dimension and improve the calculation speed, and then feature extraction is performed to obtain EEG signal feature samples.

[0136] According to the exemplary embodiment, EEG data within 200 ms before the start to 1200 ms after the start of each trial is extracted using the event channel in the data, and electrooculogram artifacts are removed using the ICA algorithm of the Python-MNE toolkit. After that, the data is downsampled to 250 Hz and filtered through a Chebyshev type-I bandpass filter (passband range: 2 - 20 Hz). Finally, the effective lead EEG data is concatenated to obtain single-channel EEG features for classification.

[0137] In step S520, an error judgment model is trained based on the EEG signal feature samples and their corresponding first training labels.

[0138] During the model training process, the support vector machine algorithm (SVM) is used to simulate and detect the offline experimental data to verify the feasibility of error correction based on EEG signals.

[0139] Specifically, determining whether the error-related potential component exists can be defined as a binary classification problem (target and non-target), and the SVM method realizes binary classification by calculating the hyperplane that maximizes the distance between the two classes.

[0140] According to the exemplary embodiment, the preprocessed data (i.e., EEG signal feature samples) is subjected to 5-fold cross-validation using the SVM-Gaussian kernel function, and classification performance evaluation indicators such as the SVM binary classification accuracy, true positive rate, and false positive rate are calculated until the indicators meet the preset conditions, that is, the training result is output as the error judgment model.

[0141] In step S530, the target EEG signal feature sample is input into the error judgment model to obtain a judgment result, where the target EEG signal feature sample is one of the EEG signal feature samples.

[0142] The trained error judgment model is used to judge the electroencephalogram signal to obtain a judgment result.

[0143] In step S540, the target head pose signal sample is input into the pre-trained head motion recognition model to obtain a head motion recognition result, where the target head pose signal sample is the head pose signal sample corresponding to the first EEG signal sample associated with the target EEG signal feature sample.

[0144] The head pose signal sample corresponding to the first EEG signal sample associated with the target EEG signal feature sample is used for anti-misoperation analysis and processing. Specifically, the head motion recognition model obtained through deep learning judges the user's nodding motion, shaking motion, and no motion to obtain the head motion recognition result.

[0145] It should be noted that during the acquisition of head pose signal samples, the sensitivity of the inertial sensor can be as low as 1 degree. The inertial sensor can capture the acceleration and angular velocity of the user's head movement when seeing the results given by the interaction system. By inputting the head acceleration and angular velocity movement information within a certain period of time into a trained long short-term memory model or a one-dimensional convolutional model, the model can determine whether the user is nodding, shaking their head, or making no movement.

[0146] In step S550, according to the judgment result and the head movement recognition result, the error judgment model is optimized to obtain an intention error correction model.

[0147] Finally, by combining the judgment result obtained from the electroencephalogram signal in the above text, the error correction algorithm can output the most accurate result.

[0148] This application can more effectively filter environmental noise and electromyogram interference, thereby ensuring the response speed and accuracy of the system.

[0149] In addition, due to the differences in brain structures and functions among different people, it is necessary to select model parameters optimally to adapt to the brain patterns of different users and improve the generalization ability of the classification model interaction system.

[0150] Furthermore, it is also possible to train a proprietary model for different users and customize the optimal SVM model parameters for each subject. Traverse the classification performance under different lead combinations, different window lengths, and different Gaussian kernel parameters, and customize the optimal parameter combination for each subject according to a certain logic.

[0151] According to some embodiments, referring to Figure 6 , in step S410, obtain the interaction recognition result output by the preset interaction result selection algorithm, which can be specifically implemented through steps S610 - S630.

[0152] In step S610, when the user executes the preset interaction task sample, collect the target test electroencephalogram signal of the user for the target interaction option, where the target interaction option is one of the interaction intention options.

[0153] This embodiment further describes the non-training method of the interaction result selection algorithm. The non-training method of the interaction result selection algorithm mainly includes an intelligent interaction result selection algorithm based on task-related component analysis and an intelligent interaction result selection algorithm based on canonical correlation analysis.

[0154] Define the target test electroencephalogram signal as N s is the number of sampling points, N c is the number of eigenvalues.

[0155] In step S620, according to the target test EEG signal and a preset number of pre-constructed EEG signal templates, the correlation coefficient between the target test EEG signal and each EEG signal template is calculated.

[0156] According to some embodiments, step S620 is specifically implemented by step S621 or step S622.

[0157] In step S621, the target test EEG signal and a preset number of EEG signal templates are input into a pre-constructed spatial filter bank to calculate the correlation coefficient between the target test EEG signal and each EEG signal template.

[0158] For the intelligent interaction result selection algorithm based on task-related component analysis, it should be noted that task-related component analysis (TRCA) is a data-driven spatial filtering method. By maximizing the repeatability of task-related components among multiple trials, it can effectively eliminate background EEG activities, extract task-related components, and improve the signal-to-noise ratio of EEG signals. The principle of the TRCA algorithm is as follows: Assume that the single-channel EEG signal collected is composed of the linear weighting of two source signals:

[0159] x j (t) = a 1,j m(t) + a 2,j n(t), j = 1, 2, …, N c

[0160] where is the task-related signal, is the task-unrelated signal, j is the serial number of the lead, N c is the number of leads, a 1,j and a 2,j are the weighting coefficients for projecting the source signal onto the collected EEG signal, respectively.

[0161] The problem of extracting the task-related component m(t) from the multi-channel EEG signal can be expressed as:

[0162]

[0163] In an ideal case, if is to be satisfied, then and should be satisfied, but it is difficult to solve this equation in actual situations.

[0164] Therefore, this problem can be transformed into maximizing the covariance of task-related components among multiple trials to solve. Define the multi-channel EEG signal of the h-th trial as Define the task-related components extracted therefrom as where h = 1, 2, …, N t , N t is the total number of trials. Then the covariance between the h 1 -th trial and the h 2 -th trial can be defined as:

[0165]

[0166] Define the matrix where:

[0167]

[0168] Then the sum of the covariances between any two different trials can be expressed as:

[0169]

[0170] To obtain a finite solution, constrain the variance of to be:

[0171]

[0172] In summary, the constrained optimization problem can be solved as:

[0173]

[0174] For a matrix c ×N c of dimension N perform eigenvalue decomposition, and N c eigenvalues and eigenvectors can be calculated. The magnitude of the eigenvalue λ is the cost function value of the corresponding eigenvector , representing the consistency of the task-related components among multiple trials. Sort the eigenvalues in descending order, and use the eigenvector corresponding to the largest eigenvalue as the optimized lead weighting coefficient, i.e., the TRCA spatial filter.

[0175] Project the target test EEG signal and the k-th target EEG signal template respectively through the spatial filter bank W, and calculate the Pearson correlation coefficient between the projected signals:

[0176]

[0177] The visual evoked potential classification algorithm based on TRCA has a wide range of applications and can identify EEG signals induced by various stimulus coding methods such as frequency coding and phase coding.

[0178] In step S622, according to the target test EEG signal and a preset number of EEG signal templates, the first preset formula is solved to obtain the correlation coefficients between the target EEG signal and each EEG signal template.

[0179] Regarding the intelligent interaction result selection algorithm based on canonical correlation analysis, it should be noted that the essence of canonical correlation analysis (CCA) is to extract the narrowband frequency components of SSVEP from EEG signals, and it is a commonly used algorithm in SSVEP-BCI detection based on frequency-coded visual stimuli. It is a multivariate statistical method that extends ordinary correlation to two sets of canonical correlation variables and maximizes the correlation between them by finding a pair of linear combination methods for the two sets of variables.

[0180] Suppose there are weighting coefficients W X and W Y , which are used to linearly combine the template signal X k and the EEG signal Y respectively, to obtain x k =X k T W X and y = Y T W Y . The CCA algorithm solves the first preset formula to obtain the maximum canonical correlation coefficient r k between the template signal X k with the stimulation frequency of f k and the EEG signal Y, and its corresponding weighting coefficients W X and W Y .

[0181] In step S630, the interaction intention option corresponding to the EEG signal template with the largest correlation coefficient is selected as the interaction recognition result.

[0182] For the intelligent interaction result selection algorithm based on task-related component analysis, the correlation coefficients between the test signal and each target template are calculated respectively, and the target corresponding to the maximum correlation coefficient is the recognition result of the test signal:

[0183] T = arg k max r k , k = 1, 2,..., N f

[0184] For the intelligent interaction result selection algorithm based on canonical correlation analysis, the correlation coefficients between the test EEG signal and each stimulation frequency template signal are calculated in turn, and the stimulation frequency with the maximum correlation is considered to be the frequency of the test EEG signal:

[0185] C = arg k max rk , k = 1, 2, …, N f 。

[0186] According to some embodiments, the first preset formula is:

[0187]

[0188] where E represents expectation; X k represents the k-th electroencephalogram (EEG) signal template, Y represents the target test EEG signal, r k represents the correlation coefficient between the EEG signal template X k with a stimulation frequency of f k and the EEG signal Y, x k represents the first intermediate variable, x k = X k T W X , y represents the second intermediate variable, y = Y T W Y , W X represents the first weighting coefficient, and W Y represents the second weighting coefficient.

[0189] Furthermore, step S620 can also be implemented by the filter bank algorithm. It should be noted that the filter bank algorithm utilizes the independent information of EEG signals at the stimulation frequency and harmonic frequencies, effectively improving the detection accuracy of SSVEP. The CCA algorithm and TRCA algorithm using the filter bank are called the filter bank canonical correlation analysis algorithm (FBCCA) and the filter bank task-related component analysis algorithm (FBTRCA).

[0190] According to the exemplary embodiments, the filter bank algorithm mainly includes four steps: (1) Extract sub-band components. The test EEG signal Y is passed through N different Chebyshev type I band-pass filters to decompose N sub-band components where n = 1, 2, …, N. In the design of sub-bands in the filter bank, a scheme with different starting frequencies and fixed cut-off frequencies is referred to. Therefore, each sub-band contains multiple harmonic frequency components. The starting frequency is usually set according to the stimulation frequency and its harmonic frequencies, and can also be adjusted and optimized according to actual situations. (2) TRCA or CCA analysis. For the k-th target, the correlation coefficients between the test signal and the target template are calculated on each sub-band component It should be noted specifically that the EEG signal template and spatial filter in the TRCA algorithm need to be calculated separately on each sub-band, while the template signal of the CCA algorithm is uniformly a group of sine and cosine signals containing each harmonic frequency. (3) Sub-band information integration. For the k-th target, the correlation coefficients corresponding to each sub-band component are summed according to the following formula:

[0191]

[0192] Among them, w(n) is the weight coefficient of the nth sub-band, defined as:

[0193] w(n) = n -a +b;

[0194] In the formula, both a and b are constants. In a specific embodiment, their value ranges are within [0:0.25:2] and [0:0.25:1] respectively. Based on the offline experimental data, this embodiment uses the grid search method to find the corresponding filter bank parameter values (N, a, and b) when the system performance is optimal.

[0195] After that, target recognition can be performed, that is, calculate the correlation coefficients between the test signal and each target respectively Determine the target corresponding to the maximum correlation coefficient as the label of the test data:

[0196]

[0197] According to some embodiments, referring to Figure 7 , the construction of the spatial filter bank can be specifically implemented through step S710 - step S720.

[0198] In step S710, multi-trial multi-channel EEG signals are obtained based on the target interaction options.

[0199] In this step, the preprocessed multi-channel EEG signal corresponding to the kth interaction intention option is defined as where k = 1, 2,..., N f , N f is the number of classifiable targets, and N s is the number of sampling points.

[0200] In step S720, using the TRCA algorithm, based on the multi-channel EEG signals, the spatial filters for the target interaction options are calculated, and then the spatial filter bank is obtained.

[0201] In this step, the spatial filter for the kth target (i.e., the target interaction option) is calculated according to the TRCA algorithm and the spatial filter bank is constituted thereby

[0202] According to some embodiments, referring to Figure 8 , the construction of the EEG signal template can be specifically implemented through step S810 or step S820.

[0203] In step S810, the multi-trial single-channel EEG signals of the target interaction option are superimposed and averaged to obtain an EEG signal template.

[0204] In step S820, according to the single-trial EEG signals of the target interaction option, an EEG signal template with the same data length as the single-trial EEG signals and containing the stimulation frequency and harmonic frequencies is constructed according to a second preset formula, where the stimulation frequency is the frequency at which the target interaction option visually stimulates the user.

[0205] That is to say, the single-trial multi-channel EEG signals are defined as And for each stimulation frequency f k (k = 1, 2, …, N f , N f being the number of frequency encoding targets), a sine-cosine signal template X k (t) with the same data length as the EEG signals and containing the stimulation frequency and harmonic frequencies is constructed according to the second preset formula.

[0206] According to some embodiments, the second preset formula is:

[0207]

[0208] where X k (t) represents the k-th EEG signal template, f k represents the k-th stimulation frequency, k = 1, 2, …, N f , N f being the number of frequency encoding targets, N h represents the harmonic order, and t represents time.

[0209] This application can effectively distinguish the valid signals and noise signals in the EEG signals, and combine with the user input signals to achieve accurate and real-time interaction control. It effectively solves the main defects of the existing EEG signal and gesture recognition system in terms of signal accuracy, real-time response, and user experience. It not only improves the reliability and accuracy of the interaction system, but also provides a new direction for the development of brain-computer interface and gesture recognition technology.

[0210] The following describes the device embodiments of the present application, which can be used to execute the method embodiments of the present application. For the details not disclosed in the device embodiments of the present application, reference can be made to the method embodiments of the present application.

[0211] Figure 9 The block diagram of an EEG-action fusion interaction system according to an exemplary embodiment is shown.

[0212] Figure 9 The device shown can execute the aforementioned EEG-action fusion interaction method according to the embodiments of the present application.

[0213] As Figure 9 shown, the EEG-action fusion interaction system may include:

[0214] Referring to Figure 9 and the previous description, the signal acquisition module 910 is configured to acquire user input signals, user EEG signals, and user head pose signals, where the user input signal is a voice signal or a gesture signal.

[0215] The intention recognition module 920 is configured to input the user input signal and the user EEG signal into a pre-trained intention recognition model to obtain the user interaction intention.

[0216] The noise reduction and enhancement module 930 is configured to perform noise reduction and signal enhancement processing on the user EEG signal according to the user head pose signal to obtain a target EEG signal.

[0217] The intention error correction module 940 is configured to input the target EEG signal and the user head pose signal into a pre-trained intention error correction model to obtain the user feedback intention.

[0218] The interaction result module 950 is configured to determine the interaction content according to the user feedback intention and the user interaction intention, and obtain the interaction result according to the interaction content.

[0219] The device performs functions similar to those of the method provided above. Other functions can be referred to the previous description and will not be elaborated here.

[0220] Figure 10 An electronic device according to an exemplary embodiment of the present application is shown. The following refers to Figure 10 to describe the electronic device 1000 according to this embodiment of the present application. Figure 9 The shown electronic device 1000 is merely an example and should not impose any limitation on the functions and usage scope of the embodiments of the present application.

[0221] As Figure 10 shown, the electronic device 1000 is presented in the form of a general computing device. The components of the electronic device 1000 may include but are not limited to: at least one processing unit 1010, at least one storage unit 1020, a bus 1030 connecting different system components (including the storage unit 1020 and the processing unit 1010), a display unit 1040, etc.

[0222] Among them, the storage unit stores program codes, and the program codes can be executed by the processing unit 1010, so that the processing unit 1010 executes the methods according to various exemplary embodiments of the present application described in this specification. For example, the processing unit 1010 can execute the method as described above.

[0223] The storage unit 1020 may include a readable medium in the form of a volatile storage unit, such as a random access memory (RAM) 10201 and / or a cache storage unit 10202, and may further include a read-only memory (ROM) 10203.

[0224] The storage unit 1020 may also include a program / utilities 10204 having a set (at least one) of program modules 10205. Such program modules 10205 include, but are not limited to: an operating system, one or more application programs, other program modules, and program data. Each or some combination of these examples may include an implementation of a network environment.

[0225] The bus 1030 may represent one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus structures.

[0226] The electronic device 1000 may also communicate with one or more external devices 300 (such as a keyboard, a pointing device, a Bluetooth device, etc.), may also communicate with one or more devices that enable a user to interact with the electronic device 1000, and / or may communicate with any device that enables the electronic device 1000 to communicate with one or more other computing devices (such as a router, a modem, etc.). Such communication may be through an input / output (I / O) interface 1050. Also, the electronic device 1000 may communicate with one or more networks (such as a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) through a network adapter 1060. The network adapter 1060 may communicate with other modules of the electronic device 1000 through the bus 1030. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the electronic device 1000, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems, etc.

[0227] Through the description of the above embodiments, those skilled in the art can easily understand that the example embodiments described herein can be implemented by software, or can be implemented by a combination of software and necessary hardware. The technical solutions according to the embodiments of the present application can be embodied in the form of a software product. The software product can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, including several instructions to enable a computing device (which can be a personal computer, a server, or a network device, etc.) to execute the above methods according to the embodiments of the present application.

[0228] A software product may employ any combination of one or more readable media. The readable media can be a readable signal medium or a readable storage medium. The readable storage medium can be, for example but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the foregoing. More specific examples (a non-exhaustive list) of the readable storage medium include: an electrical connection having one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0229] A computer-readable storage medium may include a data signal propagated in a baseband or as part of a carrier wave, in which case the data signal carries the readable program code. Such a propagated data signal may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the foregoing. The readable storage medium may also be any readable medium other than a readable storage medium, which can send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device. The program code contained on the readable storage medium may be transmitted using any appropriate medium, including but not limited to wireless, wired, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0230] The program code for performing the operations of this application may be written in any combination of one or more programming languages, including object-oriented programming languages such as Java, C++, etc., and also including conventional procedural programming languages such as the "C" language or similar programming languages. The program code may be executed entirely on the user's computing device, partially on the user's device, executed as a stand-alone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server. In the case of a remote computing device, the remote computing device may be connected to the user's computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., through the Internet using an Internet service provider).

[0231] The foregoing computer-readable medium bears one or more programs, and when the one or more programs are executed by a device, the computer-readable medium implements the foregoing functions.

[0232] Those skilled in the art can understand that the above-mentioned modules can be distributed in the device according to the description of the embodiments, or can be correspondingly changed and distributed in one or more devices that are only different from this embodiment. The modules of the above embodiments can be combined into one module, or can be further split into multiple sub-modules.

[0233] Through the description of the above embodiments, those skilled in the art can easily understand that the exemplary embodiments described herein can be implemented by software, or can be implemented by the way of software combined with necessary hardware. Therefore, the technical solution according to the embodiments of the present application can be embodied in the form of a software product, and the software product can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on the network, including several instructions to enable a computing device (which can be a personal computer, a server, a mobile terminal, or a network device, etc.) to execute the method according to the embodiments of the present application.

[0234] The above specifically shows and describes the exemplary embodiments of the present application. It should be understood that the present application is not limited to the detailed structure, setting method or implementation method described herein; on the contrary, the present application is intended to cover various modifications and equivalent settings included in the spirit and scope of the appended claims.

Claims

1. A method for brain wave action fusion interaction, characterized in that: include: Acquire a user input signal, a user electroencephalogram signal, and a user head posture signal, wherein the user input signal is a voice signal or a gesture action signal; Inputting the user input signal and the user electroencephalogram signal into a pre-trained intention recognition model to obtain the user interaction intention; Performing noise reduction and signal enhancement processing on the user's EEG signal according to the user's head posture signal to obtain a target EEG signal; Inputting the target EEG signal and the user's head posture signal into a pre-trained intention error correction model to obtain user feedback intention; The interaction content is determined according to the user feedback intention and the user interaction intention, and the interaction result is obtained according to the interaction content.

2. The method according to claim 1, characterized in that: The method further comprises: According to the preset interactive task samples, construct error correction training data set and recognition training data set; The intention recognition model is obtained by training based on the recognition training data set, and the intention error correction model is obtained by training based on the error correction training data set; Collecting noisy EEG signal samples and noise-free EEG signal samples of the user in response to the same stimulus source, denoising the noisy EEG signal samples to obtain noise-reduced EEG signal samples, and constructing a noise training data set based on the noise-reduced EEG signal samples and the noise-free EEG signal samples; The EEG signal enhancement model is trained based on the noise training data set.

3. The method according to claim 2, characterized in that The performing noise reduction and signal enhancement processing on the user's EEG signal according to the user's head posture signal to obtain a target EEG signal includes: Decomposing the user's EEG signal using an empirical mode decomposition algorithm to obtain a plurality of decomposed signals in different frequency bands, and removing the decomposed signal in a preset frequency band to obtain an initial noise-reduced EEG signal; Simulating a noise signal according to the user's head posture signal and the user's electroencephalogram signal to obtain a motion artifact; Filtering the initial denoised EEG signal using the motion artifact to obtain a denoised result; The denoising result is input into the EEG signal enhancement model, and the target EEG signal is output.

4. The method according to claim 2, characterized in that: According to the preset interactive task samples, construct the error correction training data set, including: In response to the user executing an error correction task sample in the preset interaction task sample, obtaining an interaction recognition result output by a preset interaction result selection algorithm or an interaction result recognition model; Collecting user feedback, and recording the first EEG signal sample and the head posture signal sample generated by the user in response to the interactive recognition result, wherein the user feedback is the user's satisfaction with the interactive recognition result; In the case where the user feedback is unsatisfactory, recording the interaction intention option selected by the user, and acquiring the current EEG signal as the second EEG signal sample; updating the first EEG signal sample according to the second EEG signal sample, and updating the interaction recognition result according to the interaction intention option selected by the user; The first EEG signal sample and the corresponding head posture signal sample are used as first training data, the user feedback is used as a first training label of the first training data, and an error correction training data set is constructed according to the first training data with the first training label.

5. The method according to claim 2, characterized in that: According to the preset interactive task samples, a recognition training dataset is constructed, including: In response to the user performing an EEG recognition task sample in a preset interactive task sample, collecting an interactive EEG signal sample of the user; In response to the user performing an active recognition task sample in a preset interactive task sample, collecting a user input signal sample of the user, wherein the user input signal sample is a voice signal sample or a gesture action signal sample; Using the interactive EEG signal sample as second training data, using the EEG recognition task sample as a second training label for the second training data, and constructing an EEG recognition training data set according to the second training data with the second training label; Using the user input signal sample as third training data, using the active recognition task sample as a third training label for the third training data, and constructing an action recognition training data set according to the third training data with the third training label; Among them, the EEG recognition training data set and the action recognition training data set constitute the recognition training data set.

6. The method according to claim 4, characterized in that The intention error correction model is obtained by training based on the error correction training data set, including: Preprocessing the first EEG signal sample in the error correction training data set and performing feature extraction to obtain an EEG signal feature sample; Training an error judgment model based on the EEG signal feature samples and their corresponding first training labels; Inputting a target EEG signal feature sample into the error judgment model to obtain a judgment result, wherein the target EEG signal feature sample is one of the EEG signal feature samples; Inputting a target head posture signal sample into a pre-trained head action recognition model to obtain a head action recognition result, wherein the target head posture signal sample is a head posture signal sample corresponding to a first EEG signal sample associated with the target EEG signal feature sample; According to the judgment result and the head movement recognition result, the error judgment model is optimized to obtain an intention error correction model.

7. The method according to claim 4, characterized in that Obtain the interactive recognition results output by the preset interactive result selection algorithm, including: When the user performs the preset interactive task sample, collecting the user's target test EEG signal for the target interactive option, wherein the target interactive option is one of the interactive intention options; According to the target test EEG signal and a preset number of pre-constructed EEG signal templates, calculating the correlation coefficient between the target test EEG signal and each of the EEG signal templates; The interaction intention option corresponding to the EEG signal template with the largest correlation coefficient is selected as the interaction recognition result.

8. The method according to claim 5, characterized in that Inputting the user input signal and the user EEG signal into a pre-trained intention recognition model to obtain the user interaction intention, including: Inputting the user input signal into an action intention recognition model to obtain a first recognition result; Inputting the user's EEG signal into an EEG intention recognition model to obtain a second recognition result; Calculating the user interaction intention according to the first recognition result and the second recognition result, and the preset first weight and second weight; Among them, the EEG intention recognition model is obtained by training based on the EEG recognition training data set, and the action intention recognition model is obtained by training based on the action recognition training data set.

9. The method according to claim 7, characterized in that: The step of calculating the correlation coefficient between the target test EEG signal and each of the EEG signal templates according to the target test EEG signal and a preset number of pre-constructed EEG signal templates includes: Inputting the target test EEG signal and a preset number of the EEG signal templates into a pre-constructed spatial filter bank to calculate the correlation coefficient between the target test EEG signal and each of the EEG signal templates; or According to the target test EEG signal and a preset number of the EEG signal templates, a first preset formula is solved to obtain a correlation coefficient between the target test EEG signal and each of the EEG signal templates.

10. The method according to claim 9, characterized in that The first preset formula is: Among them, E represents expectation; X k represents the kth EEG signal template, Y represents the target test EEG signal, r k The stimulation frequency is f k EEG signal template X k Correlation coefficient with EEG signal Y, x k represents the first intermediate variable, x k =X k T W X , y represents the second intermediate variable, y = Y T W Y , W X represents the first weighting coefficient, W Y represents the second weighting coefficient.

11. The method according to claim 9, characterized in that The spatial filter bank is constructed by the following steps: Acquire multi-lead EEG signals of multiple trials based on the target interaction options; The TRCA algorithm is used to calculate the spatial filter of the target interaction option based on the multi-lead EEG signal, thereby obtaining the spatial filter group.

12. The method according to claim 7 or 9, characterized in that: The EEG signal template is constructed by the following steps: Superimposing and averaging the single-lead EEG signals of multiple trials of the target interaction option to obtain the EEG signal template; or According to the single-trial EEG signal of the target interaction option, an EEG signal template having the same data length as the single-trial EEG signal and including stimulation frequency and harmonic frequency is constructed according to a second preset formula, wherein the stimulation frequency is the frequency at which the target interaction option visually stimulates the user.

13. The method according to claim 12, characterized in that The second preset formula is: Among them, X k (t) represents the kth EEG signal template, f k represents the kth stimulation frequency, k=1,2,…,N f , N f is the frequency encoding target number, N h represents the harmonic order, and t represents the time.

14. A brain wave action fusion interaction system, characterized in that: include: A signal acquisition module, used to acquire user input signals, user EEG signals and user head posture signals, wherein the user input signals are voice signals or gesture action signals; An intention recognition module, used for inputting the user input signal and the user EEG signal into a pre-trained intention recognition model to obtain the user interaction intention; A noise reduction and enhancement module, used to perform noise reduction and signal enhancement processing on the user's EEG signal according to the user's head posture signal to obtain a target EEG signal; An intention error correction module, used for inputting the target EEG signal and the user's head posture signal into a pre-trained intention error correction model to obtain user feedback intention; The interaction result module is used to determine the interaction content according to the user feedback intention and the user interaction intention, and obtain the interaction result according to the interaction content.

15. An electronic device, characterized in that: include: one or more processors; A storage device for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 13.

16. A computer-readable storage medium having a computer program or instruction stored thereon, characterized in that: When the computer program or instruction is executed by a processor, the method according to any one of claims 1 to 13 is implemented.

Images