Data processing method and device for brain-computer interface based on object motion imagination

Abstract

The application provides a brain-computer interface data processing method and device based on object motion imagination, and relates to the field of brain-computer interface, and solves the technical problems of low universality of the existing object motion control method based on brain-computer interface. The method comprises the following steps: in response to detecting that a user has a control demand for a controlled object, acquiring multi-channel electroencephalogram signals of the user through a brain-computer interface; performing feature extraction on the multi-channel electroencephalogram signals, and decoding the imagination intention of the user for the motion of the controlled object based on the extracted features to obtain an electroencephalogram intention representation result; the imagination intention is the imagination of the user on the motion state of the controlled object; according to the electroencephalogram intention representation result, a control instruction is generated, and the controlled object in the external device is controlled to perform corresponding motion according to the control instruction.

Description

Technical Field

[0001] This application relates to the field of brain-computer interface technology, and in particular to a brain-computer interface data processing method and apparatus based on object motion imagination. Background Technology

[0002] Currently, non-invasive brain-computer interfaces (BMIs) are a technology that can acquire brain signals without surgery and convert them into control signals for external devices. Compared to invasive methods (such as implanted electrodes), non-invasive BCIs have the advantages of high safety and low risk, and are therefore more widely used. In tasks where the control target is the movement of objects, existing methods often employ stimulus-driven selective control or mapping methods mediated by limb motor imagery. This involves first decoding the visual selection intention or limb motor intention, and then mapping it to the direction of object movement. This method usually relies on the ERD / ERS changes in the motor cortex induced by limb motor imagery, which has limited applicability to individuals with large state fluctuations or damaged motor cortex, resulting in low versatility of brain-computer interface-based object movement control methods. Summary of the Invention

[0003] The purpose of this invention is to provide a brain-computer interface data processing method and device based on object motion imagination, so as to solve the technical problem of low versatility of existing brain-computer interface-based object motion control methods.

[0004] In a first aspect, this application provides a brain-computer interface data processing method based on object motion imagination, the method comprising: In response to the detection that the user has a need to control the controlled object, the multi-channel EEG signals of the user are collected through the brain-computer interface; Feature extraction is performed on the multi-channel EEG signals, and the user's imagined intention regarding the movement of the controlled object is decoded based on the extracted features to obtain the EEG intention representation result; wherein, the imagined intention is the user's imagination of the movement state of the controlled object; Based on the EEG intention representation results, control commands are generated, and the controlled object in the external device is controlled to perform corresponding movements according to the control commands.

[0005] In one possible implementation, the step of acquiring the user's multi-channel EEG signals via a brain-computer interface in response to detecting a user's need to control the controlled object includes: In response to detecting that the user has a control demand for the controlled object, motion prompt data of the controlled object is output to guide the user to directly imagine the movement of the controlled object; wherein, the motion prompt data includes at least one of the following: the direction of movement to be moved, the trajectory of movement to be moved, the mode of movement to be moved, and the target position of movement to be moved; In response to the output of the motion cue data, multi-channel EEG signals generated by the user during the direct motion visualization are acquired via a brain-computer interface.

[0006] In one possible implementation, the step of extracting features from the multi-channel EEG signals and decoding the user's imagined intention regarding the movement of a controlled object based on the extracted features to obtain an EEG intention representation result includes: Based on the specified frequency band signals in the multi-channel EEG signals, the interhemispheric difference features of the EEG signals in the parietal-occipital region are extracted; wherein, the specified frequency band signals include at least the θ frequency band signals; Based on the interhemispheric difference characteristics of the EEG signals in the parietal-occipital region, the user's intention to imagine movement toward the controlled object is decoded to obtain the EEG intention representation result.

[0007] In one possible implementation, based on the interhemispheric difference features of the EEG signals in the parietal-occipital region, the user's motor imagery intention towards the controlled object is decoded to obtain the EEG intention representation result, including: If the peak of the left hemisphere occipital electrode is earlier than the peak of the right hemisphere occipital electrode, or the trough of the left hemisphere occipital electrode is earlier than the trough of the right hemisphere occipital electrode, then the EEG intention representation result is determined to be that the controlled object moves directly to the right. If the peak of the right hemisphere occipital electrode is earlier than the peak of the left hemisphere occipital electrode, or the trough of the right hemisphere occipital electrode is earlier than the trough of the left hemisphere occipital electrode, then the EEG intention representation result is determined to be that the controlled object moves directly to the left.

[0008] In one possible implementation, based on the interhemispheric difference features of the EEG signals in the parietal-occipital region, the user's motor imagery intention towards the controlled object is decoded to obtain the EEG intention representation result, including: The feature vectors corresponding to the interhemispheric differences are input into a lightweight model. The lightweight model decodes the user's intention to directly visualize the controlled object and outputs the EEG intention representation result. The lightweight model includes a random forest model and / or a linear classifier. In one possible implementation, the interhemispheric differential features of the electroencephalogram (EEG) signals in the parieto-occipital region include at least one of the following: Cross-hemispherical channel cross-correlation delay data within a specified key time window, peak and / or trough latency difference between the left and right hemisphere channels, and inter-hemispherical phase difference or phase synchronization index data. In one possible implementation, the step of extracting features from the multi-channel EEG signals and decoding the user's imagined intention regarding the movement of a controlled object based on the extracted features to obtain an EEG intention representation result includes: The EEG segment data corresponding to the multi-channel EEG signal is input into the classification model in the form of channel multiplied by time. The classification model performs end-to-end decoding of the direction of movement of the controlled object corresponding to the user's intention to imagine movement of the controlled object, and outputs the movement direction category in the EEG intention representation result. In one possible implementation, the classification model is a convolutional neural network, which includes left and right electrode pairing difference layer, multi-branch temporal convolutional layer and classification output layer; The left and right electrode pairing difference layer is used to extract the left and right hemisphere difference features in the multi-channel EEG signal, the multi-branch temporal convolutional layer is used to extract the temporal features of the left and right hemisphere difference features at different time scales, and the classification output layer is used to output the category result corresponding to the movement direction of the controlled object.

[0009] In one possible implementation, the left and right electrode pairing differential layer is used to select a preset left and right symmetrical electrode pair from the multi-channel EEG signal, and perform differential operation on the EEG signal corresponding to each pair of left and right symmetrical electrodes to generate differential EEG features. The preset left-right symmetrical electrode pair includes at least a portion of the left-right symmetrical electrode pairs in the forehead region, central region, vertex region, and / or vertex-occipital region.

[0010] In one possible implementation, the multi-branch temporal convolutional layer includes multiple temporal convolutional branches with different kernel lengths and / or multiple dilated temporal convolutional branches with different dilation rates, for extracting multi-scale temporal dynamic features related to the user's imagined intention to move the controlled object directly.

[0011] In one possible implementation, the multi-branch temporal convolutional layer focuses on extracting temporal features related to the θ band, and / or the EEG segment data input to the classification model is EEG segment data filtered by the θ band.

[0012] In one possible implementation, generating control commands based on the EEG intention representation results, and controlling the controlled object in the external device to perform corresponding movements according to the control commands, includes: Based on the EEG intention representation results, motion control commands are generated for the controlled object in an external device other than the user; wherein the controlled object includes a virtual controlled object and / or a physical controlled object; The motion control command controls the controlled object in the external device to move according to the user's direct imagined motion intention for the controlled object, and outputs motion feedback information of the controlled object performing the corresponding motion.

[0013] Secondly, this application provides a brain-computer interface data processing device based on object motion imagination, comprising: The acquisition module is used to acquire the user's multi-channel electroencephalogram (EEG) signals through a brain-computer interface in response to the detection that the user has a control demand for the controlled object. The decoding module is used to extract features from the multi-channel EEG signals and decode the user's imagined intentions about the movement of the controlled object based on the extracted features, so as to obtain the EEG intention representation result; wherein, the imagined intention is the user's imagination of the movement state of the controlled object; The control module is used to generate control commands based on the EEG intention representation results, and to control the controlled object in the external device to perform corresponding movements according to the control commands.

[0014] Thirdly, this application also provides an electronic device, including a memory and a processor, wherein the memory stores a computer program that can run on the processor, and the processor executes the computer program to implement the method described in the first aspect above.

[0015] Fourthly, this application also provides a computer-readable storage medium storing computer-executable instructions that, when invoked and executed by a processor, cause the processor to perform the method described in the first aspect above.

[0016] This application brings the following beneficial effects: This application provides a brain-computer interface data processing method and apparatus based on object movement imagination. The method, when detecting a user's control request for a controlled object, extracts features from the multi-channel EEG signals and decodes the user's imagined intention regarding the movement of the controlled object based on the extracted features, obtaining an EEG intention representation result. The imagined intention refers to the user's imagination of the movement state of the controlled object, rather than control through the user's imagination of limb movement. Based on the EEG intention representation result, a control command is generated, and the controlled object in an external device is controlled to perform the corresponding movement according to the control command. This application employs an object movement imagination EEG paradigm centered on the movement of the controlled object, acquiring multi-channel EEG signals while the user directly imagines the controlled object moving in space. Unlike traditional indirect control methods that require imagining hand or other limb movements before mapping them to object motion, this application eliminates the need for users to imagine specific limb actions or muscle exertion processes. Instead, it allows users to directly imagine the direction, trajectory, mode of movement, and / or target location of the controlled object at the cognitive level. This induces decodeable EEG representations highly corresponding to the controlled object's movement, generating control commands accordingly. Therefore, this application implements an intuitive brain-computer interface paradigm for object motion manipulation. This paradigm aligns more closely with the natural thought process humans use in daily tasks to move, locate, and manipulate external objects, shortening the intent chain from intent generation to control output, improving the consistency between intent space and control space, and conforming more to natural manipulation habits. This contributes to enhancing the intuition and stability of online control. Furthermore, this application does not rely on users' imagination of limb movements, making it equally applicable to individuals with fluctuating states or damaged motor cortex. This improves the versatility of brain-computer interface-based object motion control and solves the technical problem of low versatility in existing brain-computer interface-based object motion control technologies.

[0017] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in the specific embodiments of this application or the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0019] Figure 1 A flowchart illustrating the brain-computer interface data processing method based on object motion imagination provided in this application embodiment; Figure 2 A schematic diagram of the motion decoding and peripheral control based on the object movement imagination paradigm in the brain-computer interface data processing method based on object movement imagination provided in the embodiments of this application; Figure 3 Another flowchart illustrating motion decoding and peripheral control based on the object movement imagination paradigm in the brain-computer interface data processing method based on object movement imagination provided in the embodiments of this application. Figure 4 A schematic diagram of the classification model structure based on left and right electrode pairing difference and multi-branch temporal convolution provided in the embodiments of this application; Figure 5 An example of brain region occlusion analysis results in the brain-computer interface data processing method based on object motion imagination provided in the embodiments of this application; Figure 6 An example of channel occlusion analysis results in the brain-computer interface data processing method based on object motion imagination provided in the embodiments of this application; Figure 7 An example of frequency band occlusion analysis results in the brain-computer interface data processing method based on object motion imagination provided in the embodiments of this application; Figure 8 An example of the temporal window occlusion analysis results in the brain-computer interface data processing method based on object motion imagination provided in the embodiments of this application; Figure 9 An example of parietal-occipital lateralization features in the brain-computer interface data processing method based on object motion imagination provided in the embodiments of this application; Figure 10 An example of time-frequency analysis results in the brain-computer interface data processing method based on object motion imagination provided in the embodiments of this application; Figure 11 An example of DeepLIFT attribution analysis results in the brain-computer interface data processing method based on object motion imagination provided in the embodiments of this application; Figure 12 A schematic diagram of the structure of a brain-computer interface data processing device based on object motion imagination provided in an embodiment of this application; Figure 13 A schematic diagram of the structure of an electronic device provided in an embodiment of this application is shown. Detailed Implementation

[0020] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0021] The terms "comprising" and "having," and any variations thereof, used in the embodiments of this application, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or units is not limited to the steps or units listed, but may optionally include other steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or devices.

[0022] Currently, common paradigms for brain-computer interfaces include steady-state visual evoked potentials (SSVEP), P300, and motor imagery (MI). SSVEP refers to the synchronized electrical activity generated in the visual cortex when a user focuses on a visual stimulus flashing at a specific frequency. P300 refers to the positive fluctuations in brain waves occurring within approximately 300 milliseconds when a user notices a rare stimulus within a series of routine stimuli. MI involves the user imagining different body parts (e.g., hands, feet, or tongue) to induce event-related desynchronization and synchronization in the cortex. Different paradigms induce different neural patterns; by decoding the acquired signals using algorithms to generate commands, control of external devices can be achieved.

[0023] However, SSVEP / P300 relies on continuous and significant visual stimuli (flickering / character matrices, etc.), and prolonged training with these stimuli can lead to visual fatigue in patients. Furthermore, this paradigm is incompatible with the body's habitual control patterns, limiting its application potential. The MI paradigm, on the other hand, indirectly maps object movements by decoding "limb movement intentions." The control chain is often "imagined hand movement → system mapping → object movement," resulting in a long intention chain and inconsistency with intuitive control intentions. Additionally, this type of method largely relies on the sensorimotor cortex μ / β rhythm (ERD / ERS), limiting its stability and applicability in individuals with fluctuating states or damaged motor cortex. Furthermore, existing task-oriented MI solutions attempt to introduce controlled objects into the scene, but their core chain remains an indirect mapping of "hand movement intention → object movement." Users often need to maintain a dual representation of "hand movement imagery + object movement target," making it difficult to fundamentally solve the problems of excessively long intention chains and inconsistencies in the intention space, and the dependence on motor cortex rhythms remains significant. Therefore, in tasks where object movement is the control objective, common approaches include stimulus-driven selection control or indirect mapping mediated by limb motor imagery. This involves first decoding visual selection or limb intention, then mapping it to the object's direction of movement. These methods largely rely on sensorimotor cortical μ / β rhythms (ERD / ERS) and are not suitable for individuals with fluctuating states or impaired motor cortex function. Consequently, existing brain-computer interface-based object movement control methods have limited versatility. Furthermore, such brain-computer interface-based object movement control is inefficient.

[0024] It is evident that in tasks where object movement is the control objective, existing methods often employ stimulus-driven selection control or mapping methods mediated by limb motor imagery. This involves first decoding the visual selection intention or limb motor intention, and then mapping it to the object's direction of movement. Because the control chain of these methods is an indirect mapping, there is an additional conversion process between the user's intention and the control objective. Consequently, these methods often suffer from problems such as insufficient intuitiveness, lack of naturalness, and high cognitive burden, limiting their practicality. Furthermore, these methods typically rely on limb motor imagery-induced changes in the μ / β rhythm of the motor cortex (ERD / ERS), which has limited applicability to individuals with significant state fluctuations or motor cortex damage, further reducing the universality of brain-computer interface-based object movement control methods.

[0025] Based on this, the present application provides a brain-computer interface data processing method and apparatus based on object motion imagination, which can solve the technical problems of low universality of existing brain-computer interface-based object motion control.

[0026] The embodiments of the present invention will be further described below with reference to the accompanying drawings.

[0027] Figure 1This is a flowchart illustrating a brain-computer interface data processing method based on object motion imagery, provided as an embodiment of this application. Figure 1 As shown, the method includes: Step S110: In response to detecting that the user has a need to control the controlled object, the user's multi-channel EEG signals are collected through the brain-computer interface.

[0028] It should be noted that brain-computer interface (BCI) technology does not rely on the brain's normal physiological output pathways (peripheral nervous system and muscle tissue). It can decode a patient's intentions by detecting neural signals and is considered an ideal method for intention recognition. It is finding increasingly widespread application in fields such as rehabilitation medicine, smart home control, entertainment, and gaming.

[0029] In one possible implementation, the above-mentioned response to detecting a user's need to control the controlled object, and the acquisition of the user's multi-channel EEG signals via a brain-computer interface, may specifically include the following steps: in response to detecting a user's need to control the controlled object, outputting motion cue data of the controlled object to guide the user in direct motion visualization of the controlled object; wherein, the motion cue data includes at least one of the controlled object's direction of movement, trajectory of movement, mode of movement, and target location of movement; in response to the output of the motion cue data, acquiring the multi-channel EEG signals generated by the user during the direct motion visualization process via a brain-computer interface.

[0030] For example, in a task scenario, only the controlled object and its direction / trajectory cues are presented, guiding the user to directly imagine the controlled object (such as a box, cursor, vehicle, etc.) moving left / right (which can be translation or turning) in two-dimensional space, without requiring the user to imagine any hand or other limb movements. The presentation of the direction / trajectory cues is not limited to left and right arrows; it can be replaced with target location points, trajectory lines, path guide lines, turning symbols, color changes / flashing cues, sound cues, or tactile cues, as long as they guide the user to form a direct intention that "the controlled object moves left / right (or turns)."

[0031] Step S120: Feature extraction is performed on the multi-channel EEG signal, and the user's imagined intention to move the controlled object is decoded based on the extracted features to obtain the EEG intention representation result.

[0032] Among them, imagining intent refers to the user's imagination of the motion state of the controlled object.

[0033] In this embodiment of the application, the motion type of the controlled object is not limited to translation, but can also be various types of motion such as turning, curvilinear motion, moving along a preset path, and combinations thereof.

[0034] In practical applications, the above-mentioned intent decoding methods can include a variety of approaches: As an optional implementation, the above-described method of extracting features from multi-channel EEG signals and decoding the user's imagined intention to move a controlled object based on the extracted features to obtain an EEG intention representation result may specifically include the following steps: extracting interhemispheric difference features of the parietal-occipital region EEG signals based on specified frequency band signals in the multi-channel EEG signals; wherein, the specified frequency band signals include at least the theta frequency band signals; and decoding the user's imagined intention to move a controlled object based on the interhemispheric difference features of the parietal-occipital region EEG signals to obtain an EEG intention representation result.

[0035] For example, the above-mentioned intention decoding of the user's motor imagery intention towards the controlled object based on the interhemispheric difference features of the EEG signal in the parietal-occipital region to obtain the EEG intention representation result may specifically include the following steps: inputting the interhemispheric difference features into a lightweight model, decoding the user's motor imagery intention towards the controlled object through the lightweight model, and outputting the EEG intention representation result; wherein, the lightweight model includes a random forest model and / or a linear classifier. Among them, the interhemispheric difference characteristics of the above-mentioned parietal-occipital region EEG signals include at least one of the following: cross-correlation delay data of cross-hemispheric channels within a specified key time window, peak and / or trough latency difference between the left and right hemisphere channels, and interhemispheric phase difference or phase synchronization index data.

[0036] One of the decoding methods for intent decoding, such as Figure 2 As shown, in the machine learning approach, interpretable features are constructed for the EEG segment data of the imagination phase, with the left-right difference in the parietal-occipital region of the theta band (4–8 Hz) as the core. The features include, but are not limited to: the cross-correlation peaks of cross-hemispheric channel pairs (such as PO7–PO8, PO3–PO4, P7–P8, P3–P4) within key time windows (such as 0.4–1.0s) and their corresponding delays (such as xcorr-peak / xcorr-lag), the peak / valley latency difference between the left and right channels (Δt, used to characterize the lead / lag relationship), and one or more of the phase difference or phase synchronization indicators (Δφ or PLV, etc.). The above features are combined into feature vectors and then input into lightweight models such as random forests and linear classifiers to output intention categories such as "left / right". This approach has the advantages of being fast, easy to deploy, and friendly to small samples.

[0037] For example, if the peak of the left hemispherical occipital electrode is earlier than the peak of the right hemispherical occipital electrode, or the trough of the left hemispherical occipital electrode is earlier than the trough of the right hemispherical occipital electrode, then the EEG intention representation result is determined to be that the controlled object moves directly to the right; if the peak of the right hemispherical occipital electrode is earlier than the peak of the left hemispherical occipital electrode, or the trough of the right hemispherical occipital electrode is earlier than the trough of the left hemispherical occipital electrode, then the EEG intention representation result is determined to be that the controlled object moves directly to the left.

[0038] Under the above paradigm, decoding focuses on the theta band (4–8 Hz) and utilizes the interhemispheric differences in EEG signals in the parieto-occipital region as a criterion, such as interhemispheric phase difference or peak / valley latency (delay) difference; directional patterns: when imagining to the right, the left parieto-occipital response leads the right; when imagining to the left, the right parieto-occipital response leads the left. It should be noted that the "relative relationship between the left and right hemispheres (lateralization)" encoding directional information in the embodiments of this application emphasizes directional representations related to the visual-spatial network, rather than relying on traditional motor cortex rhythms.

[0039] Experiments have demonstrated that the proposed object movement imagery paradigm can induce stable and separable EEG patterns in the directional intention decoding task. Under broadband input (0.1–40 Hz), the average decoding accuracy reached 0.874 (see Table 1 below for a comparison of directional decoding accuracy between broadband (0.1–40 Hz) and the θ band (4–8 Hz) inputs in the direct object movement imagery task). To locate the key information sources relied upon by the model, occlusion analysis was employed: this involves phase randomization of signals within specific brain regions / channels / frequency bands / time windows, and the magnitude of the decoding performance degradation was observed to measure its importance. Occlusion analysis combining brain regions, channels, frequency bands, and time windows (e.g.) Figure 5 , Figure 6 , Figure 7 and Figure 8 As shown in the figure, the results indicate that the discriminative information of this paradigm is mainly concentrated in the theta band (4–8 Hz) 0.4–1.0 s after the start of imagination, and the contributing region is mainly the parietal-occipital region. This is different from traditional limb motor imagery, which mainly relies on the rhythm of the sensorimotor cortex, indicating that this paradigm relies more on brain networks related to visual-spatial representation and attention regulation.

[0040] To verify the specific manifestation of this directional discrimination information, the ERP in the θ band was further analyzed, such as... Figure 9As shown, stable lateralization timing characteristics can be observed: when imagining an object moving to the left, the peaks / troughs of the right parietal electrodes (P4, P8, PO4, PO8) precede those of the left (P3, P7, PO3, PO7); when imagining moving to the right, the peaks / troughs of the left parietal electrodes precede those of the right, indicating a stable timing difference between the left and right hemispheres of the parietal region when imagining in different directions. This result demonstrates the existence of direction-related temporal lateralization coding in the parietal region and can serve as a key direction discrimination feature in this paradigm. After restricting the input to the θ band (4–8 Hz), the decoding accuracy of all subjects (S1–S5) further improved, with the average accuracy increasing from 0.874 to 0.946 (+7.3 percentage points, as shown in Table 1 below), validating from the results that the discrimination information is indeed more concentrated in the θ band.

[0041]

[0042] Table 1 In addition, such as Figure 10 As shown, the time-frequency results indicate that this paradigm has a weak effect on inducing traditional sensorimotor cortical rhythms: the negative ERD of C3 / C4 is not significant, suggesting reduced motor cortex involvement. Finally, to explain its decision-making basis at the model level, DeepLIFT attribution analysis was used to quantify the contribution of each channel and time point to the output, such as... Figure 11 As shown, the attribution results showed a consistent posterior dominance in subjects S1–S5, with contributions mainly concentrated in the parietal-occipital related channels (P / PO / O bands, such as P7 / P8, PO7 / PO8, PO3 / PO4, Oz), further indicating that the model mainly utilizes information related to orientation representation in the visual-spatial network.

[0043] As another optional implementation, the above-mentioned feature extraction of multi-channel EEG signals and decoding of the user's imagined intention to move a controlled object based on the extracted features to obtain the EEG intention representation result can specifically include the following steps: inputting the EEG segment data corresponding to the multi-channel EEG signals into a classification model in the form of channel multiplied by time, performing end-to-end decoding of the direction of movement of the controlled object corresponding to the user's imagined intention to move a controlled object through the classification model, and outputting the movement direction category in the EEG intention representation result.

[0044] For another type of decoding method, such as Figure 3As shown, this is an end-to-end deep learning approach. The data from the imagination stage is directly input into a lightweight network such as EEGNet in a "channel × time" format (optionally with initial theta band pass-through). The output is a direction category, enabling the automatic learning of more complex cross-channel spatiotemporal patterns and improving generalization performance across subjects and tasks. Finally, the decoded directional intent is mapped to motion control commands from peripheral devices (such as virtual objects on a screen), and closed-loop control is formed through real-time visual feedback. This allows the device response to more closely resemble the user's actual object movement intent, thereby improving decoding stability and control accuracy.

[0045] In one possible implementation, such as Figure 4 As shown, the classification model can be an improved convolutional neural network optimized for the aforementioned object motion imagination paradigm. For example, the classification model can be an improved EEGNet network based on left-right electrode pairing difference and multi-branch temporal convolution, hereinafter referred to as ThetaDiffEEGNet. The classification model is used to combine hemispherical lateralization features and theta-band temporal dynamic features in the object motion imagination task to perform end-to-end decoding of the user's direct imaginary intent regarding the motion of a controlled object.

[0046] Specifically, in one possible implementation, the classification model includes left-right electrode pairing differential layering. Left-right electrode pairing differential layering is used to select pre-defined left-right symmetrical electrode pairs from multi-channel EEG signals and perform differential operations on the signals of each pair of left-right symmetrical electrodes at corresponding time points to obtain differential EEG features characterizing the differences between the left and right hemispheres. Left-right symmetrical electrode pairs may include left-right symmetrical electrode pairs in the frontal, central, parietal, and / or parieto-occipital regions. Through left-right electrode pairing differential layering, interhemispheric difference information can be explicitly introduced during the input stage, thereby making subsequent networks more conducive to learning lateralization representations related to the left-right movement of the controlled object.

[0047] In one optional implementation, the classification model further includes a multi-branch temporal convolutional layer. This multi-branch temporal convolutional layer is positioned after the left-right electrode pairing differential layer and is used for multi-timescale temporal modeling of differential EEG features. Specifically, the multi-branch temporal convolutional layer may include multiple standard temporal convolutional branches with different kernel lengths and / or multiple dilated temporal convolutional branches with different dilation rates. Different branches are used to extract rhythmic variation features at different time scales to enhance the representation of low-frequency oscillation information, phase delay information, and dynamic patterns within mid-term critical time windows. In some implementations, the multi-branch temporal convolutional layer focuses on extracting temporal features related to the theta band to adapt to the signal characteristics in the object's motion imagery paradigm where the main discriminative information is concentrated in the theta band.

[0048] In one possible implementation, the classification model further includes a spatial convolutional layer, a feature fusion layer, and a classification output layer. The spatial convolutional layer is used to further extract cross-electrode spatial distribution features, the feature fusion layer is used to concatenate, normalize, and nonlinearly transform the features output from each temporal convolutional branch, and the classification output layer is used to output the category result corresponding to the motion direction of the controlled object. The category result can include at least one of a left-direction category and a right-direction category.

[0049] As one possible implementation, the EEG segment data input to the classification model can be broadband EEG segment data or bandpass filtered EEG segment data in a specific frequency band. For example, the EEG segment data can be first filtered using the theta band before being input into the classification model to further highlight the rhythmic components related to the object's motor imagery intention.

[0050] Through the above setup, the classification model utilizes a paired differential structure of left and right electrodes to explicitly encode interhemispheric differences in EEG signals. Simultaneously, it employs a multi-branch temporal convolutional structure to extract multi-scale temporal dynamic features related to the subject's motor imagery, thereby improving the ability to recognize the controlled object's intended direction of movement. Compared to classification models that directly use a single temporal convolutional structure, this model better adapts to the characteristics of EEG representations in the subject's motor imagery task, which are dominated by low-frequency rhythms, have relatively concentrated key time windows, and exhibit significant hemispheric lateralization. This approach improves decoding performance and generalization ability while maintaining a lightweight model.

[0051] Step S130: Based on the EEG intention representation results, generate control instructions, and control the controlled object in the external device to perform the corresponding movement according to the control instructions.

[0052] As an optional implementation, the above-mentioned method of generating control commands based on EEG intention representation results and controlling the controlled object in the external device to perform corresponding movements according to the control commands may specifically include the following steps: generating movement control commands for the controlled object in the external device other than the user based on the EEG intention representation results; wherein, the controlled object includes a virtual controlled object and / or a physical controlled object; controlling the controlled object in the external device to perform corresponding movements according to the user's direct imagined movement intentions for the controlled object through the movement control commands, and outputting movement feedback information of the controlled object performing the corresponding movements.

[0053] The controlled objects mentioned above are not limited to Figure 2 and Figure 3The box in the diagram can be replaced with any object or device that can express direction / trajectory in a two-dimensional plane, including but not limited to: virtual objects such as cursors, spheres, icons, vehicle models, game characters, etc.; and physical devices such as mobile robots, smart wheelchairs, unmanned vehicle / car chassis, planar displacement targets of robotic arm end effectors, navigation target points, etc. The replacement of the controlled object does not change the basic principle of "direct object motion imagination → control command output" and all fall within the scope of protection of this invention.

[0054] In this embodiment, a direct object movement imagination EEG paradigm centered on the direct movement of the controlled object is employed. In this paradigm, multi-channel EEG signals are collected as the user directly imagines the controlled object moving automatically. Users can directly imagine the controlled object moving directly in space without imagining any driving or force exertion by hands or other limbs. This induces decodeable EEG representations of the controlled object's direct movement, which are then used to generate control commands. This realizes an intuitive brain-computer interface paradigm for object movement manipulation. This paradigm aligns with the natural thought process humans use in daily tasks to move, locate, and manipulate the trajectory of external objects, eliminating the need for users to imagine specific limb movements or... Instead of relying on muscle exertion, this approach directly visualizes the direction or trajectory of the controlled object (such as a box or other target object) in space at the cognitive level, thus forming an EEG intention representation highly consistent with the task objective. Compared to the traditional indirect motor imagery paradigm that maps hand-driven force to object movement, this approach shortens the intention chain, aligns the intention space with the control space, and better conforms to natural human manipulation habits. This helps improve the intuition and stability of online control and is also applicable to individuals with fluctuating states or impaired motor cortex, enhancing the versatility of brain-computer interface-based object motion control and solving the technical problem of low versatility in existing brain-computer interface-based object motion control.

[0055] Furthermore, by directly aligning the user's intention space with the controlled object's motion space through the aforementioned EEG paradigm of direct object motion imagery, a direct mapping from the controlled object's motion intention to control commands is established. This aligns more closely with human cognitive habits of natural manipulation and motion planning, thereby shortening the control chain, reducing learning and adaptation costs, and improving control intuitiveness. Moreover, the formation of a consistent and decodeable EEG representation in the directional dimension provides a more stable foundation for continuous control. Furthermore, it reduces the singular dependence on traditional sensorimotor cortex rhythms, improving robustness and versatility across different individuals and states. Therefore, this solution effectively improves upon the shortcomings of existing technologies, such as "strong external stimulus dependence" and "indirect mapping between limb intention and object motion," and provides a potential pathway for the natural manipulation of virtual / physical objects.

[0056] Figure 12A schematic diagram of a brain-computer interface data processing device based on object motion imagery is provided. (For example...) Figure 2 As shown, the brain-computer interface data processing device 1200 based on object motion imagery includes: The acquisition module 1201 is used to acquire the user's multi-channel electroencephalogram (EEG) signals through a brain-computer interface in response to the detection that the user has a control demand for the controlled object. The decoding module 1202 is used to extract features from the multi-channel EEG signals and decode the user's imagined intentions about the movement of the controlled object based on the extracted features to obtain the EEG intention representation result; wherein, the imagined intentions are the user's imagination of the movement state of the controlled object; The control module 1203 is used to generate control commands based on the EEG intention representation results, and to control the controlled object in the external device to perform corresponding movements based on the control commands.

[0057] The brain-computer interface data processing device based on object motion imagination provided in this application embodiment has the same technical features as the brain-computer interface data processing method based on object motion imagination provided in the above embodiment, so it can also solve the same technical problems and achieve the same technical effects.

[0058] An electronic device provided in this application embodiment, such as Figure 13 As shown, the electronic device 1300 includes a processor 1302 and a memory 1301. The memory stores a computer program that can run on the processor. When the processor executes the computer program, it implements the steps of the method provided in the above embodiments.

[0059] See Figure 13 The electronic device also includes a bus 1303 and a communication interface 1304. The processor 1302, the communication interface 1304 and the memory 1301 are connected via the bus 1303. The processor 1302 is used to execute executable modules, such as computer programs, stored in the memory 1301.

[0060] The memory 1301 may include high-speed random access memory (RAM) or non-volatile memory, such as at least one disk storage device. Communication between this system network element and at least one other network element is achieved through at least one communication interface 1304 (which can be wired or wireless), such as the Internet, wide area network, local area network, metropolitan area network, etc.

[0061] Bus 1303 can be an ISA bus, PCI bus, or EISA bus, etc. The bus can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 13 The symbol is represented by a single double-headed arrow, but this does not mean that there is only one bus or one type of bus.

[0062] The memory 1301 is used to store programs. After receiving an execution instruction, the processor 1302 executes the program. The method executed by the apparatus defined by the process disclosed in any of the preceding embodiments of this application can be applied to the processor 1302 or implemented by the processor 1302.

[0063] The processor 1302 may be an integrated circuit chip with signal processing capabilities. In implementation, each step of the above method can be completed by the integrated logic circuitry in the hardware of the processor 1302 or by instructions in software form. The processor 1302 may be a general-purpose processor, including a Central Processing Unit (CPU), a Network Processor (NP), etc.; it may also be a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field-Programmable Gate Array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor may be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this application can be directly manifested as execution by a hardware decoding processor, or execution by a combination of hardware and software modules in the decoding processor. The software module can reside in a mature storage medium in the field, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, or registers. This storage medium is located in memory 1301. Processor 1302 reads the information in memory 1301 and, in conjunction with its hardware, completes the steps of the above method.

[0064] Corresponding to the above-described brain-computer interface data processing method based on object motor imagery, this application embodiment also provides a computer-readable storage medium storing computer-executable instructions. When the computer-executable instructions are invoked and executed by a processor, the computer-executable instructions cause the processor to perform the steps of the above-described brain-computer interface data processing method based on object motor imagery.

[0065] The brain-computer interface data processing device based on object motor imagination provided in this application embodiment can be specific hardware on a device or software or firmware installed on the device. The implementation principle and technical effects of the device provided in this application embodiment are the same as those in the foregoing method embodiments. For the sake of brevity, any parts not mentioned in the device embodiment can be referred to the corresponding content in the foregoing method embodiments. Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can all be referred to the corresponding processes in the above method embodiments, and will not be repeated here.

[0066] In the embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. The apparatus embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. Furthermore, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Additionally, the displayed or discussed mutual couplings, direct couplings, or communication connections may be through some communication interfaces; indirect couplings or communication connections between devices or units may be electrical, mechanical, or other forms.

[0067] For example, the flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of apparatus, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram and / or flowchart, and combinations of blocks in block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0068] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0069] In addition, the functional units in the embodiments provided in this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0070] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the brain-computer interface data processing method based on object motor imagery described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0071] It should be noted that similar labels and letters in the following figures indicate similar items. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. In addition, the terms "first", "second", "third", etc. are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0072] Finally, it should be noted that the above-described embodiments are merely specific implementations of this application, used to illustrate the technical solutions of this application, and not to limit them. The protection scope of this application is not limited thereto. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can still modify or easily conceive of changes to the technical solutions described in the foregoing embodiments, or make equivalent substitutions for some of the technical features, within the scope of the technology disclosed in this application; and these modifications, changes, or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application. All should be covered within the protection scope of this application. Therefore, the protection scope of this application should be determined by the protection scope of the claims.

Claims

1. A brain-computer interface data processing method based on object motion imagery, characterized in that, The method includes: In response to the detection that the user has a need to control the controlled object, the multi-channel EEG signals of the user are collected through the brain-computer interface; Feature extraction is performed on the multi-channel EEG signals, and the user's imagined intention regarding the movement of the controlled object is decoded based on the extracted features to obtain the EEG intention representation result; wherein, the imagined intention is the user's imagination of the movement state of the controlled object; Based on the EEG intention representation results, control commands are generated, and the controlled object in the external device is controlled to perform corresponding movements according to the control commands; The step of collecting multi-channel EEG signals from the user via a brain-computer interface in response to detecting a user's need to control the controlled object includes: outputting motion cue data of the controlled object in response to detecting a user's need to control the controlled object, so as to guide the user to perform direct motion visualization of the controlled object through the motion cue data; wherein the motion cue data includes at least one of the controlled object's direction of movement, trajectory of movement, mode of movement, and target location of movement; and collecting multi-channel EEG signals generated by the user during the direct motion visualization process in response to the output of the motion cue data.

2. The method according to claim 1, characterized in that, The step of extracting features from the multi-channel EEG signals and decoding the user's imagined intention regarding the movement of a controlled object based on the extracted features to obtain an EEG intention representation result includes: Based on the specified frequency band signals in the multi-channel EEG signals, the interhemispheric difference features of the EEG signals in the parietal-occipital region are extracted; wherein, the specified frequency band signals include at least the θ frequency band signals; Based on the interhemispheric difference characteristics of the EEG signals in the parietal-occipital region, the user's intention to imagine movement toward the controlled object is decoded to obtain the EEG intention representation result.

3. The method according to claim 2, characterized in that, Based on the interhemispheric difference characteristics of the EEG signals in the parietal-occipital region, the user's motor imagery intention towards the controlled object is decoded to obtain the EEG intention representation result, including: If the peak of the left hemisphere occipital electrode is earlier than the peak of the right hemisphere occipital electrode, or the trough of the left hemisphere occipital electrode is earlier than the trough of the right hemisphere occipital electrode, then the EEG intention representation result is determined to be that the controlled object moves directly to the right. If the peak of the right hemisphere occipital electrode is earlier than the peak of the left hemisphere occipital electrode, or the trough of the right hemisphere occipital electrode is earlier than the trough of the left hemisphere occipital electrode, then the EEG intention representation result is determined to be that the controlled object moves directly to the left.

4. The method according to claim 2, characterized in that, Based on the interhemispheric difference characteristics of the EEG signals in the parietal-occipital region, the user's motor imagery intention towards the controlled object is decoded to obtain the EEG intention representation result, including: The interhemispheric difference features are input into a lightweight model, which decodes the user's motor imagery intentions toward the controlled object and outputs EEG intention representation results; wherein, the lightweight model includes a random forest model and / or a linear classifier.

5. The method according to claim 4, characterized in that, The interhemispheric differential characteristics of the EEG signals in the parieto-occipital region include at least one of the following: Cross-hemispherical channel cross-correlation delay data within a specified key time window, peak and / or trough latency difference between the left and right hemisphere channels, and inter-hemispherical phase difference or phase synchronization index data.

6. The method according to claim 1, characterized in that, The step of extracting features from the multi-channel EEG signals and decoding the user's imagined intention regarding the movement of a controlled object based on the extracted features to obtain an EEG intention representation result includes: The EEG segment data corresponding to the multi-channel EEG signal is input into the classification model in the form of channel multiplied by time. The classification model performs end-to-end decoding of the direction of movement of the controlled object corresponding to the user's intention to imagine movement of the controlled object, and outputs the movement direction category in the EEG intention representation result.

7. The method according to claim 6, characterized in that, The classification model is a convolutional neural network, which includes left and right electrode pairing difference layer, multi-branch temporal convolutional layer and classification output layer; The left and right electrode pairing difference layer is used to extract the left and right hemisphere difference features in the multi-channel EEG signal, the multi-branch temporal convolutional layer is used to extract the temporal features of the left and right hemisphere difference features at different time scales, and the classification output layer is used to output the category result corresponding to the movement direction of the controlled object.

8. The method according to claim 7, characterized in that, The left and right electrode pairing differential layer is used to select a preset left and right symmetrical electrode pair from the multi-channel EEG signal, and perform differential operation on the EEG signal corresponding to each pair of left and right symmetrical electrodes to generate differential EEG features. The preset left-right symmetrical electrode pair includes at least a portion of the left-right symmetrical electrode pairs in the forehead region, central region, vertex region, and / or vertex-occipital region.

9. The method according to claim 7 or 8, characterized in that, The multi-branch temporal convolutional layer includes multiple temporal convolutional branches with different kernel lengths and / or multiple dilated temporal convolutional branches with different dilation rates, used to extract multi-scale temporal dynamic features related to the user's imagined intention to move the controlled object directly.

10. The method according to claim 9, characterized in that, The multi-branch temporal convolutional layer focuses on extracting temporal features related to the θ band, and / or the EEG segment data input to the classification model is EEG segment data filtered by the θ band.

11. The method according to claim 1, characterized in that, The step of generating control commands based on the EEG intention representation results, and controlling the controlled object in the external device to perform corresponding movements according to the control commands, includes: Based on the EEG intention representation results, motion control commands are generated for the controlled object in an external device other than the user; wherein the controlled object includes a virtual controlled object and / or a physical controlled object; The motion control command controls the controlled object in the external device to move according to the user's direct imagined motion intention for the controlled object, and outputs motion feedback information of the controlled object performing the corresponding motion.

12. A brain-computer interface data processing device based on object motion imagination, characterized in that, include: The acquisition module is used to acquire the user's multi-channel electroencephalogram (EEG) signals through a brain-computer interface in response to the detection that the user has a control demand for the controlled object. The decoding module is used to extract features from the multi-channel EEG signals and decode the user's imagined intentions about the movement of the controlled object based on the extracted features, so as to obtain the EEG intention representation result; wherein, the imagined intention is the user's imagination of the movement state of the controlled object; The control module is used to generate control commands based on the EEG intention representation results, and to control the controlled object in the external device to perform corresponding movements based on the control commands. The acquisition module is specifically used for: in response to detecting that the user has a control demand for the controlled object, outputting motion prompt data of the controlled object, so as to guide the user to perform direct motion imagination of the controlled object through the motion prompt data; wherein, the motion prompt data includes at least one of the controlled object's direction of movement, trajectory of movement, mode of movement, and target location of movement; in response to the output of the motion prompt data, acquiring multi-channel EEG signals generated by the user during the direct motion imagination through a brain-computer interface.

13. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions that, when invoked and executed by a processor, cause the processor to perform the method according to any one of claims 1 to 11.

Images