A three-dimensional motor intention decoding method based on electroencephalic source localization

Abstract

The application relates to the technical field of electroencephalogram monitoring and processing, in particular to a three-dimensional motion intention decoding method based on electroencephalography source localization. The method comprises the following steps: acquiring electroencephalogram data of a target user during three-dimensional motion imagination, wherein the three-dimensional motion imagination comprises imagined activities of three-dimensional limb movement; determining source estimation data of a motion brain area based on the electroencephalogram data of the target user during the three-dimensional motion imagination, wherein the motion brain area is part of all brain areas in the whole brain; combining the source estimation data of the motion brain area and a motion coordinate model of the target user during the three-dimensional motion imagination to decode corresponding coordinate information of the target user during the three-dimensional motion imagination; wherein the coordinate information serves as indication information for position movement of an external device. Through the motion intention decoding mode and the structured training scheme based on electroencephalography source localization, fine positioning and continuous movement of a mechanical arm can be finally realized under the support of an electroencephalography device.

Description

Technical Field

[0001] This application relates to the field of electroencephalogram (EEG) signal monitoring and processing technology, and in particular to a three-dimensional motion intention decoding method based on brain power source localization. Background Technology

[0002] Electroencephalography (EEG), as a non-invasive brain function testing technology, has been widely used in neuroscience, medical diagnosis, and brain-computer interfaces due to its high temporal resolution, low cost, and ease of operation.

[0003] Motor imagery (MI) refers to the mental process by which an individual simulates specific motor movements in their mind without actually performing the actions. Motor imagery is widely used in brain-computer interface systems, allowing users to control corresponding external devices (such as robotic arms) simply by visualizing motor movements, without actually moving the device.

[0004] However, traditional electrode-based motion visualization decoding methods generally suffer from low control precision for robotic arms. Summary of the Invention

[0005] One objective of this application is to propose a three-dimensional motion intention decoding method based on brain power source localization. The method provided in this application can convert the collected EEG data of the user into activation data of the brain source space. By combining the EEG data and the source space data, three-dimensional motion imagery decoding can be performed to achieve control of three-dimensional motion and improve the accuracy of three-dimensional motion imagery decoding. Through the motion intention decoding method based on brain power source localization and the structured training scheme proposed in this invention, the precise localization and continuous movement of the robotic arm can be achieved with the support of EEG devices.

[0006] According to an embodiment of the first aspect of this application, this application provides a three-dimensional motion intention decoding method based on brain power localization, the method comprising:

[0007] Acquire EEG data of target users during three-dimensional motion visualization, wherein the three-dimensional motion visualization includes the visualization of three-dimensional limb movements;

[0008] Based on the EEG data of the target user during three-dimensional motor imagination, source estimation data of the motor brain region is determined, wherein the motor brain region is a part of the whole brain region.

[0009] By combining the source estimation data of the motor brain region and the action coordinate model of the target user's three-dimensional motor imagination, the coordinate information corresponding to the target user's three-dimensional motor imagination is decoded; wherein, the coordinate information serves as an indication of the movement of the external device.

[0010] In some embodiments, source estimation data for motor brain regions are determined based on the target user's electroencephalogram (EEG) data during three-dimensional motor imagery, including:

[0011] Based on the EEG data of the target user's three-dimensional motor imagery, the source estimation data of the whole brain is determined;

[0012] Obtain the motor brain regions associated with the three-dimensional motor imagery, and determine the source estimation data of the motor brain regions based on the source estimation data of the whole brain and the motor brain regions of the brain.

[0013] In some embodiments, the step of determining the motion coordinate model includes:

[0014] Acquire the EEG data of the user during three-dimensional motion visualization;

[0015] Based on the EEG data of the user during three-dimensional motor imagery, the source estimation data of the motor brain region of the user during each three-dimensional motor imagery are determined.

[0016] Based on the user's three-dimensional motion imagination and the source estimation data of the motor brain region during three-dimensional motion imagination, the action coordinate model is trained; wherein, the action coordinate model is the mapping relationship between the neural activity of the motor brain region and the external space.

[0017] In some embodiments, the motion coordinate model is used to represent the motion coordinates of a three-dimensional motion image in a three-dimensional space.

[0018] In some embodiments, the motion coordinate model includes multiple motion coordinate sub-models representing the movement of the three-dimensional motion image in different dimensional spaces, wherein each motion coordinate sub-model represents the movement coordinates of the motion image in different dimensional directions in the external space.

[0019] In some embodiments, the motion coordinate sub-model of the three-dimensional motion imagination moving in different dimensional spaces includes at least one or more of the following:

[0020] Action coordinate sub-models at different depth directions; and / or,

[0021] A sub-model of motion coordinates in the left and right directions of a two-dimensional plane; and / or,

[0022] A sub-model of motion coordinates in the vertical direction of a two-dimensional plane.

[0023] In some embodiments, the motion coordinate sub-model of the three-dimensional motion imagination moving in different dimensional spaces includes motion coordinate sub-models in different depth directions, motion coordinate sub-models in the left and right directions of the two-dimensional plane, and motion coordinate sub-models in the up and down directions of the two-dimensional plane.

[0024] The step of combining the source estimation data of the motor brain region and the action coordinate model of the target user's three-dimensional motor imagination to decode the coordinate information corresponding to the target user's three-dimensional motor imagination includes:

[0025] Based on the source estimation data of the motor brain region and the action coordinate sub-model in different depth directions, the depth coordinate information corresponding to the target user's three-dimensional motion imagination is decoded.

[0026] Based on the source estimation data of the motor brain region and the action coordinate sub-model in the left and right directions of the two-dimensional plane, the left and right coordinate information corresponding to the target user during three-dimensional motor imagination is decoded.

[0027] Based on the source estimation data of the motor brain region and the action coordinate sub-model in the vertical direction of the two-dimensional plane, the vertical coordinate information corresponding to the target user during three-dimensional motor imagination is decoded.

[0028] The coordinate information corresponding to the target user's three-dimensional motion imagination is determined based on the depth coordinate information, the left and right coordinate information, and the up and down coordinate information.

[0029] In some embodiments, the method further includes:

[0030] Using the three-dimensional coordinate information in the coordinate information as the target position of the external device, the external device is controlled to move the target user's limbs to the target position; and / or...

[0031] Using the depth coordinates from the coordinate information as the target position of the external device, the external device is controlled to move the target user's limbs to the target position; and / or...

[0032] Using the vertical coordinates from the coordinate information as the target position of the external device, the external device is controlled to move the target user's limbs to the target position; and / or...

[0033] The left and right coordinates in the coordinate information are used as the target position of the external device, and the external device is controlled to move the target user's limbs to the target position.

[0034] In some embodiments, the method further includes:

[0035] Obtain the head structure image corresponding to the target user;

[0036] Based on the head structure image of the target user, construct the head model of the target user and the brain source space to be estimated;

[0037] Based on the electrode layout of the EEG acquisition device, the head model of the target user, and the brain source space to be estimated, a forward model for source localization is determined; wherein, the forward model for source localization is used to characterize the physical relationship between the brain source space to be estimated and the EEG data through the head model;

[0038] The source estimation data for determining the whole brain includes:

[0039] The EEG data of the target user is acquired, and the EEG data is solved based on the inverse solution model and the forward model of the source localization to determine the source estimation result of the source space corresponding to the EEG data.

[0040] In some embodiments, the method further includes:

[0041] The acquisition of the target user's EEG data includes acquiring the user's EEG data through a contact-type EEG acquisition device or monitoring the user's EEG data through a non-contact EEG device;

[0042] Monitoring users' brainwave data through non-contact EEG devices, including:

[0043] Obtain the head image of the target user at the current moment;

[0044] Determine the head pose of the target user at the current time based on the head image corresponding to the current time;

[0045] Based on the head pose at the current moment and the head pose at the previous moment, the pose of the EEG cap is determined, wherein the EEG cap includes multiple EEG electrodes, and the multiple EEG electrodes monitor the EEG data of the target user under the pose corresponding to the EEG cap.

[0046] This application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the three-dimensional motion intent decoding method based on brain power localization provided in any of the above embodiments.

[0047] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of a three-dimensional motion intent decoding method based on brain power localization provided in any of the above embodiments.

[0048] This application proposes a three-dimensional motion intention decoding method based on brain power source localization, which can obtain the brain's neural source activity while preserving the temporal resolution advantage of EEG data, and use it for motion image decoding to achieve motion image decoding results with high spatial accuracy.

[0049] In some scenarios, the control of robotic arms achieved through decoding motor imagery based on EEG electrode-level data is relatively discrete, mostly only capable of limited predefined movements, making it difficult to achieve continuous movement and spatial positioning of the robotic arm. This is due to the limited spatial resolution of EEG electrode-level data, resulting in low accuracy for relatively complex and fine motor imagery decoding. This application proposes a three-dimensional motor intention training and decoding process based on EEG source localization, ultimately achieving fine control of the robotic arm. The training process accompanying this patent defines a training space, within which several sub-training spaces are defined. In each sub-training space, the subject performs horizontal hand movements along the coordinate axes, vertical hand movements along the coordinate axes, quadrant positioning, and other procedures. During training, EEG data and the resulting motor cortex source localization results are simultaneously collected, establishing a mapping relationship between the training space and motor cortex activity patterns. Through the proposed EEG source localization-based motor intention decoding method and structured training scheme, fine positioning and continuous movement of the robotic arm can ultimately be achieved with the support of EEG devices.

[0050] Additional aspects and advantages of this application will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of this application. Attached Figure Description

[0051] The above and / or additional aspects and advantages of this application will become apparent and readily understood from the description of the embodiments taken in conjunction with the following drawings, wherein:

[0052] Figure 1 This is a flowchart illustrating a three-dimensional motion intent decoding method based on brain power localization provided in one embodiment of this application;

[0053] Figure 2 This is a flowchart illustrating a method for determining coordinate information for three-dimensional motion visualization provided in one embodiment of this application.

[0054] Figure 3 This is a schematic diagram of constructing a training space provided in one embodiment of this application;

[0055] Figure 4 This is a flowchart illustrating a three-dimensional motion intent decoding method based on brain power localization provided in another embodiment of this application;

[0056] Figure 5 This is a schematic diagram of controlling the movement of a robotic arm based on acquired three-dimensional coordinates, provided in one embodiment of this application;

[0057] Figure 6 This is a schematic diagram of a non-contact EEG data acquisition device provided in one embodiment of this application;

[0058] Figure 7 This is a schematic diagram of the structure of a computer device according to an embodiment of this application. Detailed Implementation

[0059] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0060] The technical solutions of the various embodiments of the present invention can be combined with each other, but only if they are based on the ability of those skilled in the art to implement them. When the combination of technical solutions is contradictory or cannot be implemented, it should be considered that such combination of technical solutions does not exist and is not within the scope of protection claimed by the present invention.

[0061] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, specific embodiments of this application will be described in detail below with reference to the accompanying drawings.

[0062] This application provides a three-dimensional motion intention decoding method based on brain-source localization. The method decodes three-dimensional motion intentions by acquiring the target user's EEG data and corresponding source estimation data, and controls external devices to move based on the decoded instruction information. This method obtains information about brain neural activity while preserving the temporal resolution advantage of EEG data, achieving three-dimensional motion intention decoding with high spatial accuracy and enabling precise control of external devices.

[0063] Specifically, such as Figure 1 As shown, this application provides a three-dimensional motion intention decoding method based on brain power localization, wherein the method includes:

[0064] S101, acquire the EEG data of the target user during three-dimensional motion imagination, wherein the three-dimensional motion imagination includes the imagination activity of three-dimensional limb movement.

[0065] In some embodiments, the target user can be a person to be trained or a person to be tested, specifically individuals of different genders, ages, and ethnicities. The training needs (testing needs or neurotraining goals) of the person to be trained (or the person to be tested) can be rehabilitation training or targeted training or testing for a specific cause or training goal. By acquiring the target user's EEG data, without requiring the target user to perform actual actions, the brain activity classification of the target user is determined, and external devices are used to control the target user to complete the actions imagined in the target user's brain, thereby achieving the targeted training goal.

[0066] In some embodiments, the target user performs three-dimensional motion visualization, which includes the visualization of three-dimensional limb movements, including hands, legs, or other limb parts. Three-dimensional motion includes movement in any direction in external space, which can be up, down, left, right, forward, backward, or a combination of these directions. For example, when visualizing three-dimensional hand movements, the target user can imagine moving their hand forward eight positions, to the right one position, and upward one position in their mind. The EEG data of the target user's three-dimensional motion visualization is acquired, and then the acquired EEG data is decoded to obtain the decoded information of the hand movements. This can then be used to realize the actual movement visualized in the mind through an external device.

[0067] In some embodiments, the target user may acquire EEG data by wearing a contact-based EEG device. In other embodiments, the target user may also acquire EEG data using a non-contact EEG device.

[0068] S102, Based on the EEG data of the target user during three-dimensional motor imagination, determine the source estimation data of the motor brain region, wherein the motor brain region is a part of the whole brain region.

[0069] Among them, the source estimation data is obtained by brain power estimation technology, which is to perform mathematical modeling and inversion of multi-channel EEG data by combining brain structural information obtained from magnetic resonance imaging, and to estimate the neural source activity in the cerebral cortex and even the subcortical region.

[0070] Because the EEG signals (EEG data) measured on a limited number of electrodes on the scalp are affected by the volume conduction effect, the measured EEG signals (EEG data) are only a mixed representation of neural source activity and have low accuracy.

[0071] In some embodiments, source estimation results can be obtained by constructing forward and backward computational models to reconstruct the neural source activity of EEG signals in a given source space. The spatial resolution of the EEG data after source estimation (i.e., the source estimation results) is improved, and the temporal state of specific brain functional areas (such as the lateral occipital cortex, medial occipital cortex, or ventral occipital cortex) can be reconstructed. Thus, the accuracy of subsequent data calculations can be improved based on the source estimation results.

[0072] In the above embodiments, the acquired EEG data is converted into a source estimation result of the whole brain, and then three-dimensional motion intention decoding can be performed based on the acquired source estimation result. Three-dimensional motion intention decoding based on EEG data and its source estimation data can significantly enhance spatial information, achieve more accurate three-dimensional motion intention decoding, and ultimately realize fine positioning and continuous movement of external devices.

[0073] In some embodiments, a physical model mapping between EEG data and source estimation data can be pre-constructed, and then the source estimation data can be calculated based on the acquired EEG data using this physical model.

[0074] In some embodiments, determining the source estimation data of the whole brain based on the EEG data of the target user includes: acquiring the EEG data of the target user, solving the EEG data based on the inverse solving model and the forward model of source localization, and determining the source estimation result of the source space corresponding to the EEG data. The method for determining the forward model includes: acquiring a head structure image corresponding to the target user; constructing a head model of the target user and a brain source space to be estimated based on the head structure image of the target user; determining a forward model of source localization based on the electrode layout of the EEG acquisition device, the head model of the target user, and the brain source space to be estimated; wherein the forward model of source localization is used to characterize the physical relationship between the brain source space to be estimated and the EEG data through the head model.

[0075] The reverse calculation model is shown in the following formula.

[0076] W = eLORETAW(G,λ)

[0077] J = W·X EEG ,J∈R 3m×t

[0078] Among them, eLORETAW is a standard source localization method with good localization accuracy. W represents the inverse model constructed based on the forward model G and the covariance matrix of the EEG data itself, which is used to estimate the activation value of each source point in the source space from the EEG data.

[0079] G represents a forward model composed of a 3-layer head model constructed from MRI structural images, EEG channel information, and a constructed source space. It is used to describe how each source point in the source space collectively influences and forms the signal observed on each channel of the EEG.

[0080] J represents the activation map of the source space estimated based on the collected EEG data and the constructed inverse model, where each source point in the source space has an estimated activation value.

[0081] X EEG This represents the collected EEG data.

[0082] In the above embodiments, the source estimation data of the source space is calculated by using the EEG data obtained by the EEG acquisition device, thereby knowing the brain neural source activity of the target user. By combining the EEG data and the source estimation data, rich and anatomically significant features are obtained, thereby achieving more accurate motor imagery classification and realizing the training goal.

[0083] In some embodiments, the motor brain region is a portion of all brain regions in the whole brain. In this embodiment, the motor brain region refers to the brain region in which the activity patterns of certain neurons in the motor cortex of the brain have a high degree of consistency with the position of their limbs in the external objective three-dimensional space.

[0084] In some embodiments, the source estimation data for different brain regions includes brain regions of interest corresponding to the training target. The brain regions of interest are the brain regions in the whole brain of the target user that are activated to a degree greater than a preset threshold when the target user imagines three-dimensional motion intentions based on the training target.

[0085] Based on the target user's EEG data, brain regions of interest (ROIs) corresponding to the target user's training objectives are determined. These ROIs are determined based on the target user's training objectives. For example, if the target user imagines their hand moving in external space, source estimation data for the motor brain regions related to hand movement are obtained based on these movement-related ROIs.

[0086] The aforementioned motor brain regions can refer to brain regions that have a high degree of consistency with the movement of the target user's hands, legs, or other limbs, or brain regions that have a high degree of consistency with one or more of the movements of the target user's hands, legs, or other limbs.

[0087] In the above embodiments, extracting source estimation results for specific brain regions (regions of interest or motor regions) closely related to the training objective for the target user, and then using these source estimation results for 3D motion intent decoding, can improve data processing efficiency. Simultaneously, the source estimation data for the regions of interest is closely related to the training objective, resulting in more targeted data and a better match between the obtained 3D motion intent decoding results and the user's expectations. Therefore, acquiring only the source estimation data for the regions of interest reduces data processing pressure and improves operational efficiency and accuracy.

[0088] S103, combining the source estimation data of the motor brain region and the action coordinate model of the target user's three-dimensional motion imagination, decode the coordinate information corresponding to the target user's three-dimensional motion imagination; wherein, the coordinate information serves as indication information for the movement of the external device.

[0089] During the target user's three-dimensional motion intention visualization, although no actual movement occurs, brain regions that highly overlap with actual movement are still activated, including the primary motor cortex and sensorimotor association areas. By combining the target user's motion coordinate model during three-dimensional motion visualization, the user's imagined motion intention in three-dimensional external space is decoded. The decoded three-dimensional information is then combined to form coordinate information, which is input to an external device to move it to a specified coordinate position, thereby enabling the target user to control the position of the external device.

[0090] In some embodiments, the decoded coordinate information corresponding to the target user's three-dimensional motion imagination may include three-dimensional coordinate information or two-dimensional coordinate information, etc. The external device is controlled to move according to the decoded coordinate information. For example, the decoded three-dimensional information is combined to form coordinate information, which is then input to a robotic arm to move it to a specified coordinate position, thereby completing the user's control over the robotic arm's position.

[0091] In some embodiments, source estimation data for motor brain regions are determined based on the target user's electroencephalogram (EEG) data during three-dimensional motor imagery, including:

[0092] Based on the EEG data of the target user's three-dimensional motor imagery, source estimation data for the whole brain is determined. Motor brain regions associated with the three-dimensional motor imagery are acquired, and based on the source estimation data of the whole brain and the motor brain regions of the cerebrum, source estimation data for the motor brain regions is determined.

[0093] In the above embodiments, source estimation data of the whole brain can be obtained based on the acquired EEG data, and then source estimation data corresponding to the motor brain regions can be extracted from the source estimation data of the whole brain.

[0094] In another embodiment, source estimation data of the motor brain region can be obtained directly based on EEG data, without the need to obtain source estimation data of the whole brain.

[0095] In some embodiments, the action coordinate model is a mapping relationship between external spatial coordinate information and source estimation data of motor brain regions. Therefore, before decoding the user's three-dimensional motor intention, an action coordinate model must first be constructed to determine the mapping relationship between the neural activity of the motor brain regions and external space, that is, the position coordinates of the user's imagined movement in external three-dimensional space.

[0096] The steps for determining the motion coordinate model include:

[0097] Obtain the EEG data of the user during three-dimensional motion visualization.

[0098] In some embodiments, a three-dimensional training space is constructed in front of the user. Based on the three-dimensional training space, the target user performs three-dimensional motion visualization during the training phase, and the EEG data of the three-dimensional motion visualization and the coordinate information of the corresponding visualization position are obtained for each training phase.

[0099] Based on the EEG data of the user during three-dimensional motor imagery, the source estimation data of the motor brain region of the user during each three-dimensional motor imagery are determined.

[0100] The motion coordinate model is trained based on the user's three-dimensional motion imagination and the source estimation data of the motor brain region during three-dimensional motion imagination. The motion coordinate model represents the mapping relationship between the neural activity of the motor brain region and the external space.

[0101] For example, a training coordinate point (8, 1, 1) is determined in the training space. The target user imagines controlling their hand (or other limbs) to touch this coordinate point, and EEG data for this training task is acquired. Based on the EEG data from this training task, the source estimation data of the motor brain region when the target user's hand moves to coordinate (8, 1, 1) is determined. A mapping relationship is established between the coordinate information (8, 1, 1) and the source estimation data of the motor brain region for this training task.

[0102] Similarly, the mapping relationship between the source estimation data of the motor brain region and the training space coordinate information is obtained for each training task. Based on the mapping relationship between the source estimation data of the motor brain region and the training space coordinate information, the action coordinate model is constructed.

[0103] In some embodiments, the motion coordinate model is used to represent the motion coordinates of a three-dimensional motion image in a three-dimensional space.

[0104] In some embodiments, the motion coordinate model includes multiple motion coordinate sub-models representing the movement of the three-dimensional motion image in different dimensional spaces, wherein each motion coordinate sub-model represents the movement coordinates of the motion image in different dimensional directions in the external space.

[0105] For example, different dimensions can refer to the up and down direction, or the left and right direction, or the front and back direction, etc.

[0106] In some embodiments, the motion coordinate sub-model of the three-dimensional motion imagination moving in different dimensional spaces includes at least one or more of the following:

[0107] For example, it includes a single model in different dimensional directions, such as an action coordinate sub-model in different depth directions, or an action coordinate sub-model in the left-right direction of a two-dimensional plane, or an action coordinate sub-model in the up-down direction of a two-dimensional plane.

[0108] For example, it includes combinations of two models in different dimensional directions, such as motion coordinate sub-models in different depth directions, or motion coordinate sub-models in the left and right directions of a two-dimensional plane, or motion coordinate sub-models in the up and down directions of a two-dimensional plane, or combinations of any two models in the above three dimensional models.

[0109] For example, it includes a combination of models in three dimensions, such as motion coordinate sub-models in different depth directions, motion coordinate sub-models in the left and right directions of a two-dimensional plane, and motion coordinate sub-models in the up and down directions of a two-dimensional plane.

[0110] In the above embodiments, models corresponding to the three dimensions can be trained separately. This allows for the acquisition of coordinate information in the three dimensions through the three models, enabling control of three-dimensional motion. Alternatively, models corresponding to only one or two dimensions can be acquired, with the coordinates of the other dimensions pre-specified.

[0111] See Figure 2 In some embodiments, the motion coordinate sub-models of the three-dimensional motion imagination in different dimensional spaces include motion coordinate sub-models in different depth directions, motion coordinate sub-models in the left and right directions of the two-dimensional plane, and motion coordinate sub-models in the up and down directions of the two-dimensional plane.

[0112] The step of combining the source estimation data of the motor brain region and the action coordinate model of the target user's three-dimensional motor imagination to decode the coordinate information corresponding to the target user's three-dimensional motor imagination includes:

[0113] S201, based on the source estimation data of the motor brain region and the action coordinate sub-model in different depth directions, decode the depth coordinate information corresponding to the target user's three-dimensional motion imagination.

[0114] S202, based on the source estimation data of the motor brain region and the action coordinate sub-model in the left and right directions of the two-dimensional plane, decode the left and right coordinate information corresponding to the target user's three-dimensional motor imagination.

[0115] S203, based on the source estimation data of the motor brain region and the action coordinate sub-model in the vertical direction of the two-dimensional plane, decode the vertical coordinate information corresponding to the target user's three-dimensional motor imagination.

[0116] S204, Based on the depth coordinate information, the left and right coordinate information, and the up and down coordinate information, determine the coordinate information corresponding to the target user's three-dimensional motion imagination.

[0117] In the above embodiments, coordinate information of the corresponding dimensions is obtained based on the three-dimensional coordinate model, and then combined to obtain three-dimensional coordinate information, so as to realize precise control of the robotic arm and achieve fine control of motion.

[0118] In some embodiments, the method further includes:

[0119] Using the three-dimensional coordinate information in the coordinate information as the target position of the external device, the external device is controlled to move the target user's limbs to the target position; and / or...

[0120] Using the depth coordinates from the coordinate information as the target position of the external device, the external device is controlled to move the target user's limbs to the target position; and / or...

[0121] Using the vertical coordinates from the coordinate information as the target position of the external device, the external device is controlled to move the target user's limbs to the target position; and / or...

[0122] The left and right coordinates in the coordinate information are used as the target position of the external device, and the external device is controlled to move the target user's limbs to the target position.

[0123] In the above embodiments, the obtained coordinate information is used as the target position for controlling the external device, thereby achieving fine-grained control of the external device.

[0124] It should be noted that motion in three-dimensional space can refer to planar motion in three-dimensional space. For example, it can be motion only in the left and right directions, or motion in the up and down directions, or motion in the plane containing both the left and right and up and down directions.

[0125] In another embodiment, when the obtained coordinate information contains less than three dimensions, the method further includes supplementing the coordinate information based on predefined coordinate information. For example, when only depth coordinate information is obtained, the method further includes combining the obtained depth information with predefined left and right coordinate information and up and down coordinate information to obtain three-dimensional coordinate information, and performing position control of external devices based on the obtained three-dimensional coordinate information.

[0126] In some embodiments, limb movements include hand movements, and the activity patterns of certain neurons in the motor cortex of the brain show a high degree of consistency with the position of the hand in the external objective three-dimensional space. By establishing a mapping relationship between the human's imagined position of their hand in the external objective three-dimensional space and the activity patterns of their motor cortex, more precise control of the robotic arm can be achieved.

[0127] See Figures 3 to 5 In some embodiments, the three-dimensional motion intent decoding method based on brain power localization provided in this application includes the following steps:

[0128] Step 1: Construct a 3D training space in front of the user, dividing it along the depth direction (d) into multiple sub-2D training spaces (x, y). See details below. Figure 3 .

[0129] Step 2: Collect EEG data of the user's hand spatial positioning imagery in the training space. Each 3D spatial location to be trained can be represented by 3D coordinates in the training space, such as (8,1,1) representing the training point in the first quadrant at a depth of 8. See details... Figure 3 .

[0130] See Figure 3 Specifically, this is achieved by a computer screen that moves horizontally in different depth directions. For example, if the screen moves to a position at a depth of 8 (coordinates (8, 1, 1)) and marks a red dot in the first quadrant, the user needs to imagine reaching out to touch the red dot on the screen, and the EEG data of this three-dimensional motion imagery is recorded.

[0131] Step 3: Construct a three-layer head model based on standard brain or the user's own MRI structural images: a scalp model, an external skull model, and an internal skull model. Source localization of EEG data is performed and combined with the motor cortex extracted from the MRI structural images to obtain the activation map of the motor cortex for each training trial, which serves as the input feature. See also Figure 4 Specifically, the process involves collecting EEG data by having the user wear an EEG cap. A three-layer head model is constructed based on the acquired MRI head images. A forward-looking model is then built based on the EEG data and the head model. Source estimation is performed on the EEG data using the forward-looking model to obtain source estimation data, which is then combined with the motor cortex to obtain source estimation data for the motor brain regions. Finally, motion decoding is performed on the three pre-trained independent coordinate models to obtain the three-dimensional motion coordinates (x, y, z).

[0132] Step 4: Split the training samples into three categories: training samples at different depths (8 categories), training samples horizontally on the two-dimensional plane (2 categories), and training samples vertically on the two-dimensional plane (2 categories). Using a multi-layer perceptron, establish the mapping relationship between the motor cortex and three-dimensional space under each training method.

[0133] Step 5: After training is complete, it can be applied to control external devices.

[0134] For example, if a user wants to control a robotic arm to move to a specific location and perform motor imagery, the corresponding EEG data is source-localized and the motor cortex (motor brain region) is extracted. This data is then input into three trained models. The decoded information from the three dimensions is combined to form coordinates, which are then input into the robotic arm to move it to the specified coordinate position, thus enabling the user to control the position of the robotic arm.

[0135] In some embodiments, the step of constructing a forward model of source localization between the brain source space to be estimated and the EEG data includes:

[0136] Step 1: Obtain the head structure image corresponding to the target user.

[0137] Among them, the head structure image can be the target user's MRI structure image.

[0138] In some embodiments, the head MRI image may be the user's own MRI image; in other embodiments, the head MRI image may be a head MRI image similar to the user's, such as an MRI image matched from a standard brain database.

[0139] Step 2: Based on the head structure image of the target user, construct the head model of the target user and the brain source space to be estimated.

[0140] Obtain the MRI structural image corresponding to the target user, and construct the corresponding head model based on the MRI structural image.

[0141] The head model may include a three-layer head model: a scalp model, an external skull model, and an internal skull model.

[0142] The specific formula is as follows:

[0143]

[0144] Where B represents the head model, S scalp Indicates a scalp model (left side), S outer_skull Indicates the external skull model (middle), S inner_skull This shows a model of the internal skull (right side).

[0145] It is understandable that the precision of the constructed head model can be customized.

[0146] Obtain the MRI structural image corresponding to the target user, and determine the corresponding brain source space to be estimated based on the MRI structural image.

[0147] The specific formula is as follows:

[0148]

[0149] Where S represents the brain source space to be estimated, s1, s2, ..., s m This represents the source point in the brain's source space.

[0150] Point data in the brain source space represent source points. The cortical-level source space, constructed based on standard brain or user-generated MRI structural images, contains 8196 source points. It should be noted that the spatial resolution and depth of the constructed brain source space can be customized.

[0151] Step 3: Based on the electrode layout of the EEG acquisition device, the head model of the target user, and the brain source space to be estimated, determine the forward model for source localization.

[0152] The forward model for source localization is used to characterize the physical relationship between the brain source space to be estimated and the EEG data mapped through the head model.

[0153] Among them, dots represent brain electrodes, which are used to collect brain electrical data. The electrode layout includes, but is not limited to, the number and location of brain electrodes.

[0154] Based on the electrode layout of the EEG acquisition device, the head model of the target user, and the brain source space to be estimated, a forward model for source localization is determined.

[0155] The electrode layout is based on the following formula:

[0156]

[0157] Among them, e1…e n This indicates the spatial location information of several electrodes in the EEG.

[0158] Through the above steps, the layout of the electrodes in the EEG acquisition device is registered with the head model space, establishing how the source space to be estimated is mapped onto the electrodes. This establishes an objective physical model between the electrode layout and the head model space. Based on this objective physical model, the corresponding source space data can be estimated using the acquired EEG data.

[0159] In some embodiments, the MRI structural image acquisition step of the target user includes:

[0160] First, check if the target user has their own MRI structure image. If the target user's own MRI structure image is found, the found MRI structure image is used as the target user's MRI structure image. If the target user's own MRI structure image is not found, obtain the target user's user feature vector. Match the target user's user feature vector with the standard feature vector in the standard brain database, and use the MRI structure image corresponding to the standard brain with the highest matching degree as the target user's MRI structure image.

[0161] Specifically, if the target user has their own MRI structural image data, their own MRI structural image is used directly; otherwise, the standard brain with the highest similarity is selected from the standard brain database as the standard brain used by the target user. Based on the obtained standard brain or the user's own MRI structural image, the brain source space to be estimated is constructed.

[0162] In some embodiments, the user feature vector and the standard feature vector include the user's demographic information and head circumference data.

[0163] The user's demographic information and head circumference data are collected to obtain a feature vector specific to that user. The feature vector includes parameters such as age, gender, ethnicity, and head circumference, as shown below:

[0164]

[0165] Establish a standard brain bank, including standard brains of different genders, ages, and races (MNI152, fsaverage, UNCinfant, NKI…) (see…) Figure 5 For each standard brain record, demographic information and head model data of the standard brain are constructed, resulting in a feature vector for each standard brain, as shown below:

[0166]

[0167] Here, T refers to the standard brain bank, which includes feature vectors of different users.

[0168] The user's feature vector is compared with the feature vector of each standard brain in the constructed standard brain database, and the standard brain with the highest similarity is selected as the standard brain used by the user. If the user has their own MRI structural image data, there is no need to match in the standard brain database, and their own MRI structural image is used directly. The formula for matching the standard brain with the highest similarity to the target user in the standard brain database is as follows:

[0169]

[0170] Where argmax represents solving for similarity, f user f represents the feature vector of the target user. template This represents the feature vector of the standard brain in the standard library.

[0171] It should be noted that the aforementioned standard brain database includes standard brains of different genders, ages, and ethnicities. For each standard brain, demographic information and head model data are recorded to construct its feature vector. For target users without their own MRI images, demographic information and head circumference data are collected to obtain a feature vector specific to that user. This feature vector is then compared with the feature vectors of every standard brain in the standard brain database, and the standard brain with the highest similarity is selected as the standard brain used by that user.

[0172] In the above embodiments, by establishing a standard brain database and a matching algorithm, the most suitable standard brain template is provided for users without MRI structural images, which has the advantage of reducing costs while maximizing source localization accuracy.

[0173] In some embodiments, EEG data to the target user's brain region of interest is determined based on the source estimation results of the source space and the target user's brain region of interest.

[0174] In some embodiments, the examination purposes of different target users are different. It may be a health check or a targeted examination for a specific cause of disease, so as to achieve the purpose of cognitive improvement or intervention of pathological state. For example, the purpose of neurotraining may include attention deficit hyperactivity disorder, depression, autism, sleep disorders, etc.

[0175] Depending on the training objective, the target users will focus on different brain regions of interest. Therefore, by extracting EEG data of the brain regions corresponding to the training objective and feeding it back to the user, the user can understand the activity status of the corresponding brain regions and then further intervene, adjust or train the brain regions in a targeted manner to achieve the purpose of neurofeedback.

[0176] In some embodiments, determining the target user's brain regions of interest includes:

[0177] Step 1: Obtain a brain segmentation atlas library, which includes brain segmentation atlases that divide the brain into different regions based on one or more principles.

[0178] Step 2: Obtain a whole-brain activation map, which is used to characterize the association between the activated brain regions and the neural training target.

[0179] Step 3: Based on the target user's neural training goals, match the whole-brain activation map with the brain region segmentation map in the brain region segmentation map library, obtain the brain region with the highest activation level in the brain region segmentation map, and select the brain region with the highest activation level as the target user's brain region of interest.

[0180] In some embodiments, the brain can be divided into different brain region segmentation maps based on different segmentation principles. For example, brain region segmentation maps can be divided based on brain structure, or brain structure can be divided into different brain region segmentation maps based on the principle of dividing brain functional areas.

[0181] The cortex was divided into several functional areas based on brain region segmentation maps. The formula is shown below:

[0182]

[0183] Wherein, the formula represents the brain region segmentation atlas library L atlas Different brain region segmentation maps in the image represent R1...R k .

[0184] The whole-brain activation map is used to characterize the activation status of brain source points corresponding to the neural training goals, with different neural training goals corresponding to different activation status of brain source points.

[0185] The whole-brain activation map represents the activated states of the entire brain. Within the whole-brain activation map, the association between activated source points and neural training goals can be found. For example, if the neural training goal is to improve attention, the corresponding activated source point in the whole-brain activation map will be the first source point region; if the neural training goal is to overcome sleep disorders, the corresponding activated source point in the whole-brain activation map will be the second source point region.

[0186] The whole-brain activation map can reveal the source data of activated points in the brain under different neural training goals.

[0187] After determining the whole-brain activation map corresponding to the neural training goal, the process involves matching the whole-brain activation map with brain region segmentation maps in a brain region segmentation map library. This allows for the identification of the brain region with the highest activation level within the segmentation map, which is then selected as the target user's region of interest. In other words, the process ultimately obtains the brain region segmentation map most relevant to the target user's training objective. This map can then be used as the target user's region of interest for this neural training session. By focusing on the neural feedback parameters of this specific brain region, the neural feedback training goal can be achieved. Furthermore, this process converts source point data associated with the neural training goal into specific brain regions. Since source point data can be very large, converting it into brain regions reduces the data's dimensionality and allows the user's attention to be focused on a specific area rather than scattered source points, thus improving the effectiveness of the neural feedback intervention.

[0188] In some embodiments, the whole-brain activation map includes a region composed of one or more activated source points. The region composed of the source points is matched with the brain region segmentation map, and the brain region with the highest matching degree in the brain region segmentation map is selected as the brain region of interest for the target user.

[0189] It should be noted that the activated source points in the whole-brain activation atlas may be scattered across multiple regions, and these regions may differ in size and shape. By matching one or more activated source points from the whole-brain activation atlas with brain region segmentation maps in a brain region segmentation atlas library, the brain region with the highest activation level (the brain region corresponding to the brain region segmentation map) can be matched as the user's region of interest based on the principle of maximum matching. Among the selected regions of interest, these are the regions with the highest relevance to the neural training target and the regions with the highest activation level during training.

[0190] In some embodiments, the step of matching the whole-brain activation map with brain region segmentation maps in the brain region segmentation map library based on the target user's neural training goals, obtaining the brain region with the highest activation level in the brain region segmentation map, and selecting the brain region with the highest activation level as the target user's brain region of interest includes: determining the whole-brain activation map corresponding to the neural training goals based on meta-analysis; matching the whole-brain activation map with brain region segmentation maps in the brain region segmentation map library, obtaining the brain region with the highest activation level in the brain region segmentation map, and selecting the brain region with the highest activation level as the target user's brain region of interest.

[0191] In other words, the brain regions of interest to the user are determined from a brain region segmentation atlas database. Specifically, this determination method involves matching the activated source points in the whole-brain activation atlas with the maximum values ​​in the brain region segmentation atlas database. The specific formula is as follows:

[0192]

[0193] Among them, a j R represents the average activation intensity of the j-th brain region; j S represents the j-th brain region, which is composed of several source points in the source space; i A represents the i-th source point in a certain brain region; F This represents the meta-analysis activation map obtained from neurosynth; j* represents the index of the most activated brain region found among all brain regions.

[0194] In the above embodiments, a brain segmentation atlas is established and combined with functional magnetic resonance imaging (fMRI) analysis results to assist in localization, providing the most suitable brain functional area selection for the specified cognitive function and user, which has the advantage of providing the optimal selection for brain functional areas.

[0195] Real-time source localization during EEG data acquisition provides users with neural feedback indicators for specific brain functional areas (regions of interest), which has the advantages of improving the accuracy of neural feedback and being adaptable to different cognitive functions and individuals.

[0196] In some embodiments, based on EEG data of brain regions of interest to the target user, feature values ​​of the brain regions of interest are determined as neural feedback indicators of user attention.

[0197] During the EEG data acquisition process, real-time source localization is performed to provide users with EEG data of specific brain functional areas (regions of interest), and the feature values ​​of these regions of interest are calculated. These feature values ​​are then used as the neural feedback indicators that users are interested in. Users can achieve neural adjustment and training by simply focusing on the neural feedback indicators of these local regions of interest.

[0198] The characteristic value can be the peak value or the mean value, etc., and there are no restrictions.

[0199] In the above embodiments, the collected EEG data of the user is converted into source estimation results of the source space. Considering that the data in the source estimation results of the source space exists in the form of source points, the data volume is large and cannot locate specific brain regions. Since neuromotor activity is associated and linked with specific brain regions, this application uses meta-analysis to determine the brain region of interest corresponding to the current neural training target, and then converts the source estimation results of the user's source space into EEG data of the specific brain region. The user only needs to focus on the data changes of that brain region to achieve neural feedback training.

[0200] In some embodiments, neurofeedback involves providing users with specific brain activity waveforms for neural adjustment training. However, these waveforms are often abstract and not intuitive, making it difficult for users to directly perceive their own brain activity and hindering neural training. In the above embodiments, the waveforms are visually represented as a feature of a brain region of interest (BRIE), and the feature value of that region is calculated as the neurofeedback indicator the user needs to focus on. This allows users to intuitively understand their own brain activity based on the feature value of the BRIE, enabling targeted neural training and improving its effectiveness.

[0201] Electroencephalography (EEG), as a non-invasive brain function detection technology, has been widely used in neuroscience, medical diagnosis, and brain-computer interfaces due to its high temporal resolution, low cost, and ease of operation. Neurofeedback, as a special form of biofeedback training, generally calculates various brain activity indicators based on real-time collected EEG signals as control feedback signals. Users can observe these feedbacks and autonomously adjust their cognition and brain activity state to achieve cognitive improvement or intervention in pathological states. Currently, there are many applications of EEG-based neurofeedback methods, such as for intervention in attention deficit hyperactivity disorder, depression, autism, and sleep disorders, but the following limitations exist: (1) Limited feedback positioning accuracy, most neurofeedback systems are based on frequency / power feedback through electrode channels (such as C3, C4, etc.), resulting in poor spatial resolution; (2) Lack of specificity in feedback, neurofeedback usually cannot be clearly mapped to specific brain functional areas, making it difficult to further refine training objectives; (3) Single feedback dimension, ignoring the spatiotemporal dynamic changes of neural networks in brain activity.

[0202] To address the insufficient spatial resolution of traditional EEG-based neurofeedback methods, some embodiments may employ EEG source localization to estimate brain source space activity, thereby improving spatial resolution.

[0203] Brain source localization technology combines structural information from magnetic resonance imaging (MRI) with mathematical modeling and inversion of multi-channel EEG data to estimate neural source activity in the cerebral cortex and even subcortical regions. The core idea of ​​brain source localization is that, due to the volume conduction effect, EEG signals measured on a limited number of electrodes on the scalp are only a mixed representation of neural source activity. By constructing a forward model and an inverse solution model, the EEG signals are restored to neural source activity in a given source space. After source localization, the spatial resolution of the EEG data is improved, and the temporal state of specific brain functional areas (such as the lateral occipital cortex, medial occipital cortex, or ventral occipital cortex) can be reconstructed. Neurofeedback based on brain source localization can significantly improve the accuracy of neurofeedback, allow for adjustments based on specific cognitive functions and users, and expand the dimensions of neurofeedback indicators.

[0204] In some embodiments, the method further includes monitoring the user's EEG data via a non-contact EEG device, such as for target users who are infants, young children, or vegetative state patients who cannot control their bodies. The non-contact EEG acquisition device is not directly worn on the target user's head, allowing for more flexible data acquisition, and the device's position can adaptively adjust when the target user's head posture changes.

[0205] In this way, for infants and young children who need neurological training, real-time acquisition of EEG data can be achieved through non-contact EEG acquisition devices. Then, EEG data of brain regions of interest and corresponding feature values ​​can be extracted from the acquired EEG data as indicators for neural feedback.

[0206] A non-contact EEG signal acquisition method is adopted, in which the EEG electrodes are not directly attached to the scalp during the EEG signal acquisition process. The EEG signals of the monitored brain regions are accurately monitored by adjusting the position and posture of the EEG electrodes.

[0207] In some embodiments, the step of monitoring a user's EEG data using a non-contact EEG device includes: Step 1, acquiring the head image of the target user at the current moment.

[0208] In some embodiments, target users may include infants or other users who require EEG monitoring.

[0209] In some embodiments, the head image at the current moment may include one acquired in real time by a depth sensor or other image acquisition device.

[0210] In some embodiments, the image acquisition device can be fixed to a bracket. The bracket is immovable, but the orientation of the image acquisition device can be adjusted.

[0211] Step 2: Determine the head pose of the target user at the current time based on the head image corresponding to the current time.

[0212] Pose includes position and orientation.

[0213] In some embodiments, after acquiring the head image of the target user at the current moment through an image acquisition device, the method further includes performing pose analysis on the head image to determine the head position and head direction of the target user at the current moment.

[0214] Step 3: Based on the head position at the current moment and the head position at the previous moment, determine the pose of the EEG cap, wherein the EEG cap includes multiple EEG electrodes, and the multiple EEG electrodes monitor the EEG data of the target user under the pose corresponding to the EEG cap.

[0215] The EEG cap is similar to a helmet, and the EEG electrodes can be placed inside the helmet. Each EEG electrode can move horizontally and vertically.

[0216] In some embodiments, the previous time step is earlier than the current time step, and the head pose corresponding to the previous time step is the historical head position and orientation calculated based on the head image acquired at the previous time step.

[0217] In some embodiments, the pose of the EEG cap can be determined based on the current head pose and the previous N historical head poses. The EEG cap includes multiple EEG electrodes used to monitor the target user's EEG data. N is an integer greater than or equal to 1.

[0218] In some embodiments, the target user may include infants and young children. When collecting brain data from infants and young children, their heads are prone to uncontrolled movement, which may cause the relative position between the EEG electrodes and the infant's scalp to shift, resulting in unstable signal acquisition. Therefore, during EEG monitoring, it is necessary to continuously acquire images of the infant's head. Based on the acquired head image at the current moment, the infant's head position and orientation at that moment are determined. The current head posture is compared with historical postures from one or more previous moments to assess the extent of changes in the infant's head posture, thereby determining whether the position and orientation of the EEG electrodes need to be adjusted to ensure the accuracy of the monitoring of the infant's brain data.

[0219] In the above embodiments, when the target user is a newborn or infant, a non-contact EEG monitoring method can be used to avoid irritation and damage to the skin of the newborn or infant. This non-contact method prevents the EEG cap from directly contacting the infant's head. However, when the infant's head moves, the EEG electrodes may easily deviate from the monitoring area, leading to inaccurate EEG data monitoring.

[0220] The EEG monitoring method provided in this application can calculate the similarity between the current head pose and the previous N historical head poses based on the current head pose. This helps determine whether the EEG cap needs to be adjusted and how to adjust its pose, ensuring that the EEG electrodes can continuously and accurately monitor the brain region, thereby continuously and stably monitoring the target user's EEG signals.

[0221] In some embodiments, determining the pose of the EEG cap based on the head pose at the current moment and the head pose at the previous moment includes:

[0222] Step 1: Determine the distance between the head pose at the current moment and the head pose at the previous moment.

[0223] Step two: In response to the distance being greater than a preset threshold, the current pose of the EEG cap is adjusted based on the head pose at the current moment, and the EEG electrodes in the EEG cap continue to monitor the EEG data of the target user in the adjusted pose.

[0224] Step 3: In response to the distance not being greater than a preset threshold, the EEG electrodes in the EEG cap continue to monitor the target user's EEG data in the current pose.

[0225] In some embodiments, the pose of the EEG cap can be determined based on the current head pose and the previous N historical head poses, including: determining the distance between the current head pose and the previous N head poses. If the distance is greater than a preset threshold, it indicates that the target user's head pose has changed significantly, and the pose of the EEG cap needs to be adjusted synchronously. Therefore, the pose of the EEG cap can be adjusted based on the current head pose, and the head EEG data monitoring of the target user can continue based on the adjusted EEG cap pose, thereby ensuring that the pose of the EEG cap always remains consistent with the target user's head pose.

[0226] In the above embodiments, EEG monitoring of a specific area of ​​the target user's head is performed using EEG electrodes in the EEG cap. A pre-set EEG electrode deviation threshold is established, and the distance between the current head pose and the previous N head poses is calculated in real time. If the distance is greater than the preset threshold, the EEG cap pose is adjusted based on the previous pose, allowing the EEG electrodes in the cap to continuously and accurately monitor the specific area of ​​the target user's head. If the distance is less than or equal to the preset threshold, it indicates that the target user's pose is consistent or substantially consistent between the current and previous moments. In this case, no adjustment to the EEG cap pose or the EEG electrodes is needed at the current moment, and EEG monitoring can continue based on the previous EEG cap pose. At the next moment, the target user's head pose can be obtained again through sensors, and the distance between the current head pose and the previous N head poses is repeatedly determined. The magnitude of this distance relative to the preset threshold is then assessed, and the result determines whether to adjust the EEG pose.

[0227] Understandably, since the EEG electrodes are placed inside the EEG cap, adjusting the position of the EEG cap also adjusts the position of the EEG electrodes.

[0228] In some embodiments, adjusting the current pose of the EEG cap based on the head pose at the current moment includes: using the head pose of the target user at the current moment as the control target of the robotic arm, the robotic arm controls and adjusts the current position and current direction of the EEG cap so that the EEG cap moves to a pose that matches the head position and head direction of the target user at the current moment, and the EEG electrodes in the EEG cap collect the EEG data of the target user at the adjusted pose.

[0229] Specifically, the EEG cap can be placed at the end of a robotic arm, and the robotic arm controls the cap's pose. When a significant change in the target user's head pose is detected, the robotic arm synchronously adjusts the cap's pose to match the target user's head pose at that moment, enabling continuous and accurate detection of the target user's EEG data.

[0230] In some embodiments, the step of determining the head pose of the target user includes: determining the head edge point coordinate set of the target user based on the head image; and determining the head pose of the target user based on the head edge point coordinate set.

[0231] In some embodiments, the weighted head image can be segmented based on an image segmentation algorithm to remove irrelevant information and obtain a target image. Then, an edge detection algorithm is used to detect the edges of the target image to obtain a set of edge point coordinates of the head image. The head pose of the target user can be determined based on the set of head edge point coordinates.

[0232] It is understandable that edge detection algorithms and image segmentation algorithms can be existing, mature algorithms, and no restrictions are imposed here.

[0233] In some embodiments, the method further includes: determining the head size of the target user based on the head image of the target user; determining the scaling factor of the EEG cap and the scaling factor of the arrangement of brain electrodes in the EEG cap based on the head size of the target user; adjusting the size of the EEG cap based on the scaling factor; and adjusting the coordinate position layout of the brain electrodes in the EEG cap based on the arrangement scaling factor.

[0234] It is understandable that different target users have different head sizes, so the size requirements for EEG caps also vary. It is very important to set an EEG cap that matches the head size of the corresponding target user, as this is the basic condition for obtaining accurate monitoring data.

[0235] In some embodiments, the method further includes determining the head size of the target user based on the target user's head image, and then adjusting the size of the EEG cap in a personalized and adaptive manner based on the target user's head size, so that the size of the EEG cap matches the head size of the target user.

[0236] The brain electrodes are placed inside the EEG cap, and the positions of each brain electrode inside the EEG cap can be changed. When the size of the EEG cap changes, the positions of each brain electrode inside the EEG cap will also change adaptively.

[0237] In some embodiments, the relative positional relationship of the brain electrodes in the EEG cap is fixed. The initial layout coordinates of the brain electrodes in the EEG cap can be obtained first, and the target coordinates of the brain electrodes can be obtained by scaling the spacing between the brain electrodes based on the scaling factor of the EEG cap.

[0238] In this way, the head size of the target user is determined based on the head image of the target user, and then the size of the EEG cap and the position of the brain electrodes in the EEG cap are adjusted accordingly, so that the final adjusted EEG cap and brain electrodes match the target user, so as to obtain more accurate brain data.

[0239] In some embodiments, the head size of the target user includes the head's lateral dimensions and the head's vertical dimensions;

[0240] The head size determination step for the target user includes: determining the ear key points, nose key points, and head key points of the target user based on the head image, wherein the ear key points include the left ear key point and the right ear key point; determining the horizontal head size of the target user based on the ear key points and the head key points; and determining the vertical head size of the target user based on the nose key points and the head key points.

[0241] In some embodiments, head size can be determined based on head key points. For example, after obtaining a head image based on a sensor, a trained key point detection model is used to output the coordinates of head key points, nose key points, left ear key points, and right ear key points.

[0242] By fitting the coordinates of the left ear keypoint concave point and the right ear keypoint with the coordinates of the head edge point set, a curve is fitted to the top of the head, and the curve length is calculated as the lateral dimension of the head.

[0243] By calculating the distance between the key point of the nose and the midpoint of the curve at the top of the head along the surface of the head, and multiplying it by 2, the longitudinal dimension of the head can be obtained.

[0244] The target user's head size is determined based on the target user's horizontal and vertical head dimensions, so as to determine the size of the EEG cap and the position coordinate arrangement of the EEG electrodes according to the head size.

[0245] In some embodiments, the method further includes acquiring a head image of the target user based on a depth sensor;

[0246] The step of determining the shooting pose of the depth sensor includes: acquiring an image of the working plane corresponding to the working plane and a plane normal vector corresponding to the working plane based on the depth sensor, wherein the working plane includes a plane corresponding to the working platform used to monitor the EEG data of the target user; and determining the shooting position and shooting direction of the depth sensor based on the on-site observation distance, the working distance of the depth sensor, and the plane normal vector.

[0247] In some embodiments, a depth sensor can be used to capture an image of the target user's head, and then the head image can be used to calculate the target user's head pose and head size.

[0248] In some embodiments, the sensor pose setting is very important. If the sensor pose setting is not reasonable, accurate head pose and head size data cannot be obtained from the captured head images, which may lead to unreasonable brain electrode monitoring pose and inaccurate brain data.

[0249] In some embodiments, the method further includes setting the sensor's shooting pose. First, the sensor acquires an image of the working plane corresponding to the working plane, and then determines the sensor's shooting pose based on the working plane image.

[0250] In some embodiments, after obtaining the working plane image corresponding to the working plane, the method further includes: extracting multiple sets of internal coordinate points from the working plane image based on a preset marker template; converting the coordinate system of the multiple sets of internal coordinate points to the world coordinate system; and determining the plane normal vector corresponding to the working plane based on the sets of internal coordinate points in the world coordinate system.

[0251] Specifically, the target user being inspected lies flat on the work platform, and the work platform is photographed using an RGBD sensor to obtain an image of the work plane. Using a 2D edge detection / image segmentation algorithm, combined with a marker template of known shape and size, the coordinates of the internal points of n markers on the work plane and the corresponding depth information are obtained. In some embodiments, the coordinates of these points can be recorded as C_POINTS(camera_set_1,camera_set_2,…,camera_set_n). Then, the 3D coordinates in the camera coordinate system are transformed to the world coordinate system W_POINTS(world_set_1,world_set_2,…,world_set_n) of the support base using extrinsic parameters.

[0252] Using the coordinate information of all points in W_POINTS, a plane fitting is performed to obtain the plane normal vector N = [nx, ny, nz]. The optimal observation distance d is set based on the sensor's working distance and field of view. The center position P_c = [x_c, y_c, d_c] of the working plane is calculated based on the set of marked points. Therefore, the sensor's shooting position can be P1 = P_c + N*d, with the direction perpendicular to the working plane of the platform. Then, based on the determined position and direction, the target pose of the sensor is determined. The robotic arm adjusts the sensor to this target pose to acquire head data from the target user.

[0253] like Figure 6 As shown, this application also provides a non-contact EEG monitoring system, the system comprising: a support module 402, a robotic arm module 401, an image acquisition module 403, a work platform module 405, an EEG cap module, and a control module 406;

[0254] The image acquisition module 403 is mounted on the bracket module 402 and is used to acquire the head image of the target user and / or the work platform image corresponding to the work platform module 405.

[0255] The EEG cap module is installed at the end of the robotic arm 401. The robotic arm 401 is used to adjust the position of the EEG cap. The EEG electrodes in the EEG cap are used to monitor the EEG data of the target user.

[0256] The work platform module 405 is used to carry the target user;

[0257] The control module 406 controls the robotic arm 401 to adjust the position of the EEG cap based on the head position of the target user. The EEG electrodes in the EEG cap are used to monitor the EEG data of the target user.

[0258] The end effector 404 of the robotic arm is used to hold the EEG cap, and the EEG electrodes in the cap are used to collect EEG data from the target user.

[0259] The control module 406 is used to calculate pose information, size information, etc. based on the acquired image data, and to control the movement of the robotic arm to adjust the pose of the EEG cap based on the calculated pose information, and to control the size of the EEG cap to adjust the size based on the calculated size information to adapt to the head size of the target user.

[0260] It should be noted that, Figure 6 This is merely an example architecture of the non-contact EEG monitoring system provided in this application, and this example architecture does not constitute a limitation on the system architecture of this application.

[0261] Those skilled in the art will understand that Figure 7The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0262] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the methods described above. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, or optical storage, etc. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc. The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0263] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. A three-dimensional motor intention decoding method based on electroencephalography source localization, characterized in that, The method includes: Acquire EEG data of target users during three-dimensional motion visualization, wherein the three-dimensional motion visualization includes the visualization of three-dimensional limb movements; Based on the EEG data of the target user during three-dimensional motor imagination, the source estimation data of the whole brain is determined, the motor brain regions associated with the three-dimensional motor imagination are obtained, and the source estimation data of the motor brain regions is determined based on the source estimation data of the whole brain and the motor brain regions of the brain, wherein the motor brain regions are some brain regions among all brain regions of the whole brain. The determination of source estimation data for the whole brain includes: acquiring the EEG data of the target user, solving the EEG data based on the inverse solution model and the forward model for source localization, and determining the source estimation result of the source space corresponding to the EEG data; wherein the construction step of the forward model for source localization includes: acquiring the head structure image corresponding to the target user, constructing the head model of the target user and the brain source space to be estimated based on the head structure image of the target user, and determining the forward model for source localization based on the electrode layout of the EEG acquisition device, the head model of the target user, and the brain source space to be estimated, wherein the forward model for source localization is used to characterize the physical relationship of the brain source space to be estimated mapped to the EEG data through the head model; By combining the source estimation data of the motor brain region and the action coordinate model of the target user's three-dimensional motor imagination, the coordinate information corresponding to the target user's three-dimensional motor imagination is decoded; wherein, the coordinate information serves as an indication of the movement of the external device.

2. The method of claim 1, wherein, The steps for determining the motion coordinate model include: Acquire the EEG data of the user during three-dimensional motion imagination, wherein the three-dimensional motion imagination includes imagining that the limbs are performing three-dimensional spatial movements; Based on the EEG data of the user during three-dimensional motor imagery, the source estimation data of the motor brain region of the user during each three-dimensional motor imagery are determined. Based on the user's three-dimensional motion imagination and the source estimation data of the motor brain region during three-dimensional motion imagination, the action coordinate model is trained; wherein, the action coordinate model is the mapping relationship between the neural activity of the motor brain region and the external space.

3. The method of claim 1, wherein, The motion coordinate model is used to represent the motion coordinates of three-dimensional motion imagination in three-dimensional space.

4. The method of claim 1, wherein, The motion coordinate model includes multiple motion coordinate sub-models representing the movement of the three-dimensional motion imagination in different dimensional spaces. Each of the motion coordinate sub-models represents the movement coordinates of the motion imagination in different dimensional directions in the external space.

5. The method of claim 4, wherein, The motion coordinate sub-model for imagining movement in different dimensional spaces in three-dimensional motion includes at least one or more of the following: Action coordinate sub-models at different depth directions; and / or, A sub-model of motion coordinates in the left and right directions of a two-dimensional plane; and / or, A sub-model of motion coordinates in the vertical direction of a two-dimensional plane.

6. The method according to claim 4, characterized in that, The motion coordinate sub-models for the three-dimensional motion imagination to move in different dimensional spaces include motion coordinate sub-models in different depth directions, motion coordinate sub-models in the left and right directions of the two-dimensional plane, and motion coordinate models in the up and down directions of the two-dimensional plane. The step of combining the source estimation data of the motor brain region and the action coordinate model of the target user's three-dimensional motor imagination to decode the coordinate information corresponding to the target user's three-dimensional motor imagination includes: Based on the source estimation data of the motor brain region and the action coordinate sub-model in different depth directions, the depth coordinate information corresponding to the target user's three-dimensional motion imagination is decoded. Based on the source estimation data of the motor brain region and the action coordinate sub-model in the left and right directions of the two-dimensional plane, the left and right coordinate information corresponding to the target user during three-dimensional motor imagination is decoded. Based on the source estimation data of the motor brain region and the action coordinate sub-model in the vertical direction of the two-dimensional plane, the vertical coordinate information corresponding to the target user during three-dimensional motor imagination is decoded. The coordinate information corresponding to the target user's three-dimensional motion imagination is determined based on the depth coordinate information, the left and right coordinate information, and the up and down coordinate information.

7. The method according to claim 6, characterized in that, The method further includes: Using the three-dimensional coordinate information in the coordinate information as the target position of the external device, the external device is controlled to move the target user's limbs to the target position; and / or... Using the depth coordinates from the coordinate information as the target position of the external device, the external device is controlled to move the target user's limbs to the target position; and / or... Using the vertical coordinates from the coordinate information as the target position of the external device, the external device is controlled to move the target user's limbs to the target position; and / or... The left and right coordinates in the coordinate information are used as the target position of the external device, and the external device is controlled to move the target user's limbs to the target position.

8. The method according to claim 1, characterized in that, The method further includes: The acquisition of the target user's EEG data includes acquiring the user's EEG data through a contact-type EEG acquisition device or monitoring the user's EEG data through a non-contact EEG device; Monitoring users' brainwave data through non-contact EEG devices, including: Obtain the head image of the target user at the current moment; Determine the head pose of the target user at the current time based on the head image corresponding to the current time; Based on the head pose at the current moment and the head pose at the previous moment, the pose of the EEG cap is determined, wherein the EEG cap includes multiple EEG electrodes, and the multiple EEG electrodes monitor the EEG data of the target user under the pose corresponding to the EEG cap.

Images