Brain region signal reconstruction method and device, electronic equipment and readable storage medium

Abstract

The embodiment of the invention provides a brain region signal reconstruction method and device, electronic equipment and a readable storage medium. The brain region signal reconstruction method comprises the following steps: acquiring an electroencephalogram signal of an alternative brain region of a brain injury patient; inputting the electroencephalogram signal of the replacement brain region into a first generation model to obtain a first electroencephalogram signal, output by the first generation model, of a target brain region of the brain injury patient; inputting the electroencephalogram signal of the replacement brain region into a second generation model to obtain a predicted electroencephalogram signal of a target brain region of the healthy subject output by the second generation model, and taking the predicted electroencephalogram signal as a second electroencephalogram signal of the target brain region of the brain injury patient; the electroencephalogram information precision of the first electroencephalogram signal and the electroencephalogram information precision of the second electroencephalogram signal are compared, and the electroencephalogram signal with the high electroencephalogram information precision serves as the electroencephalogram signal of the target brain area of the brain injury patient. According to the embodiment of the invention, the electroencephalogram information of the target brain region of the brain injury patient can be reconstructed, so that the brain injury patient can also use the interbrain interface technology to assist in completing the purposes of rehabilitation, communication, equipment control and the like.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of brain-computer interface systems, and in particular to a brain region signal reconstruction method and device, electronic equipment and a readable storage medium. BACKGROUND

[0002] Brain-Computer Interface (BCI) technology is a technology that realizes human-computer interaction by analyzing electroencephalogram signals, and has been widely applied in medical rehabilitation, auxiliary communication and other fields. With the development of neuroscience and artificial intelligence technology, brain-computer interface systems have shown great potential in the rehabilitation of brain injury patients and auxiliary communication. At present, brain-computer interface systems mainly collect electroencephalogram signals of specific brain regions, process and decode the signals to realize control or information transmission to external devices.

[0003] Traditional brain-computer interface systems usually include core components such as electroencephalogram signal acquisition module, signal processing module and decoding module. Among them, the electroencephalogram signal acquisition module is responsible for collecting the electroencephalogram signals of the user; the signal processing module preprocesses, extracts features and classifies the collected electroencephalogram signals; the decoding module converts the processed signals into control instructions or other forms of output. This system architecture has been widely applied in various brain-computer interface systems, such as CN118535892A discloses a brain-computer interface system, which collects electroencephalogram signals through the acquisition module, detects and identifies the preprocessed electroencephalogram signals through the signal processing module, reads the neural activity state information and neural activity regulation mode and neural activity intention information through the collected electroencephalogram signals, and improves the effectiveness of brain-computer interface information analysis.

[0004] In the field of rehabilitation training, brain-computer interface systems based on motor imagery are widely studied. CN113398422A discloses a rehabilitation training system based on motor imagery-brain-computer interface and virtual reality, which uses electroencephalogram signal acquisition equipment to collect electroencephalogram signals of subjects in real time during rehabilitation, and preprocesses and extracts features of the electroencephalogram signals to establish a motor imagery intention recognition model to assist in completing user rehabilitation training. This type of system usually needs to arrange electrodes in specific brain regions to collect high-quality electroencephalogram signals related to motor imagery.

[0005] With the development of deep learning technology, the decoding ability of electroencephalogram signals has been significantly improved. CN117130490A discloses a brain-computer interface control system, which includes an electroencephalogram data acquisition module, a saliency feature screening and verification module, a speech imagination electroencephalogram signal decoding module and an understandable speech synthesis module, which can effectively improve the effect of communication between speech disorder patients and the outside world by collecting electroencephalogram signals during speech imagination. Specifically, the electroencephalogram signals are decoded by a deep learning algorithm to realize the conversion from electroencephalogram signals to speech information.

[0006] In the field of multi-modal brain-computer interface, WO2023246226A1 discloses a brain-computer interface-based computing system, which includes a plurality of acquisition devices, an external control circuit of a neuromorphic chip, a neuromorphic chip, and a feedback device, and can efficiently and low-delay implement a multi-modal brain-computer interface algorithm. The system improves the performance and stability of the brain-computer interface system by integrating multiple physiological signals.

[0007] In the field of visual image reconstruction, CN111539331A discloses a visual image reconstruction system based on a brain-computer interface, which includes an electroencephalogram signal acquisition subsystem, an electroencephalogram signal feature extraction and classification subsystem, an image encoder, an image decoder, a discriminator, an electroencephalogram feature mapper, and an image quality evaluation subsystem, and can realize real-time image reconstruction based on electroencephalogram signals. The system converts the electroencephalogram signal feature vector into a feature vector in the latent space of the image decoder through an electroencephalogram feature mapping network, and realizes the reconstruction from the electroencephalogram signal to the visual image.

[0008] However, although brain-computer interface technology is diverse in form, direct acquisition of electroencephalogram signals in specific brain regions is still an indispensable step in current implementation of human-computer interface technology; that is, in the application process of the above brain-computer interface, the electroencephalogram signals of the target subject need to be directly acquired. However, for brain injury patients, due to the inability to arrange electroencephalogram measurement electrodes in the damaged brain region, high-quality electroencephalogram signals related to specific functions cannot be acquired, and thus brain-computer interface technology based on corresponding electroencephalogram signals cannot be used. SUMMARY

[0009] In view of the above problems, the embodiments of the present application are proposed to provide a brain region signal reconstruction method which overcomes the above problems or at least partially solves the above problems, can reconstruct the electroencephalogram information of the target brain region of the brain injury patient, and further supports the brain injury patient to use the brain interface technology to assist in completing the purposes of rehabilitation, communication, device control, etc.

[0010] Correspondingly, the embodiments of the present application also provide a brain region signal reconstruction device, an electronic device, and a readable storage medium to ensure the implementation and application of the above method.

[0011] In a first aspect, the embodiments of the present application disclose a brain region signal reconstruction method, which comprises: acquiring electroencephalogram signals of a substitute brain region of a brain injury patient; inputting the electroencephalogram signals of the substitute brain region of the brain injury patient into a first generation model to obtain first electroencephalogram signals of a target brain region of the brain injury patient output by the first generation model; wherein the first generation model is trained by electroencephalogram signal samples of the target brain region and electroencephalogram signal samples of the substitute brain region of a healthy subject. input the electroencephalogram signal of the alternative brain area of the brain injury patient into a second generation model to obtain a predicted electroencephalogram signal of the target brain area of the healthy subject output by the second generation model, and take the predicted electroencephalogram signal as a second electroencephalogram signal of the target brain area of the brain injury patient; wherein the second generation model is trained according to the electroencephalogram signal sample of the alternative brain area of the brain injury patient and the electroencephalogram signal sample of the target brain area of the healthy subject; compare the electroencephalogram information accuracy of the first electroencephalogram signal and the second electroencephalogram signal, in the case that the electroencephalogram information accuracy of the first electroencephalogram signal is higher than that of the second electroencephalogram signal, take the first electroencephalogram signal as the electroencephalogram signal of the target brain area of the brain injury patient; in the case that the electroencephalogram information accuracy of the second electroencephalogram signal is higher than that of the first electroencephalogram signal, take the second electroencephalogram signal as the electroencephalogram signal of the target brain area of the brain injury patient.

[0012] In a second aspect, the present application discloses a brain area reconstruction device, which comprises: an electroencephalogram acquisition module, configured to acquire an electroencephalogram signal of an alternative brain area of a brain injury patient; a first generation module, configured to input the electroencephalogram signal of the alternative brain area of the brain injury patient into a first generation model to obtain a first electroencephalogram signal of a target brain area of the brain injury patient output by the first generation model; wherein the first generation model is trained by an electroencephalogram signal sample of a target brain area of a healthy subject and an electroencephalogram signal sample of an alternative brain area; a second generation module, configured to input the electroencephalogram signal of the alternative brain area of the brain injury patient into a second generation model to obtain a predicted electroencephalogram signal of the target brain area of the healthy subject output by the second generation model, and take the predicted electroencephalogram signal as a second electroencephalogram signal of the target brain area of the brain injury patient; wherein the second generation model is trained according to the electroencephalogram signal sample of the alternative brain area of the brain injury patient and the electroencephalogram signal sample of the target brain area of the healthy subject; a target determination module, configured to compare the electroencephalogram information accuracy of the first electroencephalogram signal and the second electroencephalogram signal, in the case that the electroencephalogram information accuracy of the first electroencephalogram signal is higher, take the first electroencephalogram signal as the electroencephalogram signal of the target brain area of the brain injury patient; in the case that the electroencephalogram information accuracy of the second electroencephalogram signal is higher, take the second electroencephalogram signal as the electroencephalogram signal of the target brain area of the brain injury patient.

[0013] In a third aspect, an electronic device is disclosed, comprising: a processor, a memory, a communication interface and a communication bus, the processor, the memory and the communication interface complete the communication among each other through the communication bus; the memory is used to store at least one executable instruction, the executable instruction makes the processor execute the steps of the brain region signal reconstruction method according to any one of the preceding.

[0014] In a fourth aspect, a readable storage medium is disclosed, the readable storage medium stores a program or instructions, the program or instructions are executed by a processor to implement the brain region signal reconstruction method according to any one of the preceding.

[0015] In a fifth aspect, a computer program product is disclosed, comprising a computer program, the computer program is executed by a processor to implement the steps of the brain region signal reconstruction method according to any one of the preceding.

[0016] Embodiments of the present application include the following advantages: In the embodiments of the present application, two methods for obtaining the electroencephalogram signal of the target brain region of the brain injury patient are provided, and the electroencephalogram signal of the target brain region of the brain injury patient is obtained by inputting the electroencephalogram signal of the substitute brain region of the brain injury patient into the first generation model and the second generation model respectively. The first generation model is trained by the electroencephalogram signal sample of the substitute brain region of the healthy subject and the electroencephalogram signal sample of the substitute brain region, and the idea of model migration is adopted. The second generation model is trained by the electroencephalogram signal sample of the substitute brain region of the brain injury patient and the electroencephalogram signal sample of the target brain region of the healthy subject, and the idea of data migration is adopted. In addition, in the present scheme, the electroencephalogram information accuracy of the first electroencephalogram signal and the second electroencephalogram signal obtained by comparing the two schemes respectively is compared, and the one with higher accuracy is selected as the electroencephalogram signal of the target brain region of the brain injury patient, which can further ensure the quality of the predicted electroencephalogram signal of the target brain region of the brain injury patient and improve the user experience. BRIEF DESCRIPTION OF DRAWINGS

[0017] Figure 1 is a flowchart of a brain region signal reconstruction method of the present application; Figure 2 is a schematic diagram of a brain region signal reconstruction provided by an embodiment of the present application; Figure 3 is a schematic diagram of a brain region signal reconstruction device provided by an embodiment of the present application; Figure 4 is a structural schematic diagram of an electronic device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0018] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, the present application will be further described in detail below with reference to the drawings and specific embodiments.

[0019] The terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described herein, and the objects distinguished by "first", "second", etc. are generally a class and do not limit the number of objects, for example, the first object can be one or more. In addition, the term "and / or" in the specification and claims is used to describe the relationship between the associated objects, which means that there are three relationships, for example, A and / or B, which means that there are three cases: A exists alone, A and B exist together, and B exists alone. The character " / " generally represents an "or" relationship between the associated objects. The term "multiple" in the embodiments of the present application means two or more, and other quantifiers are similar.

[0020] Brain-Computer Interface (BCI) technology establishes a direct communication path between the brain and the external world. This technology can bypass the traditional neural and muscle system, interpret the signals generated by brain activity, and convert the relevant signals into control commands for computers, mechanical devices or communication systems. BCI has broad prospects in the fields of medical rehabilitation, intelligent control and human enhancement.

[0021] A general BCI system usually follows four steps of operation. First, the brain activity signals of the target individual are collected by means of sensors; second, the collected brain activity signals are preprocessed to improve their parameter quality and reduce noise interference, etc.; third, features representing specific thoughts or intentions are extracted from the preprocessed brain activity signals; and finally, the extracted features are converted into actual instructions that can drive external devices or express.

[0022] At present, many mature BCI applications are highly dependent on placing electrodes in specific areas of the head to record electroencephalogram signals, i.e. collecting electroencephalogram signals from different specific brain regions. This is because the brain function has a clear zoning characteristic, and only by accurately monitoring the activity of the relevant brain area can the user's intention be effectively identified.

[0023] However, if the relevant brain area of the target user is damaged, the electroencephalogram signals from the specific brain region cannot be directly collected, and therefore the BCI technology cannot be used to assist in completing the purposes of rehabilitation, communication, device control, etc. Therefore, it is urgent to provide a method that can still obtain electroencephalogram signals in the case of damage to the brain area of the target user, breaking the traditional restriction that the corresponding type of brain-computer interface system can only be used by placing electroencephalogram electrodes in the target brain area, enabling brain injury patients to use BCI technology to interact with the outside world, and improving the user experience.

[0024] To this end, the application provides a brain region signal reconstruction method, which obtains the electroencephalogram signal of the brain region (substitute brain region) that can be collected by the brain injury patient, and obtains the electroencephalogram signal of the target brain region of the brain injury patient, to support the brain injury patient to use the BCI technology. The target brain region of the brain injury patient is the brain region whose electroencephalogram signal cannot be directly collected by the brain injury patient, but the corresponding electroencephalogram signal is needed in the BCI technology.

[0025] Reference Figure 1 The steps of an embodiment of the brain region signal reconstruction method of the application are shown in the flowchart, and the method can include the following steps: Step 101, obtaining the electroencephalogram signal of the substitute brain region of the brain injury patient; Step 102, inputting the electroencephalogram signal of the substitute brain region of the brain injury patient into a first generation model to obtain the first electroencephalogram signal of the target brain region of the brain injury patient output by the first generation model; wherein the first generation model is obtained by training the electroencephalogram signal sample of the target brain region of the healthy subject and the electroencephalogram signal sample of the substitute brain region; Step 103, inputting the electroencephalogram signal of the substitute brain region of the brain injury patient into a second generation model to obtain the predicted electroencephalogram signal of the target brain region of the healthy subject output by the second generation model, and taking the predicted electroencephalogram signal as the second electroencephalogram signal of the target brain region of the brain injury patient; wherein the second generation model is obtained by training the electroencephalogram signal sample of the substitute brain region of the brain injury patient and the electroencephalogram signal sample of the target brain region of the healthy subject.

[0026] Step 104, comparing the electroencephalogram information accuracy of the first electroencephalogram signal and the second electroencephalogram signal, and in the case that the electroencephalogram information accuracy of the first electroencephalogram signal is higher than that of the second electroencephalogram signal, taking the first electroencephalogram signal as the electroencephalogram signal of the target brain region of the brain injury patient; in the case that the electroencephalogram information accuracy of the second electroencephalogram signal is higher than that of the first electroencephalogram signal, taking the second electroencephalogram signal as the electroencephalogram signal of the target brain region of the brain injury patient.

[0027] That is, the first generation model and the second generation model are provided in the scheme, and the electroencephalogram of the target brain area of the brain injury patient is obtained through different mapping relationships. Among them, the first generation model refers to the model migration idea, and the electroencephalogram of the target brain area and the electroencephalogram of the substitute brain area in the individual are trained to obtain the mapping relationship between the electroencephalogram of the target brain area and the electroencephalogram of the substitute brain area in the individual, so that when used, the electroencephalogram of the target brain area of the brain injury patient can be inferred from the electroencephalogram of the substitute brain area of the brain injury patient. The second generation model refers to the data migration idea, and the electroencephalogram of the target brain area and the electroencephalogram of the substitute brain area between two different individuals are trained to obtain the mapping relationship between the electroencephalogram of the target brain area and the electroencephalogram of the substitute brain area between two different individuals, so that when used, the predicted electroencephalogram of the target brain area of the healthy subject can be inferred from the electroencephalogram of the substitute brain area of the brain injury patient, and the predicted electroencephalogram is used as the second electroencephalogram of the target brain area of the brain injury patient.

[0028] Through the first generation model and the second generation model, the electroencephalogram of the target brain area of the brain injury patient is obtained through two different mapping relationships respectively through the electroencephalogram of the substitute brain area of the brain injury patient. In the case that the electroencephalogram of the target brain area of the brain injury patient cannot be directly collected, the brain-computer interface technology can still be used to assist in completing the purposes of rehabilitation, communication, device control, etc.

[0029] In addition, the first electroencephalogram obtained according to the first generation model and the second electroencephalogram obtained according to the second generation model are compared according to the standard of electroencephalogram information accuracy, and the optimal parameters in the two are used as the electroencephalogram of the target brain area of the brain injury patient. The quality of the electroencephalogram of the target brain area of the brain injury patient is improved, and the reliability of the human-computer interaction technology is ensured, and the user experience is improved.

[0030] For step 101, in actual application, it is collected, for example, by a patch-type electroencephalogram electrode. Specifically, the patch-type electroencephalogram electrode is arranged on the substitute brain area of the brain injury patient to collect the electroencephalogram of the substitute brain area of the brain injury patient.

[0031] It can be understood that the brain region of the brain injury patient from which the electroencephalogram can be collected except the target brain area is the substitute brain area.

[0032] For step 102, the first generation model is trained by the electroencephalogram of the target brain area and the electroencephalogram of the substitute brain area of the healthy subject. Specifically, the electroencephalogram of the substitute brain area of the healthy subject is used as an input sample, and the electroencephalogram of the target brain area of the healthy subject is used as an output sample to complete the training of the first generation model; the mapping relationship between the substitute brain area and the target brain area of the individual is included in the trained first generation model.

[0033] In the training process, if the distance between the electroencephalogram signal output by the first generation model and the electroencephalogram signal of the target brain region of the healthy subject converges, the training of the first generation model is completed.

[0034] In this step, the cross-subject model (i.e., the first generation model) constructed using the electroencephalogram signal of the healthy subject is migrated to the data of the brain injury patient, and the electroencephalogram signal of the target brain region of the brain injury patient can be reconstructed only by collecting the electroencephalogram signal of the alternative brain region of the brain injury patient.

[0035] For step 103, it can be understood that the second generation model is obtained by training the electroencephalogram signal of the target brain region and the electroencephalogram signal of the alternative brain region of different subjects. Specifically, the electroencephalogram signal of the alternative brain region of the brain injury patient is taken as the input sample, and the electroencephalogram signal of the target brain region of the healthy subject is taken as the output sample to complete the training of the second generation model; the second generation model includes the mapping relationship between the electroencephalogram signal of the alternative brain region of the brain injury patient and the electroencephalogram signal of the target brain region of the healthy subject after the training.

[0036] It can be understood that the reconstructed electroencephalogram signal of the target brain region of the brain injury patient is the electroencephalogram signal of the target brain region of the brain injury patient in a healthy state without brain injury, i.e., the electroencephalogram signal of a healthy target brain region, so the predicted electroencephalogram signal of the target brain region of the healthy subject output by the second generation model can be taken as the second electroencephalogram signal of the target brain region of the brain injury patient.

[0037] In the training process, if the distance between the electroencephalogram signal output by the second generation model and the electroencephalogram signal of the target brain region of the healthy subject converges, the training of the second generation model is completed.

[0038] In this step, the cross-subject data migration model (i.e., the second generation model) constructed using the electroencephalogram signal of the target brain region of the healthy subject and the electroencephalogram signal of the alternative brain region of the brain injury patient directly establishes the mapping relationship between the electroencephalogram signal of the alternative brain region of the brain injury patient and the electroencephalogram signal of the target brain region of the healthy subject using a deep learning algorithm, realizes the cross-brain region and cross-subject data reconstruction of the electroencephalogram signal of the alternative brain region of the brain injury patient to the electroencephalogram signal of the target brain region of the healthy subject, and reconstructs the electroencephalogram signal of the target brain region of the brain injury patient only by collecting the electroencephalogram signal of the alternative brain region of the brain injury patient. Compared with the first generation model, the training sample of the second generation model does not need to be fixed to the paired data of a single individual, can integrate fragmented and non-paired group data, and is convenient for the collection of training samples.

[0039] For step 104, the EEG signals of the target brain region of the brain-injured patient obtained by the first generative model and the second generative model based on the EEG signals of the alternative brain regions of the brain-injured patient are compared. Specifically, the comparison is based on the parameter of EEG information accuracy. The result with higher EEG information accuracy is taken as the final EEG signal of the target brain region of the brain-injured patient, thereby improving the data quality of the EEG signal of the target brain region of the brain-injured patient, ensuring the reliability of the brain-computer interface system and the individual adaptability of the patient, and guaranteeing the user experience.

[0040] like Figure 2 The diagram illustrates the training and use of the first and second generative models provided in this embodiment. Specifically, the first generative model is trained using EEG signals from the target brain region of a healthy subject and EEG signals from a substitute brain region; the second generative model is trained using EEG signals from the substitute brain region of a brain-injured patient and EEG signals from the target brain region of a healthy subject. It is understood that the "healthy subject" in the diagram can be the same individual; two brain regions are used for clarity in the diagram.

[0041] This invention provides two methods for obtaining EEG signals of target brain regions from patients with brain injuries. Specifically, the EEG signals of substitute brain regions from patients with brain injuries are input into a first generation model and a second generation model, respectively. The first generation model is trained using EEG signal samples from substitute brain regions of healthy subjects, employing a model transfer approach. The second generation model is trained using EEG signal samples from substitute brain regions of patients with brain injuries and EEG signal samples from target brain regions of healthy subjects, employing a data transfer approach. Furthermore, this solution compares the accuracy of the first and second EEG signals obtained by the two methods, selecting the one with higher accuracy as the EEG signal of the target brain region of the patient with brain injury. This further ensures the quality of the predicted EEG signal of the target brain region of the patient with brain injury and improves the user experience.

[0042] Furthermore, this invention overcomes the traditional limitation that brain-computer interface (BCI) systems can only be used when EEG electrodes are placed in the target brain region. This expands the application scope of BCI systems beyond simply collecting EEG signals from the target brain region, significantly broadening its application range. For brain-injured patients where electrodes cannot be placed in the target brain region, the brain region signal reconstruction method provided by this invention only requires collecting EEG signals from a substitute brain region to reconstruct the target brain region's EEG signal, enabling these patients to interact with the outside world using BCI technology. Experimental verification shows that the brain region signal reconstruction method proposed in this invention ensures the good performance of the BCI system. Through algorithmic modeling, this solution achieves significant progress in the accuracy of EEG information, providing a reliable guarantee for the practical application of BCI technology.

[0043] In an optional embodiment of the present invention, the first generative model is trained using a first loss function, wherein the first loss function is:

[0044] in, This represents the first loss function. y This represents the electroencephalogram (EEG) signals of the target brain region in the healthy subject. This represents the electroencephalogram (EEG) signal of the target brain region output by the first generative model during training. This indicates the preset error threshold, based on Distance between The magnitude relationship is used to determine the appropriate loss function formula.

[0045] Understandable, delta This represents a preset error threshold used to distinguish between "small errors" and "large errors." When the error is small, i.e., At this point, the first loss function degenerates into calculating the mean squared error: At this point, the penalty (i.e., constraint) on the error term grows quadratically, which is beneficial for rapid convergence; when the error is large, i.e. At this point, the first loss function degenerates into calculating an approximate absolute error: In this case, the penalty for the error increases linearly, which is lower than the previous quadratic penalty, thus reducing the impact of large errors on the first generative model and enhancing its robustness. The value represents the absolute error between the EEG signal of the target brain region calculated from the EEG signal of the target brain region output by the first generative model.

[0046] Optionally, In this scheme, the preferred value is 0.1, which can balance the advantages and disadvantages of mean square error and approximate absolute error.

[0047] In this embodiment, the first generative model is trained according to the first loss function, which combines the advantages of both mean squared error and mean absolute error, while avoiding their main disadvantages. It can balance the stability of model training and the robustness to outliers (extreme values ​​in the training samples that deviate significantly from other data).

[0048] In an optional embodiment of the present invention, before inputting the electroencephalogram (EEG) signal of the alternative brain region of the brain-injured patient into the first generative model, the method further includes: S01, when the first generative model converges according to the first loss function, freeze the convolutional layers in the first generative model except for the fully connected layers; S02, adjusting the full connection layer of the first generated model according to the electroencephalogram signals of the alternative brain region of the brain injury patient and the electroencephalogram signals of the target brain region of the healthy subject.

[0049] That is, in the present embodiment, the first generated model trained is fine-tuned according to the electroencephalogram signals of the alternative brain region of the brain injury patient. Specifically, the full connection layer of the first generated model is adjusted so that the output of the first generated model conforms to the relevant characteristics of the brain region of the brain injury patient.

[0050] Specifically, the first generated model is enabled to perform transfer learning according to the electroencephalogram signals of the alternative brain region of the brain injury patient. Freezing the convolutional layer of the first generated model except the full connection layer can preserve the parameters related to feature extraction obtained by training the first generated model; and training the full connection layer, i.e. adjusting the attention weight of the full connection layer to different feature vectors, can improve the correlation between the output of the full connection layer and the electroencephalogram signals of the brain injury patient.

[0051] For example, the input sample in the fine-tuning process is the electroencephalogram signals of the alternative brain region of the brain injury patient, and the output sample is the electroencephalogram signals of the target brain region of the healthy subject.

[0052] In the present embodiment, the first generated model is fine-tuned according to the electroencephalogram signals of the alternative brain region of the brain injury patient, which can improve the correlation between the first electroencephalogram signals obtained by the first generated model and the brain injury patient, and ensure the user experience.

[0053] In an optional embodiment of the present application, the second generated model is trained by a second loss function, and the second loss function is as follows:

[0054] wherein, denotes the second loss function, denotes a first loss function used for training the first generated model, y denotes the electroencephalogram signals of the target brain region of the healthy subject, denotes the electroencephalogram signals of the target brain region output by the first generated model in the training process, denotes a preset error threshold, is a regularization term of the second loss function, used for preventing overfitting of the second generated model, is a coefficient of the regularization term, theta denotes the weight to be trained of the second generated model, i denotes the number of parameters that can be trained in the second generated model.

[0055] It can be understood that, is the first loss function, wherein the parameters are the same as those of the first loss function.

[0056] The regularization term is set to prevent model overfitting. Overfitting refers to the model learning ability on the training data is too strong, learning the noise in the training samples, resulting in the trained model performing poorly on new test data. Therefore, an additional "penalty term" is needed to prevent overfitting, i.e. the regularization term. The model weight parameters in the regularization term theta directly determine the complexity of the model, and the larger the regularization term means the more complex the model.

[0057] Optionally, in the present scheme is a hyperparameter, and the preferred value is 0.01. theta is a weight that needs to be trained in the training process of the second generation model.

[0058] In the present scheme, the second generation model is trained using the second loss function, which can balance the advantages and disadvantages of mean square error and approximate absolute error, and prevent model overfitting.

[0059] In an optional embodiment of the present application, the method for obtaining the electroencephalogram signal of the substitute brain area of the brain injury patient comprises: obtaining the electroencephalogram signal of the substitute brain area of the brain injury patient, and obtaining the electroencephalogram signal of the target brain area and the electroencephalogram signal of the substitute brain area of the healthy subject according to the target brain area and the substitute brain area of the brain injury patient; wherein the electroencephalogram signal of the target brain area and the electroencephalogram signal of the substitute brain area of the healthy subject are used to train the first generation model; and the electroencephalogram signal of the substitute brain area of the brain injury patient and the electroencephalogram signal of the target brain area of the healthy subject are used to train the second generation model.

[0060] Specifically, the distribution of the substitute brain area and the target brain area of the healthy subject needs to be determined according to the distribution of the substitute brain area and the target brain area of the brain injury patient, and different model corresponds to different regional distribution. After the substitute brain area and the target brain area of the brain injury patient are determined, the electroencephalogram signals of the corresponding substitute brain area and target brain area of the healthy subject are collected for model training or model adjustment.

[0061] Optionally, after collecting the electroencephalogram signal of the substitute brain area of the brain injury patient, the electroencephalogram signal of the substitute brain area and the target brain area of the healthy subject each time, the first generation model and the second generation model are trained according to the relevant parameters. The electroencephalogram signal of the target brain area of the brain injury patient is obtained according to the trained first generation model, the second generation model and the electroencephalogram signal of the substitute brain area of the brain injury patient. That is, a new first generation model and a new second generation model are trained each time the electroencephalogram signal of the target brain area of the brain injury patient is obtained, so that the output parameters of the first generation model and the second generation model are more in line with the characteristics of the current brain injury patient.

[0062] Wherein, when collecting the electroencephalogram signals, the electroencephalogram signals collected at different time for the same individual are different samples. That is, the number of samples is not equal to the number of individuals. For example, for a healthy subject individual, 10,000 combinations of electroencephalogram signals of the substitute brain region and electroencephalogram signals of the target brain region can be collected due to time changes.

[0063] Optionally, after collecting the electroencephalogram signals of the substitute brain region of the brain injury patient, the electroencephalogram signals of the substitute brain region and the electroencephalogram signals of the target brain region of the healthy subject each time, the first generation model and the second generation model are adjusted according to the relevant parameters. That is, the first generation model and the second generation model can be historical models that have been trained according to historical sample data. When used for a new brain injury patient, the historical models can be fine-tuned by newly collected electroencephalogram signals of the brain injury patient and newly collected electroencephalogram signals of the healthy subject, to obtain the models used in steps 102-103.

[0064] It can be understood that, by fine-tuning the historical models, the number of samples required is less than training a complete model independently each time, which is conducive to the rapid execution and implementation of the scheme.

[0065] In one example, the target brain region of the brain injury patient is a brain region of the brain injury patient for which electroencephalogram signals cannot be collected, including one or any combination of the following: frontal lobe, parietal lobe, temporal lobe, and occipital lobe. The substitute brain region of the brain injury patient is a brain region of the brain injury patient that supports the collection of electroencephalogram signals, except for the target brain region. The location of the target brain region of the healthy subject is determined according to the target brain region of the brain injury patient, and the location of the substitute brain region of the healthy subject is determined according to the substitute brain region of the brain injury patient.

[0066] For example, the electroencephalogram signals of the frontal lobe of a brain injury patient a cannot be collected, for example, the frontal lobe region cannot be arranged with a patch-type electroencephalogram electrode. The target brain region of the brain injury patient a is the frontal lobe, and the substitute brain region is the parietal lobe, the temporal lobe, and the occipital lobe. The target brain region of the healthy subject b corresponding to the need to collect electroencephalogram signals is also the frontal lobe, and the substitute brain region is also the parietal lobe, the temporal lobe, and the occipital lobe. That is, according to the target brain region of the brain injury patient, the substitute brain region of the brain injury patient and the target brain region and the substitute brain region of the healthy subject are determined.

[0067] For example, the electroencephalogram signals include one or any combination of the following: steady-state visual evoked potential (SSVEP), motor imagery (MI), and event-related potential (ERP).

[0068] Generally, different brain regions correspond to different electroencephalogram signals. According to the neural action mechanism of the electroencephalogram signals, the occipital lobe collects steady-state visual evoked potentials (SSVEP), that is, the synchronous response of the visual cortex to constant frequency flicker; the parietal lobe collects motor imagery (MI), that is, the response of the brain to imagined movement (not actual movement); and the frontal lobe or temporal lobe collects event-related potentials (ERP), that is, the transient response of the brain to a specific discrete event.

[0069] It should be noted that, for the method embodiments, in order to simply describe, they are all expressed as a series of action combinations, but those skilled in the art should know that the embodiments of the present application are not limited by the action sequence described, because according to the embodiments of the present application, certain steps can be performed in other sequences or simultaneously. Secondly, those skilled in the art should know that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily necessary for the embodiments of the present application.

[0070] Referring to Figure 3 , a structural block diagram of a brain region reconstruction device is shown, which can include: The electroencephalogram acquisition module 201 is configured to acquire the electroencephalogram signals of the substitute brain region of the brain injury patient; The first generation module 202 is configured to input the electroencephalogram signals of the substitute brain region of the brain injury patient into a first generation model to obtain the first electroencephalogram signals of the target brain region of the brain injury patient output by the first generation model; wherein the first generation model is trained by the electroencephalogram signal samples of the target brain region of the healthy subject and the electroencephalogram signal samples of the substitute brain region; The second generation module 203 is configured to input the electroencephalogram signals of the substitute brain region of the brain injury patient into a second generation model to obtain the predicted electroencephalogram signals of the target brain region of the healthy subject output by the second generation model, and take the predicted electroencephalogram signals as the second electroencephalogram signals of the target brain region of the brain injury patient; wherein the second generation model is trained according to the electroencephalogram signal samples of the substitute brain region of the brain injury patient and the electroencephalogram signal samples of the target brain region of the healthy subject; The target determination module 204 is configured to compare the electroencephalogram information accuracy of the first electroencephalogram signals and the second electroencephalogram signals, and in the case that the electroencephalogram information accuracy of the first electroencephalogram signals is high, take the first electroencephalogram signals as the electroencephalogram signals of the target brain region of the brain injury patient; in the case that the electroencephalogram information accuracy of the second electroencephalogram signals is high, take the second electroencephalogram signals as the electroencephalogram signals of the target brain region of the brain injury patient.

[0071] Optionally, the first generation model in the first generation module 202 is trained by a first loss function, and the first loss function is:

[0072] wherein, represents the first loss function, y represents the electroencephalogram of the target brain region of the healthy subject, represents the electroencephalogram of the target brain region output by the first generation model during training, represents a preset error threshold, according to the distance between and the size relationship, the selected loss function formula is confirmed.

[0073] Optionally, before the first generation module 202, the apparatus further comprises: a freezing module configured to freeze the convolutional layers in the first generation model except the fully connected layers in the case that the first generation model converges according to the first loss function; an adjusting module configured to adjust the fully connected layers of the first generation model according to the electroencephalogram of the alternative brain region of the brain injury patient and the electroencephalogram of the target brain region of the healthy subject.

[0074] Optionally, in the second generation module 203, the second generation model is obtained by training a second loss function, and the second loss function is:

[0075] wherein, represents the second loss function, represents the first loss function for training the first generation model, y represents the electroencephalogram of the target brain region of the healthy subject, represents the electroencephalogram of the target brain region output by the first generation model during training, represents a preset error threshold, is a regularization term of the second loss function, used for preventing overfitting of the second generation model, theta represents the weights to be trained of the second generation model, i represents the number of trainable parameters in the second generation model.

[0076] Optionally, the electroencephalogram acquisition module 201 is specifically configured to acquire the electroencephalogram signals of the substitute brain area of the brain injury patient, and acquire the electroencephalogram signals of the target brain area and the substitute brain area of the healthy subject according to the target brain area and the substitute brain area of the brain injury patient; wherein the electroencephalogram signals of the target brain area and the substitute brain area of the healthy subject are used to train the first generation model; and the electroencephalogram signals of the substitute brain area of the brain injury patient and the electroencephalogram signals of the target brain area of the healthy subject are used to train the second generation model.

[0077] Optionally, the target brain area of the brain injury patient is a brain region of the brain injury patient that cannot acquire electroencephalogram signals, including one or any combination of the following: frontal lobe, parietal lobe, temporal lobe, and occipital lobe. The substitute brain area of the brain injury patient is a brain region of the brain injury patient that supports the acquisition of the electroencephalogram signals, except for the target brain area. The target brain area of the healthy subject is determined by the target brain area of the brain injury patient, and the substitute brain area of the healthy subject is determined by the substitute brain area of the brain injury patient.

[0078] That is, the target brain area and the substitute brain area are brain areas that need to acquire electroencephalogram signals when the brain injury patient uses the brain-computer interface technology, but are not suitable for arranging electroencephalogram acquisition electrodes, including but not limited to the frontal lobe, the parietal lobe, the temporal lobe, and the occipital lobe; the substitute brain area is a brain area that the brain injury patient is actually easy to arrange acquisition electrodes, except for the target brain area; and the establishment of the target brain area and the substitute brain area needs to be determined according to the condition of the patient using the brain-computer interface system. Different patients in different conditions have different brain area division schemes.

[0079] Optionally, the electroencephalogram signals include one or any combination of the following: steady-state visual evoked potential, motor imagery, and event-related evoked potential.

[0080] The brain area signal reconstruction device provided by the embodiment of the application provides two methods for obtaining the electroencephalogram signals of the target brain area of the brain injury patient, specifically by inputting the electroencephalogram signals of the substitute brain area of the brain injury patient into the first generation model and the second generation model respectively. The first generation model is trained by the electroencephalogram signal samples of the substitute brain area of the healthy subject and the electroencephalogram signal samples of the substitute brain area, and adopts the model migration idea. The second generation model is trained by the electroencephalogram signal samples of the substitute brain area of the brain injury patient and the electroencephalogram signal samples of the target brain area of the healthy subject, and adopts the data migration idea. In addition, in the present scheme, the electroencephalogram information accuracy of the first electroencephalogram signal and the second electroencephalogram signal obtained by the foregoing two schemes is compared, and the one with higher accuracy is selected as the electroencephalogram signal of the target brain area of the brain injury patient, which can further ensure the quality of the predicted electroencephalogram signal of the target brain area of the brain injury patient and improve the user experience.

[0081] For the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts are referred to the part of the method embodiment.

[0082] Referring to Figure 4 is a structural schematic diagram of an electronic device provided by an embodiment of the present application. As shown in Figure 4 , the electronic device comprises a processor, a memory, a communication interface and a communication bus, the processor, the memory and the communication interface complete communication with each other through the communication bus; the memory is used for storing at least one executable instruction, and the executable instruction makes the processor execute the steps of the signal reconstruction method of the foregoing embodiment.

[0083] The embodiment of the present application provides a non-transitory computer readable storage medium, when instructions in the storage medium are executed by a program or a processor of a terminal, the terminal can execute the steps of the signal reconstruction method of the foregoing embodiment.

[0084] Each embodiment in the specification is described in a progressive manner, and each embodiment focuses on the difference from other embodiments, and the same and similar parts between each embodiment can be referred to each other.

[0085] Those skilled in the art should understand that the embodiments of the embodiments of the present application can be provided as a method, device or computer program product. Therefore, the embodiments of the present application can adopt a complete hardware embodiment, a complete software embodiment or an embodiment combining software and hardware aspects. Moreover, the embodiments of the present application can adopt the form of a computer program product implemented on one or more computer usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer usable program codes.

[0086] The embodiments of the present application are described with reference to the flowcharts and / or block diagrams according to the method, terminal device (system) and computer program product of the embodiments of the present application. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, and the combination of the flows and / or blocks in the flowcharts and / or block diagrams can be realized by computer program instructions. These computer program instructions can be provided to a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing terminal device to produce a machine, so that the instructions executed by the computer or other programmable data processing terminal device produce a machine for realizing the functions specified in the flowcharts and / or block diagrams. Figure 1 The device for realizing the functions specified in one flow or multiple flows and / or blocks Figure 1 The device for realizing the functions specified in one flow or multiple flows and / or blocks

[0087] These computer program instructions may also be stored in a computer-readable storage medium capable of directing a computer or other programmable data processing terminal device to operate in a predictive manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0088] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal equipment, causing a series of operational steps to be performed on the computer or other programmable terminal equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable terminal equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0089] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal device that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or terminal device. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or terminal device that includes said element.

[0090] This document uses specific examples to illustrate the principles and implementation methods of the present invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of the present invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A method for reconstructing brain region signals, characterized in that, The method includes: Obtaining electroencephalogram (EEG) signals from alternative brain regions in patients with brain injuries; The EEG signal of the substitute brain region of the brain-injured patient is input into the first generation model to obtain the first EEG signal of the target brain region of the brain-injured patient output by the first generation model; wherein, the first generation model is obtained by training the EEG signal of the target brain region of a healthy subject with the EEG signal of the substitute brain region. The EEG signal of the substitute brain region of the brain-injured patient is input into the second generation model to obtain the predicted EEG signal of the target brain region of the healthy subject output by the second generation model. The predicted EEG signal is used as the second EEG signal of the target brain region of the brain-injured patient. The second generation model is trained based on the EEG signal of the substitute brain region of the brain-injured patient and the EEG signal of the target brain region of the healthy subject. By comparing the EEG information accuracy of the first EEG signal and the second EEG signal, if the EEG information accuracy of the first EEG signal is higher than that of the second EEG signal, the first EEG signal is used as the EEG signal of the target brain region of the brain-injured patient; if the EEG information accuracy of the second EEG signal is higher than that of the first EEG signal, the second EEG signal is used as the EEG signal of the target brain region of the brain-injured patient.

2. The brain region signal reconstruction method according to claim 1, characterized in that, The first generative model is trained using a first loss function, which is: in, Let y represent the first loss function, and y represent the electroencephalogram (EEG) signal of the target brain region in the healthy subject. This represents the electroencephalogram (EEG) signal of the target brain region output by the first generative model during training. This represents the preset error threshold, based on y and Distance between The magnitude relationship is used to determine the appropriate loss function formula.

3. The brain region signal reconstruction method according to claim 2, characterized in that, Before inputting the electroencephalogram (EEG) signals from the alternative brain region of the brain-injured patient into the first generative model, the method further includes: If the first generative model converges according to the first loss function, freeze the convolutional layers in the first generative model except for the fully connected layers; The fully connected layers of the first generative model are adjusted based on the EEG signals of the alternative brain region of the brain-injured patient and the EEG signals of the target brain region of the healthy subject.

4. The brain region signal reconstruction method according to claim 1, characterized in that, The second generative model is trained using a second loss function, which is: in, This represents the second loss function. Let represent the first loss function used to train the first generative model, and y represent the electroencephalogram (EEG) signal of the target brain region of the healthy subject. This represents the electroencephalogram (EEG) signal of the target brain region output by the first generative model during the training process. This indicates the preset error threshold. θ is the regularization term of the second loss function, used to prevent the second generative model from overfitting, θ represents the weights to be trained in the second generative model, and i represents the number of trainable parameters in the second generative model.

5. The brain region signal reconstruction method according to claim 1, characterized in that, The acquisition of electroencephalogram (EEG) signals from alternative brain regions in patients with brain injuries includes: Obtain EEG signals from a substitute brain region of a brain-injured patient, and obtain EEG signal samples from the target brain region and the substitute brain region of a healthy subject according to the target brain region and the substitute brain region of the brain-injured patient. The EEG signals of the target brain region and the EEG signals of the substitute brain region of the healthy subject are used to train the first generative model; the EEG signals of the substitute brain region of the brain-injured patient and the EEG signals of the target brain region of the healthy subject are used to train the second generative model.

6. The brain region signal reconstruction method according to claim 1, characterized in that, The target brain region of the brain injury patient is the brain region where the brain injury patient cannot collect electroencephalogram (EEG) signals, including one or any combination of the following: frontal lobe, parietal lobe, temporal lobe, and occipital lobe; The alternative brain region for the brain injury patient is the brain region of the brain injury patient that supports the acquisition of the electroencephalogram (EEG) signals, excluding the target brain region. The location of the target brain region in the healthy subject is determined by the target brain region in the brain-injured patient, and the location of the alternative brain region in the healthy subject is determined by the alternative brain region in the brain-injured patient.

7. The brain region signal reconstruction method according to claim 1, characterized in that, The EEG signals include one or any combination of the following: steady-state visual evoked potentials, motor imagery, and event-related evoked potentials.

8. A brain region signal reconstruction device, characterized in that, The brain region signal reconstruction device includes: The EEG acquisition module is used to acquire EEG signals from alternative brain regions in patients with brain injuries. The first generation module is used to input the EEG signal of the substitute brain region of the brain-injured patient into the first generation model to obtain the first EEG signal of the target brain region of the brain-injured patient output by the first generation model; wherein, the first generation model is obtained by training the target brain region EEG signal sample and the substitute brain region EEG signal sample of a healthy subject. The second generation module is used to input the EEG signal of the substitute brain region of the brain-injured patient into the second generation model to obtain the predicted EEG signal of the target brain region of the healthy subject output by the second generation model, and use the predicted EEG signal as the second EEG signal of the target brain region of the brain-injured patient; wherein, the second generation model is trained based on the EEG signal samples of the substitute brain region of the brain-injured patient and the EEG signal samples of the target brain region of the healthy subject. The target determination module is used to compare the EEG information accuracy of the first EEG signal and the second EEG signal. If the EEG information accuracy of the first EEG signal is high, the first EEG signal is used as the EEG signal of the target brain region of the brain-injured patient; if the EEG information accuracy of the second EEG signal is high, the second EEG signal is used as the EEG signal of the target brain region of the brain-injured patient.

9. An electronic device, characterized in that, include: The processor, memory, communication interface, and communication bus are provided, wherein the processor, memory, and communication interface communicate with each other via the communication bus. The memory is used to store at least one executable instruction that causes the processor to perform the steps of the brain region signal reconstruction method as described in any one of claims 1 to 7.

10. A readable storage medium, characterized in that, The readable storage medium stores a program or instructions that, when executed by a processor, implement the steps of the brain region signal reconstruction method as described in any one of claims 1 to 7.

Images