Electroencephalogram signal identity recognition method and system, electronic equipment and storage medium

Abstract

The invention relates to the technical field of biological recognition, and particularly discloses an electroencephalogram signal identity recognition method and system, electronic equipment and a storage medium, and the method comprises the steps: obtaining a plurality of face images and original electroencephalogram signals of a target user when the target user watches a target induction image; based on the multiple face images of the target user, determining an eye movement time period of the target user, deleting an electroencephalogram signal corresponding to the eye movement time period from the original electroencephalogram signals to obtain an optimized electroencephalogram signal, and preprocessing the optimized electroencephalogram signal to obtain a target electroencephalogram signal; and performing multi-modal coding on the coding information of the target induction image and the coding information of the target electroencephalogram signal to obtain multi-modal coding information of the target user, and inputting the multi-modal coding information into a trained multi-modal large model to obtain an identity recognition result. According to the method, the problem of data difference caused by different induction images can be effectively solved, and identity recognition and verification can be efficiently realized through integrated process design of visual induction, video detection and electroencephalogram signal acquisition.

Description

Background Art

[0002] With the rapid development of a new generation of artificial intelligence technology represented by deep learning, the research and application of biometric technology has been pushed to a new level. In recent years, technologies such as face recognition, fingerprint recognition, finger vein recognition, and iris recognition have been widely used in finance, public security, access control, and other fields. At the same time, academia and industry are also constantly exploring new biometric recognition methods, such as voiceprint recognition, gait recognition, DNA, and other biometric recognition technologies. In addition, identity recognition based on EEG signals has also become a new research hotspot in recent years.

[0003] Since the 1980s, EEG-based biometrics have undergone decades of development and have made certain phased progress. At present, many scholars at home and abroad have conducted research on EEG-based identity recognition. With the development of deep learning technology in recent years, many scholars have also begun to try to combine deep learning technology to conduct EEG-based identity recognition research. In 2017, Rig Das et al. tried to use CNN to extract and identify features from VEP data, achieving an accuracy rate of 97% on a data set of 40 people. In 2019, et al. used CNN as a classifier and used the EEG dataset of the RSVP task, achieving an accuracy of 99.3% on a test set of 10 subjects. JXchen et al. proposed the GSLT-CNN network architecture, and the final combination was a dataset of 157 people using the original unprocessed data to achieve an accuracy of 96%; Mota et al. studied biometrics and EEG signals from a cross-task perspective and achieved an equal error rate of 0.1%.

[0004] Through the investigation of the research results of EEG identity recognition at home and abroad, it can be found that EEG signals have unique advantages, but also have obvious disadvantages. At present, they are still in the initial stage and are still a long way from practical application. Specifically, they are manifested in several aspects: ① EEG signals are relatively weak and are more susceptible to interference from electrooculogram signals, electrocardiogram signals, etc.; ② There is currently no unified induced paradigm for EEG signal identity recognition tasks, and the evaluation data sets selected by researchers are also basically different, and the research results are not comparable; ③ The collection of EEG signals is difficult, resulting in a small amount of data. For deep learning scenarios that require a large amount of data for training, overfitting is prone to occur.

[0005] The main EEG signal data set evoked paradigms currently used include resting state, visual stimulation evoked potentials, event-related potentials, motor imagery, etc. Data collection based on the resting state evoked paradigm is relatively easy and does not require the active cooperation of the subjects, but there is a problem of being unable to determine the starting position of the data. The use of visual stimulation evoked potentials is currently more common. Because the EEG signal is very weak and the amplitude is at the μV level, no matter which method is used, the collected data needs to be pre-processed. Among them, removing artifacts is still the main difficulty faced by EEG signal processing. Studies have shown that eye contact signals are one of the most important EEG signal interference factors among EEG signal artifacts.

[0006] Therefore, it is urgent to provide a technical solution to solve the above problems. Summary of the invention

[0007] In order to solve the above technical problems, the present invention provides a method, system, electronic device and storage medium for EEG signal identification.

[0008] In a first aspect, the present invention provides a method for identifying an individual using an EEG signal. The technical solution of the method is as follows:

[0009] When the target user views the target induced image, a plurality of facial images and original EEG signals of the target user are obtained;

[0010] Based on the multiple facial images of the target user, determine the eye movement time period of the target user, and delete the EEG signal corresponding to the eye movement time period in the original EEG signal to obtain an optimized EEG signal and perform preprocessing to obtain a target EEG signal;

[0011] The coding information of the target induced image and the coding information of the target electroencephalogram signal are multimodally encoded to obtain the multimodal coding information of the target user and input it into a trained multimodal large model to obtain the identity recognition result of the target user.

[0012] The beneficial effects of the EEG signal identity recognition method of the present invention are as follows:

[0013] The method of the present invention can effectively solve the problem of data differences caused by different induced images. Through the integrated process design of visual induction, video detection and EEG signal acquisition, it can efficiently realize identity recognition and verification. It has high practical value and explores a new path for EEG signal identity recognition.

[0014] Based on the above solution, the EEG signal identity recognition method of the present invention can also be improved as follows.

[0015] In an optional manner, the step of determining the eye movement time period of the target user based on the multiple facial images of the target user includes:

[0016] By using a feature point algorithm and according to changes in eye feature points in a plurality of facial images of the target user, an eye movement time period representing changes in eye features of the target user is determined.

[0017] In an optional manner, the preprocessing method is: noise removal.

[0018] In an optional manner, the training process of the multimodal large model is:

[0019] The target EEG signals corresponding to multiple subjects when they watch different induced images are obtained, and multiple training data are generated according to the subject ID, induced image and target EEG signal as one training data, and the multiple training data are input into the multimodal large model for iterative training to obtain a trained multimodal large model.

[0020] In an optional manner, the multimodal large model is a multimodal network based on a self-attention mechanism.

[0021] In a second aspect, the present invention provides an EEG signal identity recognition system, the technical solution of which is as follows:

[0022] It includes: an acquisition module, a processing module and an identification module;

[0023] The acquisition module is used to: when the target user views the target induced image, acquire multiple facial images and original EEG signals of the target user;

[0024] The processing module is used to: determine the eye movement time period of the target user based on the multiple facial images of the target user, and delete the EEG signal corresponding to the eye movement time period in the original EEG signal to obtain an optimized EEG signal and perform preprocessing to obtain a target EEG signal;

[0025] The recognition module is used to: perform multimodal encoding on the coding information of the target induced image and the coding information of the target EEG signal to obtain the multimodal coding information of the target user and input it into the trained multimodal large model to obtain the identity recognition result of the target user.

[0026] The beneficial effects of the EEG signal identification system of the present invention are as follows:

[0027] The system of the present invention can effectively solve the problem of data differences caused by different induced images. Through the integrated process design of visual induction, video detection and EEG signal acquisition, it can efficiently realize identity recognition and verification. It has high practical value and explores a new path for EEG signal identity recognition.

[0028] Based on the above solution, the EEG signal identification system of the present invention can also be improved as follows.

[0029] In an optional manner, the step of determining the eye movement time period of the target user based on the multiple facial images of the target user in the processing module includes:

[0030] By using a feature point algorithm and according to changes in eye feature points in a plurality of facial images of the target user, an eye movement time period representing changes in eye features of the target user is determined.

[0031] In an optional manner, the preprocessing method is: noise removal.

[0032] In a third aspect, a technical solution of an electronic device of the present invention is as follows:

[0033] The method comprises a memory, a processor and a program stored in the memory and running on the processor. When the processor executes the program, the steps of the method for identifying the identity of an electroencephalogram signal of the present invention are implemented.

[0034] In a fourth aspect, the present invention provides a computer-readable storage medium having the following technical solution:

[0035] Instructions are stored in the computer-readable storage medium. When the computer-readable storage medium reads the instructions, the computer-readable storage medium executes the steps of the method for identifying the identity of an electroencephalogram signal of the present invention.

[0036] The above description is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the present invention, it can be implemented according to the contents of the specification. In order to make the above and other purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] The accompanying drawings are only used to illustrate the embodiments and are not to be considered as limiting the present invention. In addition, the same reference symbols are used to represent the same components throughout the accompanying drawings. In the accompanying drawings:

[0038] Figure 1 A schematic diagram of a flow chart of an embodiment of a method for identifying an individual using an electroencephalogram signal according to the present invention;

[0039] Figure 2 This is a schematic diagram of camera installation;

[0040] Figure 3 This is a schematic diagram of facial feature points;

[0041] Figure 4 Schematic diagram of the removal of eye movement artifacts;

[0042] Figure 5 Schematic diagram of a multimodal network based on the self-attention mechanism;

[0043] Figure 6 It is a structural schematic diagram of an embodiment of an EEG signal identity recognition system of the present invention;

[0044] Figure 7 The figure is a schematic structural diagram of an embodiment of an electronic device of the present invention. DETAILED DESCRIPTION

[0045] Exemplary embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present invention are shown in the drawings, it should be understood that the present invention can be implemented in various forms and should not be limited to the embodiments set forth herein.

[0046] Figure 1 A flow chart of an embodiment of an EEG signal identification method provided by the present invention is shown, and the EEG signal identification method can be executed by an electronic device such as a terminal device or a server. The terminal device can be any fixed or mobile terminal such as a user equipment (UE), a mobile device, a user terminal, a terminal, a cellular phone, a cordless phone, a personal digital assistant (PDA), a handheld device, a computing device, a vehicle-mounted device, a wearable device, etc. The server can be a single server or a server cluster composed of multiple servers. Any electronic device can implement the EEG signal identification method by calling computer-readable instructions stored in a memory through a processor. Figure 1 As shown, the following steps are included:

[0047] S1. When a target user views a target induced image, a plurality of facial images and original EEG signals of the target user are obtained.

[0048] The target user is the user who needs to be identified. The target induced image is a randomly generated image, which is output and displayed on the display screen.

[0049] The camera is used to collect multiple facial images of the target user, and the multiple facial images are multiple consecutive facial images (continuous frame images) when the target user views the target induced image. Figure 2 As shown, the camera is arranged above the display screen to capture the facial features and eye movement features of the target user. The original EEG signal is collected through an EEG signal collection electrode or an EEG cap containing an EEG signal collection electrode.

[0050] It should be noted that in this embodiment, a high-definition camera that can collect the user's facial features and eye movement features is added above the display screen when collecting EEG signals. When the user watches different types of visually induced sample pictures or videos displayed on the screen, the detailed features of his eyes and face are also recorded synchronously.

[0051] S2. Based on the multiple facial images of the target user, determine the eye movement time period of the target user, and delete the EEG signal corresponding to the eye movement time period in the original EEG signal to obtain an optimized EEG signal and perform preprocessing to obtain a target EEG signal.

[0052] In S2, a feature point algorithm is used to determine an eye movement time period representing changes in eye features of the target user according to changes in eye feature points in a plurality of facial images of the target user.

[0053] Among them, Figure 3 As shown, a deep learning-based face and eye feature point algorithm is used to fit the eye-related feature points of the subject, and the eye movement feature rules (changes in eye feature points) are recorded in real time to determine the eye movement time period that characterizes the changes in eye features.

[0054] Among them, in the original EEG signal obtained, the time slices (eye movement time periods) with obvious eye movement characteristics are deleted to obtain the EEG signal without interference from eye movement artifacts, that is, the EEG signal is optimized. Figure 4 The figure shows a typical EEG signal (original EEG signal) with eye movement artifacts. The area in the frame shows the serious interference caused by eye movement artifacts. By detecting the user's eye feature points, the time slices (eye movement time periods) with eye movement artifacts can be identified. In the application process, only the signal in the frame area needs to be removed to obtain an EEG signal without obvious eye movement artifacts, that is, an optimized EEG signal.

[0055] The preprocessing method is noise removal by default, and can also be adjusted according to actual conditions, and there is no limitation here. The target EEG signal is the EEG signal after noise removal of the optimized EEG signal.

[0056] S3. Perform multimodal encoding on the coding information of the target induced image and the coding information of the target EEG signal to obtain the multimodal coding information of the target user and input it into the trained multimodal large model to obtain the identity recognition result of the target user.

[0057] In S3, the multimodal large model is: a multimodal network based on the self-attention mechanism. The training process of the multimodal large model is:

[0058] The target EEG signals corresponding to multiple subjects when they watch different induced images are obtained, and multiple training data are generated according to the subject ID, induced image and target EEG signal as one training data, and the multiple training data are input into the multimodal large model for iterative training to obtain a trained multimodal large model.

[0059] Each training data is: {experimenter ID, induced image, target EEG signal}. The advantage of a large multimodal model is that it can make full use of information from different media data to extract richer and more comprehensive features, thereby improving the performance and generalization ability of the model. In addition, a large multimodal model can further enhance the semantic understanding and expression ability of the model by learning the associations between different modalities. Figure 5 As shown in the figure, the induced image and the target EEG signal preprocessed by the subject based on the EEG signal acquisition result induced by the image are separately encoded (embedded), and then multimodally encoded (embedding) and uniformly input into a multimodal network based on the self-attention mechanism for processing. After passing through the multi-layer network, a feature sequence is output, and then a softmax classifier is connected to the feature sequence to classify the corresponding identity recognition result.

[0060] It should be noted that, for the same subject, all factors that cause different induced images to produce different induced EEG signals are taken into consideration, and the induced image is equivalent to a "reference"; for different subjects, for the same induced image, the result of induced image encoding is consistent, and only the induced EEG signal data is different, so it can be used to distinguish different subjects.

[0061] The technical solution of this embodiment can effectively solve the problem of data differences caused by different induced images. Through the integrated process design of visual induction, video detection and EEG signal acquisition, it can efficiently realize identity recognition and verification. It has high practical value and explores a new path for EEG signal identity recognition.

[0062] Figure 6 FIG. 2 is a schematic diagram showing a structure of an embodiment of an EEG signal identification system 200 provided by the present invention. Figure 6 As shown, the system 200 includes: an acquisition module 210, a processing module 220 and an identification module 230;

[0063] The acquisition module 210 is used to: when the target user views the target induced image, acquire multiple facial images and original EEG signals of the target user;

[0064] The processing module 220 is used to: determine the eye movement time period of the target user based on the multiple facial images of the target user, and delete the EEG signal corresponding to the eye movement time period in the original EEG signal to obtain an optimized EEG signal and perform preprocessing to obtain a target EEG signal;

[0065] The recognition module 230 is used to: perform multimodal encoding on the coding information of the target induced image and the coding information of the target EEG signal, obtain the multimodal coding information of the target user and input it into the trained multimodal large model to obtain the identity recognition result of the target user.

[0066] In an optional manner, the step of determining the eye movement time period of the target user based on the multiple facial images of the target user in the processing module 220 includes:

[0067] By using a feature point algorithm and according to changes in eye feature points in a plurality of facial images of the target user, an eye movement time period representing changes in eye features of the target user is determined.

[0068] In an optional manner, the preprocessing method is: noise removal.

[0069] In an optional manner, the training process of the multimodal large model is:

[0070] The target EEG signals corresponding to multiple subjects when they watch different induced images are obtained, and multiple training data are generated according to the subject ID, induced image and target EEG signal as one training data, and the multiple training data are input into the multimodal large model for iterative training to obtain a trained multimodal large model.

[0071] In an optional manner, the multimodal large model is a multimodal network based on a self-attention mechanism.

[0072] The technical solution of this embodiment can effectively solve the problem of data differences caused by different induced images. Through the integrated process design of visual induction, video detection and EEG signal acquisition, it can efficiently realize identity recognition and verification. It has high practical value and explores a new path for EEG signal identity recognition.

[0073] The above parameters and steps for each module to implement corresponding functions in the EEG signal identification system 200 of this embodiment can refer to the parameters and steps in the embodiment of the EEG signal identification method above, and will not be repeated here.

[0074] like Figure 7As shown, an electronic device 300 according to an embodiment of the present invention includes a processor 320, the processor 320 is coupled to a memory 310, and the memory 310 stores at least one computer program 330, and the at least one computer program 330 is loaded and executed by the processor 320, so that the electronic device 300 implements any of the above-mentioned EEG signal identity recognition methods, specifically:

[0075] The electronic device 300 may have relatively large differences due to different configurations or performances, and may include one or more processors 320 (Central Processing Units, CPU) and one or more memories 310, wherein the one or more memories 310 store at least one computer program 330, and the at least one computer program 330 is loaded and executed by the one or more processors 320, so that the electronic device 300 implements any of the EEG signal identification methods provided in the above embodiments. Of course, the electronic device 300 may also have components such as a wired or wireless network interface, a keyboard, and an input and output interface for input and output, and the electronic device 300 may also include other components for implementing device functions, which will not be described in detail here.

[0076] A computer-readable storage medium according to an embodiment of the present invention stores at least one computer program, and the at least one computer program is loaded and executed by a processor so that a computer implements any of the above-mentioned EEG signal identity recognition methods.

[0077] Optionally, the computer-readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a compact disc (CD-ROM), a magnetic tape, a floppy disk, an optical data storage device, and the like.

[0078] In an exemplary embodiment, a computer program product or a computer program is also provided, the computer program product or the computer program comprising computer instructions, the computer instructions being stored in a computer-readable storage medium. A processor of an electronic device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the electronic device performs any of the above-mentioned methods for identifying an EEG signal.

[0079] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects and represent the definition of a specific order or sequence. The order of use of similar objects can be interchanged where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than the order shown or described.

[0080] Those skilled in the art know that the present invention can be implemented as a system, method or computer program product. Therefore, the present disclosure can be specifically implemented in the following forms, namely: it can be complete hardware, it can be complete software (including firmware, resident software, microcode, etc.), or it can be a combination of hardware and software, which is generally referred to as "circuit", "module" or "system" herein. In addition, in some embodiments, the present invention can also be implemented in the form of a computer program product in one or more computer-readable media, and the computer-readable medium contains computer-readable program code.

[0081] Any combination of one or more computer-readable media may be used. A computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium. A computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or device, or any combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present application, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in conjunction with an instruction execution system, device, or device.

[0082] Although the embodiments of the present invention have been shown and described above, it is to be understood that the above embodiments are exemplary and are not to be construed as limitations of the present invention. A person skilled in the art may change, modify, replace and vary the above embodiments within the scope of the present invention.

Claims

1. A method for identifying an individual using an electroencephalogram signal, characterized in that: include: When the target user views the target induced image, a plurality of facial images and original EEG signals of the target user are obtained; Based on the multiple facial images of the target user, determine the eye movement time period of the target user, and delete the EEG signal corresponding to the eye movement time period in the original EEG signal to obtain an optimized EEG signal and perform preprocessing to obtain a target EEG signal; The coding information of the target induced image and the coding information of the target electroencephalogram signal are multimodally encoded to obtain the multimodal coding information of the target user and input it into a trained multimodal large model to obtain the identity recognition result of the target user.

2. The method for identifying an individual using an EEG signal according to claim 1, wherein: The step of determining the eye movement time period of the target user based on the multiple facial images of the target user comprises: By using a feature point algorithm and according to changes in eye feature points in a plurality of facial images of the target user, an eye movement time period representing changes in eye features of the target user is determined.

3. The method for identifying an individual using an EEG signal according to claim 1, wherein: The preprocessing method is: noise removal.

4. The method for identifying an individual using an EEG signal according to claim 1, wherein: The training process of the multimodal large model is: The target EEG signals corresponding to multiple subjects when they watch different induced images are obtained, and multiple training data are generated according to the subject ID, induced image and target EEG signal as one training data, and the multiple training data are input into the multimodal large model for iterative training to obtain a trained multimodal large model.

5. The method for identifying an individual using an EEG signal according to any one of claims 1 to 4, characterized in that: The multimodal large model is a multimodal network based on the self-attention mechanism.

6. An EEG signal identification system, characterized in that: include: Acquisition module, processing module and recognition module; The acquisition module is used to: when the target user views the target induced image, acquire multiple facial images and original EEG signals of the target user; The processing module is used to: determine the eye movement time period of the target user based on the multiple facial images of the target user, and delete the EEG signal corresponding to the eye movement time period in the original EEG signal to obtain an optimized EEG signal and perform preprocessing to obtain a target EEG signal; The recognition module is used to: perform multimodal encoding on the coding information of the target induced image and the coding information of the target EEG signal to obtain the multimodal coding information of the target user and input it into the trained multimodal large model to obtain the identity recognition result of the target user.

7. The EEG signal identification system according to claim 6, characterized in that: The step of determining the eye movement time period of the target user based on the multiple facial images of the target user in the processing module includes: By using a feature point algorithm and according to changes in eye feature points in a plurality of facial images of the target user, an eye movement time period representing changes in eye features of the target user is determined.

8. The EEG signal identification system according to claim 6, characterized in that: The preprocessing method is: noise removal.

9. An electronic device, characterized in that: The electronic device includes a processor, the processor is coupled to a memory, the memory stores at least one computer program, and the at least one computer program is loaded and executed by the processor so that the electronic device implements the EEG signal identity recognition method as described in any one of claims 1 to 5.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores at least one computer program, and the at least one computer program is loaded and executed by the processor so that the computer-readable storage medium implements the EEG signal identity recognition method as described in any one of claims 1 to 5.

Images