Deaf-mute electroencephalogram signal intelligent interaction method based on deep learning

Through the intelligent interaction method of EEG signals of deaf and dumb people based on deep learning, using transfer learning and verification technology, the problems of intelligent interaction efficiency and personalized needs of deaf and dumb people are solved, achieving a more natural and accurate interaction experience and improving interaction efficiency.

CN118917380APending Publication Date: 2024-11-08NANTONG INST OF TECH
View PDF 8 Cites 0 Cited by

Patent Information

Application Number
CN202410956821.6
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-17
Publication Date
2024-11-08

AI Technical Summary

Technical Problem

The prior art is difficult to provide efficient and accurate intelligent interaction methods for deaf people, cannot communicate through traditional voices and languages, and lacks personalized interaction solutions.

Method used

Using the intelligent interaction method of EEG signals for deaf and dumb people based on deep learning, through initialization model, transfer learning, verification and training, combined with the dual choice of gender and age, transfer learning is used to reduce errors, and pass EEG signal verification and semantic information verification. Ensure the accuracy of the interaction.

Benefits of technology

It realizes a more natural and smooth interactive experience, improves the intelligent interaction efficiency between deaf and mute people and external devices, meets personalized needs, and simulates the human brain neural network to enable machines to understand complex data and provide accurate semantic information.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118917380A_ABST
    Figure CN118917380A_ABST
Patent Text Reader

Abstract

The invention provides a deaf-mute electroencephalogram signal intelligent interaction method based on deep learning, and relates to the field of computer sequence data, and the method comprises the following steps: S1, selecting an initialization model according to conditions; s2, carrying out transfer learning and training so as to realize specific optimization aiming at a user, and finally reducing errors; and S3, verifying the model and setting an error rate, if the error rate is lower than the error rate, the model finally passes verification and is put into use, and if the error rate is higher than the error rate, the model needs to be trained again. The invention aims to solve the problem that the current deaf-mute communication is inconvenient through the strong sequence data learning capability of deep learning. According to the method, a large amount of electroencephalogram data is combined with semantic information for labeling, a large amount of training is conducted, the process of converting the electroencephalogram data into the natural language (namely voice data) is achieved, and the method can be widely applied to the field of communication of the deaf-mute and promote communication between the deaf-mute and the normal person.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of computer sequence data, and in particular to a method for intelligent interaction of brain electrical signals of deaf-mute people based on deep learning. Background Art

[0002] In the field of intelligent interaction, with the continuous development of technology, people's demand for human-computer interaction methods is becoming increasingly diversified and personalized. Especially for the deaf-mute people, since they cannot communicate through traditional sounds and languages, it is particularly important to develop an efficient and accurate intelligent interaction method. In recent years, the rise of deep learning technology has provided new ideas for solving this problem. EEG signals are a direct reflection of the electrical activity of brain neurons and contain rich physiological and pathological information. For the deaf-mute, EEG signals are an important way for them to communicate with the outside world. By collecting, analyzing and processing EEG signals, intelligent interaction between the deaf-mute and external devices can be realized.

[0003] Therefore, using deep learning technology to apply EEG signals to intelligent interaction for deaf-mute people can greatly improve the efficiency of interaction. At the same time, each person's EEG signals are unique, and transfer learning can meet the personalized needs of specific people. Based on this, the present invention designs a deep learning-based intelligent interaction method for EEG signals for deaf-mute people. Summary of the invention

[0004] The purpose of the present invention is to solve the problems in the prior art and to propose an intelligent interaction method of EEG signals of the deaf-mute based on deep learning.

[0005] A method for intelligent interaction of brain electrical signals of deaf-mute people based on deep learning, comprising the following steps:

[0006] S1: Select the initialization model according to the conditions;

[0007] S2: Transfer learning and training to achieve user-specific optimization and finally reduce errors.

[0008] S3: Verify the model and set the error rate. If it is lower than the error rate, the model finally passes the verification and is put into use. If it is higher than the error rate, retraining is required.

[0009] In the first aspect, the present invention provides multiple initialized intelligent interaction models of EEG signals for deaf-mute people, including dual selection of gender and age, so as to facilitate the subsequent transfer learning. When the models are first activated, the gender and age will be selected for the user, and the age is presented in the form of segments. The specific segments are as follows: under 20 years old, 21-30 years old, 31-40 years old, 41-50 years old, 51-60 years old, 61-70 years old and 70 years old and above.

[0010] Optionally, in transfer learning, specific optimization is performed for the user to reduce errors, and the transfer learning method includes the setting of 100 common scenarios, each of which is set to be completed within 1 minute. The user only needs to make normal EEG signals according to the scenario used.

[0011] Optionally, after collecting the user's EEG signal, the model will be iteratively updated on the local machine, and then the EEG signal of the user will be specifically fitted.

[0012] Optionally, the present invention supports transfer learning in customized scenarios. When performing transfer learning for users, if the users have their own specific needs, EEG signals can be collected and semantic information can be provided.

[0013] Optionally, during the verification process, a systematic test is performed on the user-customized transfer learning model, wherein the verification method is divided into two parts: EEG signal verification and semantic information verification. The EEG signal verification method includes a 1-minute 12-point difference verification method. The specific verification method is as follows:

[0014] The first step is to select a specific scene. The scene here can be 100 common scenes or a custom scene, and obtain its pre-saved EEG signal.

[0015] In the second step, the EEG signal collected in the verification phase is collected at one point every 5 seconds, for a total of 12 points, and the difference at point 0 is used as the standard to calculate the error of the EEG signal in this scene. Assuming that the point values ​​in the original EEG signal are yt0,…yt11, and the collected point values ​​are y0,…,y11, the specific EEG signal error calculation method is as follows:

[0016]

[0017] Optionally, the voice information verification part is a method to determine whether the EEG signal inference results of the model are accurate. Here, it is assumed that the number of accurate inference results is n t , assuming that the number of categories tested is n, the final calculation method for the accuracy of semantic information is:

[0018]

[0019] Optionally, in order to balance the weight of EEG signal verification and semantic information verification, the present application example conducted multiple verifications, and finally the overall calculation method was implemented as follows:

[0020] error=α·point_error+(1-α)·info_error

[0021] Optionally, after testing, the default α value adopted in the example of this application is 0.4.

[0022] Optionally, custom adjustment is supported for the value of α to cope with special circumstances of different users.

[0023] Optionally, an error rate is set to confirm whether the model passes. The default threshold value set in this application example is 7%.

[0024] Compared with the prior art, the present invention has the following advantages:

[0025] 1. This method enables machines to learn and understand complex data by simulating the working mode of human brain neural network. By training the deep neural network model, the machine can recognize and understand the user's intention, thereby providing a more natural and smooth interactive experience. Using deep learning technology to apply EEG signals to intelligent interaction for deaf-mute people can greatly improve the efficiency of interaction.

[0026] 2. In the process of transfer learning, 100 common scenarios are prepared to facilitate users to conduct targeted transfer learning. Similarly, this application example supports custom scenarios. After the 100 common scenarios are selected, the custom scenarios can assist in entering EEG data to facilitate the training of the transfer model. The feasibility of the model can be verified more accurately through dual verification of EEG signal verification and semantic information verification and calculation of the overall error between the two. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 This is a flowchart of the use of a deep learning-based intelligent interaction method for brain electrical signals of the deaf-mute proposed by the present invention.

[0028] Figure 2 This is the structure of the initialization model in the intelligent interaction method of EEG signals for the deaf-mute based on deep learning proposed by the present invention.

[0029] Figure 3 This is the physical topological structure of the intelligent interaction method of EEG signals for the deaf-mute based on deep learning proposed by the present invention.

[0030] Figure 4 This is the EEG data waveform in the deep learning-based intelligent interaction method for EEG signals of the deaf-mute proposed by the present invention. DETAILED DESCRIPTION

[0031] See also Figure 1-4 , Figure 1 A flowchart of a method for intelligent interaction of brain electrical signals of deaf-mute people based on deep learning is provided in this application example. Figure 1 The steps in are described in detail.

[0032] Load the initialization model. This step requires screening the initialization model based on the gender and age of the deaf-mute people and loading it.

[0033] Also, see Figure 3 In this step, you need to use a wearable EEG signal sensor, which has a total of 5 tiny sensors to collect EEG signals from different parts of the brain. After completing the sensor wearing, you need to connect the EEG sensor to the smart terminal via Bluetooth.

[0034] To conduct transfer learning, after the initialization model is loaded, the EEG sensor is installed, and the sensor is connected to the smart terminal, it is necessary to start training the transfer learning model for a specific user.

[0035] Initialize the model structure as Figure 2 As shown in the figure, it is built based on the Transformer structure. After the initialization model is loaded, the smart terminal will display scenes in life in the form of text, for example: "Call xx". The user only needs to follow the text prompts and execute this instruction in the brain. The sensor will then capture the EEG signal and record it for the final test.

[0036] In the process of transfer learning, 100 common scenarios are prepared to facilitate users to conduct targeted transfer learning. Similarly, this application example supports custom scenarios. After the 100 common scenarios are selected, the custom scenario requires other people to assist in entering EEG data to facilitate the subsequent training of the transfer model.

[0037] After the EEG data of common scenarios and custom scenarios are recorded, model training begins.

[0038] The initialized weight file during the training process comes from the initialized model, and a small amount of training is required on this basis.

[0039] Verify the results of transfer learning. After the transfer learning is completed, the quality of the model needs to be tested. This application example sets up two parts: EEG signal verification and semantic information verification.

[0040] EEG signal verification is to ensure the trend consistency of EEG signals, and semantic information verification is to ensure that the meaning of the instructions represented by the EEG signals is accurate. Assuming that the point values ​​in the original EEG signal are yt0,…yt11, and the collected point values ​​are y0,…,y11, the specific EEG signal error calculation method is as follows:

[0041]

[0042] Assume that the number of accurate inference results is n t, assuming that the number of categories tested is n, the final calculation method for the accuracy of semantic information is:

[0043]

[0044] Finally, the overall calculation method is as follows:

[0045] error=α·point_error+(1-α)·info_error

[0046] The value of α here is set to 0.4 by default, and can be changed accordingly according to the specific user.

[0047] The default error rate is 7%. If it is lower than 7%, the model will eventually pass the verification and be put into use. If it is higher than 7%, retraining is required.

[0048] It is known from common technical knowledge that the present invention can be implemented by other embodiments that do not deviate from its spirit or essential features. Therefore, the above disclosed embodiments are only illustrative in all respects and are not exclusive. All changes within the scope of the present invention or within the scope equivalent to the present invention are included in the present invention.

Claims

1. A deep learning-based intelligent interaction method for brain wave signals of deaf-mute people, characterized in that: The following steps are involved: S1: Select the initialization model according to the conditions; S2: Transfer learning and training to achieve user-specific optimization and finally reduce errors. S3: Verify the model and set the error rate. If it is lower than the error rate, the model finally passes the verification and is put into use. If it is higher than the error rate, retraining is required.

2. According to claim 1, a method for intelligent interaction of brain electrical signals of deaf-mute people based on deep learning is characterized by: The initialization model selection conditions include the user's gender and age. The age segments included in the initialization model selection process are under 20 years old, 21-30 years old, 31-40 years old, 41-50 years old, 51-60 years old, 61-70 years old and 70 years old and above.

3. According to claim 1, a method for intelligent interaction of brain electrical signals for deaf-mute people based on deep learning, characterized in that: In transfer learning, 100 common scenarios are used to fine-tune the initialization model for the user. The initialization model is iteratively updated on the local machine using the EEG signal data of the user in 100 scenarios, and then the EEG signal of the user is fit in a targeted manner.

4. According to claim 1, a method for intelligent interaction of brain electrical signals of deaf-mute people based on deep learning is characterized by: The verification model mainly includes EEG signal verification and semantic information verification.

5. The method for intelligent interaction of brain electrical signals of deaf-mute people based on deep learning according to claim 4 is characterized by: Semantic information verification is performed by judging whether the EEG signal information inferred by the model is accurate.

6. The method for intelligent interaction of brain electrical signals of deaf-mute people based on deep learning according to claim 4 is characterized by: The method of EEG signal verification includes the 1-minute 12-point difference verification method. The specific steps of the difference verification method include: S1: Select a specific scene, which can be 100 common scenes or a custom scene, and obtain its pre-saved EEG signal. S2: Collect one point of the EEG signal collected in the verification phase every 5 seconds, for a total of 12 points, and use the difference at point 0 as the standard to calculate the error of the EEG signal in this scene. Assuming that the point values ​​in the original EEG signal are yt0,…yt11, and the collected point values ​​are y0,…,y11, the specific EEG signal error calculation method is as follows: Among them, the voice information verification part is a method to determine whether the EEG signal inference results of the model are accurate. Here, it is assumed that the number of accurate inference results is n t , assuming that the number of categories tested is n, the final calculation method for the accuracy of semantic information is: In order to balance the weight of EEG signal verification and semantic information verification, the overall calculation method is finally implemented as follows: error=α·point_error+(1-α)·info_error Among them, the α value is set as the error and is used to determine the weight relationship.

7. The method for intelligent interaction of brain electrical signals of deaf-mute people based on deep learning according to claim 6 is characterized by: The default value of α is set to 0.4, and the default error rate is 7%. If it is lower than 7%, the model will eventually pass the verification and be put into use. If it is higher than 7%, it needs to be retrained.

Citation Information

Patent Citations

  • Idiodynamic system based on deep neural networks

    CN107049308A

  • Deep learning man-machine interaction motor imagery brain-computer interface system and training method

    CN110534180A

  • Brain-myoelectricity intelligent whole limb rehabilitation method based on novel transfer learning model

    CN111544856A

  • Electroencephalogram emotion classification method based on multi-scale connectivity features and meta transfer learning

    CN113057657A

  • ALS patient man-machine interaction system and method based on brain-computer interface and artificial intelligence algorithm

    CN116400800A