Sample similarity knowledge distillation method and device based on multichannel electroencephalogram

Abstract

The invention provides a sample similarity knowledge distillation method and device based on multi-channel electroencephalogram, and the method comprises the steps: a first calculation step: inputting the whole brain electroencephalogram data of a sample into a pre-trained teacher network; a second calculation step: sequentially inputting first input data in the multiple groups of current first training data into the student network for training; a verification step: performing comparison verification according to all the student tags and the corresponding soft tags to obtain the accuracy of the student network; and under the condition that the accuracy does not meet the accuracy requirement, optimizing the student network, and sequentially and repeatedly executing the first calculation step, the second calculation step and the verification step at least once until the accuracy meets the accuracy requirement, thereby obtaining the lightweight student network. The problem that in the prior art, traditional electroencephalogram data knowledge distillation neglects the internal similarity relation between input samples, and consequently the expression ability of a student network is insufficient in the learning process is solved.

Description

Technical Field

[0001] The present invention relates to the technical field of multi-channel EEG recognition, and in particular to a method for sample similarity knowledge distillation based on multi-channel EEG, a device for sample similarity knowledge distillation based on multi-channel EEG, a computer-readable storage medium and a computer program product. Background Art

[0002] Multi-channel EEG is widely used in real-world technologies such as emotion recognition, epilepsy detection, and brain-computer interfaces. In actual applications, most scenarios use a whole-brain EEG acquisition method based on the 10-20 system, and the deep learning model trained based on this is also based on the whole brain. In addition, neuroscience has also found that the human brain is most related to cognition in the prefrontal region, which highly concentrates cognitive information. Current research has carried out channel dimensionality reduction and knowledge distillation methods to improve the operating efficiency of the network. Designing a knowledge distillation network structure for the whole-brain network model to achieve excellent performance even with fewer EEG channels is the key to studying neural networks.

[0003] Knowledge distillation is a deep learning model compression technology that simplifies the model structure and improves reasoning efficiency by transferring the knowledge of a complex large model (teacher model) to a smaller model (student model). During the knowledge distillation process, the student model learns similar feature representations and decision-making capabilities by imitating the predicted output or intermediate representation of the teacher model. This method can not only maintain high accuracy, but also significantly reduce the computational overhead and storage requirements of the model, so it has important application value in resource-constrained environments. There is a certain degree of similarity between the samples in the pre-trained teacher network and the student network. The traditional pre-trained teacher network and student network models lack consideration of the similarity of samples in the two networks, resulting in the inability to capture the intrinsic structure and relationship between the input samples, thereby reducing the expressive power of the student network. In addition, the whole-brain multi-channel EEG model relies on more channels and has high computational complexity; the traditional multi-channel EEG knowledge distillation ignores the intrinsic similarity relationship between input samples, resulting in insufficient expressive power of the student network during the learning process; the efficiency of multi-channel EEG knowledge distillation is insufficient, resulting in the difficulty of the student network to maintain excellent performance with fewer EEG channels during the learning process. Summary of the invention

[0004] The main purpose of the present application is to provide a method for sample similarity knowledge distillation based on multi-channel EEG, a device for sample similarity knowledge distillation based on multi-channel EEG, a computer-readable storage medium and a computer program product, so as to at least solve the problem in the prior art that traditional EEG data knowledge distillation ignores the intrinsic similarity relationship between input samples, resulting in insufficient expression ability of the student network during the learning process.

[0005] In order to achieve the above-mentioned purpose, according to one aspect of the present application, a sample similarity knowledge distillation method based on multi-channel EEG is provided, including: a first calculation step, inputting the sample whole-brain EEG data into a pre-trained teacher network, so as to use the pre-trained teacher network to analyze and process the sample whole-brain EEG data to obtain a first similarity matrix, a first output vector and a soft label, the pre-trained teacher network is a convergent network model obtained by inputting multiple groups of sample training data into a graph convolutional neural network for training, each group of the sample training data includes sample whole-brain EEG data, and sample EEG labels, the first similarity matrix is ​​the weighted cosine similarity matrix of the pre-trained teacher network in the last layer of the graph convolutional neural network. The first output vector is obtained by calculation, the first output vector is the original output vector output by the pre-trained teacher network without being processed by the activation function, the soft label is the label value obtained after the first output vector is processed by the activation function, and the label value is used to indicate whether the staff is in a fatigue state; the second calculation step is to input the first input data in multiple groups of current first training data into the student network in sequence, so as to use the student network to calculate the similarity matrix of the current first training data, and obtain the second similarity matrix, the second output vector and the student label, the current first training data is any group of first training data in the first training data set, and the first training data set includes multiple groups of the first training data, each group The first training data includes the first input data and the first output data, the first input data is the sample prefrontal three-channel EEG data, the first output data is the sample EEG label corresponding to the sample prefrontal three-channel EEG data, the second output vector is the original output vector output by the pre-trained teacher network without being processed by the activation function, the student label is the label value obtained after the second output vector is processed by the activation function, and the sample prefrontal three-channel EEG data is the data extracted from the sample whole-brain EEG data; the verification step is to compare and verify all the student labels with the corresponding soft labels to obtain the accuracy of the student network; if the accuracy does not meet the requirements, Under the accuracy requirement, the first similarity matrix, the second similarity matrix, the first output vector and the second output vector are input into the overall loss function of the student network for optimization, and the first calculation step, the second calculation step and the verification step are sequentially repeated at least once until the accuracy meets the above accuracy requirement, and a lightweight student network is obtained, and the overall loss function is a loss function composed of a mean square error loss function, a cross entropy loss function and a KL divergence loss function; the three-channel EEG data of the prefrontal lobe of the staff member is obtained in real time, and the three-channel EEG data of the prefrontal lobe is input into the lightweight student network to obtain a true label, and the true label is 1 or 0;When the true label is 1, a prompt message is issued, wherein the prompt message is used to prompt the staff member that the staff member is currently in the fatigue state. ;

[0006] Optionally, before inputting the sample whole-brain EEG data into a pre-trained teacher network to analyze and process the sample whole-brain EEG data using the pre-trained teacher network to obtain a first similarity matrix, a first output vector and a soft label, the method also includes: filtering the original sample whole-brain EEG data using a bandpass filtering method to filter out low-frequency noise and high-frequency noise, and using a notch filter to eliminate common frequency noise to obtain first sample data; normalizing the first sample data to obtain the sample whole-brain EEG data.

[0007] Optionally, before the first input data in multiple groups of current first training data are sequentially input into the student network to calculate the similarity matrix of the current first training data using the student network to obtain a second similarity matrix, a second output vector and a student label, the method also includes: filtering the original second training data using a bandpass filtering method to filter out low-frequency noise and high-frequency noise, and using a notch filter to eliminate common frequency noise to obtain second sample data; normalizing the second sample data to obtain the sample prefrontal three-channel EEG data.

[0008] Optionally, the sample whole-brain EEG data is input into a pre-trained teacher network, and the pre-trained teacher network is used to analyze and process the sample whole-brain EEG data to obtain a first similarity matrix, a first output vector and a soft label, including: using a first formula in the pre-trained teacher network to perform matrix calculation of a multi-layer graph convolutional neural network to obtain a teacher network output matrix corresponding to each of the sample whole-brain EEG data, wherein the first formula is Among them, H (l) represents the feature matrix of the graph convolutional neural network in layer l, H (l+1) represents the output matrix of the graph convolutional neural network described in the lth layer, and also represents the input matrix of the graph convolutional neural network described in the l+1th layer, is the connection matrix, for The degree matrix, W (l) is a learnable matrix, the teacher network output matrix is ​​the output matrix of the last layer of the graph convolutional neural network, and one sample whole-brain EEG data corresponds to one teacher network output matrix; the weighted cosine similarity is calculated according to the teacher network output matrix corresponding to each sample whole-brain EEG data to obtain the first similarity matrix; the convolution calculation is performed using the layer graph convolutional neural network in the pre-trained teacher network to obtain the first output vector and the soft label.

[0009] Optionally, the weighted cosine similarity is calculated according to the teacher network output matrix corresponding to each of the sample whole-brain EEG data to obtain the first similarity matrix, including: calculating the mean vector of the teacher network output matrix corresponding to each of the sample whole-brain EEG data, the mean vector is a vector formed by the mean of each row element of the teacher network output matrix; calculating the weighted cosine similarity between the mean vectors corresponding to each of the sample whole-brain EEG data according to the second formula, the second formula is represents the mean vector of the first sample whole-brain EEG data, represents the mean vector of the second sample whole-brain EEG data, The first sample whole-brain EEG data is one of any two sample whole-brain EEG data, the second sample whole-brain EEG data is the sample whole-brain EEG data of any two sample whole-brain EEG data except the first sample whole-brain EEG data, W is a weight vector, W=[w1, w2…, w n ]; Generate the first similarity matrix according to the weighted cosine similarities corresponding to all the sample whole-brain EEG data, the elements of the first similarity matrix are the weighted cosine similarities, the rows and columns of the first similarity matrix represent the sample whole-brain EEG data, and the first similarity matrix is ​​a symmetric matrix.

[0010] Optionally, before inputting the first similarity matrix, the second similarity matrix, the first output vector and the second output vector into the overall loss function of the student network for optimization, the method further includes: determining the overall loss function based on the mean square error loss function, the cross entropy loss function and the KL divergence loss function, and the expression of the overall loss function is Loss = (1-α)·Loss CE (y,σ(z s ))+ y is the student label, Loss CE () represents the cross entropy loss function, KL() is the KL divergence loss function, Loss SK is the mean square error loss function, z T is the first output vector, z S For the second output vector, T and α are the temperature coefficient and the balancing weight respectively.

[0011] Optionally, after acquiring the three-channel EEG data of the staff's frontal lobe in real time and inputting the three-channel EEG data of the frontal lobe into the lightweight student network to obtain the true label, the method also includes: when the true label is 0, determining that the staff is currently in a non-fatigue state.

[0012] According to another aspect of the present application, a sample similarity knowledge distillation device based on multi-channel EEG is provided, the device comprising: a first computing unit, for executing a first computing step, inputting sample whole-brain EEG data into a pre-trained teacher network, so as to use the pre-trained teacher network to analyze and process the sample whole-brain EEG data to obtain a first similarity matrix, a first output vector and a soft label, the pre-trained teacher network is a convergent network model obtained by inputting multiple groups of sample training data into a graph convolutional neural network for training, each group of the sample training data includes sample whole-brain EEG data, and sample EEG labels, the first similarity matrix is ​​the pre-trained teacher network in the last layer of the graph convolutional neural network The first output vector is obtained by calculating the weighted cosine similarity of the pre-trained teacher network, the first soft label is the label value obtained after the first output vector is processed by the activation function, and the label value is used to indicate whether the staff is in a fatigue state; the second computing unit is used to execute the second computing step, and the first input data in multiple groups of current first training data are sequentially input into the student network, so as to analyze the first input data by using the student network to obtain a second similarity matrix, a second output vector and a student label, and the current first training data is any group in the first training data set first training data, the first training data set includes multiple groups of the first training data, each group of the first training data includes the first input data and the first output data, the first input data is the sample prefrontal three-channel EEG data, the first output data is the sample EEG label corresponding to the sample prefrontal three-channel EEG data, the second output vector is the original output vector output by the pre-trained teacher network without being processed by the activation function, the student label is the label value obtained after the second output vector is processed by the activation function, and the sample prefrontal three-channel EEG data is the data extracted from the sample whole-brain EEG data; a verification unit is used to perform a verification step , comparing and verifying all the student labels with the corresponding soft labels to obtain the accuracy of the student network; an optimization iteration unit, used for inputting the first similarity matrix, the second similarity matrix, the first output vector and the second output vector into the overall loss function of the student network for optimization when the accuracy does not meet the accuracy requirement, and repeating the first calculation step, the second calculation step and the verification step at least once in sequence until the accuracy meets the above accuracy requirement, and obtaining a lightweight student network, wherein the overall loss function is a loss function composed of a mean square error loss function, a cross entropy loss function and a KL divergence loss function;An input unit is used to obtain the three-channel EEG data of the frontal lobe of the staff member in real time, and input the three-channel EEG data of the frontal lobe into the lightweight student network to obtain a true label, and the true label is 1 or 0; a prompt unit is used to issue a prompt message when the true label is 1, and the prompt message is used to prompt that the staff member is currently in the fatigue state. ;

[0013] According to another aspect of the present application, a computer-readable storage medium is provided, wherein the computer-readable storage medium includes a stored program, wherein when the program is executed, the device where the computer-readable storage medium is located is controlled to execute any one of the methods described.

[0014] According to another aspect of the present application, a computer program product is provided, comprising computer instructions, wherein when the computer instructions are executed by a processor, any one of the methods described above is implemented.

[0015] Applying the technical solution of the present application, in the knowledge distillation method of sample similarity based on multi-channel EEG, the present invention proposes a lightweight processing method of EEG signals based on knowledge distillation, by inputting high-density (whole brain) EEG data into the teacher network, and low-density (three channels of the prefrontal lobe) EEG data into the student network, and using the knowledge of the teacher network to guide the training of the student network, so that the student network can achieve performance close to that of the teacher network when relying only on low-density EEG data. Finally, the student network can achieve results comparable to the performance of the teacher network by using low-density EEG data combined with knowledge distillation learning. Thereby achieving lightweight use in reality. The final effect is to use low-density EEG data to realize the classification of fatigue and non-fatigue. The present application solves the problem in the prior art that traditional EEG data knowledge distillation ignores the intrinsic similarity relationship between input samples, resulting in insufficient expression ability of the student network during the learning process. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 A hardware structure block diagram of a mobile terminal for executing a multi-channel EEG-based sample similarity knowledge distillation method provided in an embodiment of the present application is shown;

[0017] Figure 2 A schematic diagram of a process of a multi-channel EEG-based sample similarity knowledge distillation method provided according to an embodiment of the present application is shown;

[0018] Figure 3 A schematic diagram of a multi-channel EEG-based sample similarity knowledge distillation method provided according to an embodiment of the present application is shown;

[0019] Figure 4 A structural block diagram of a multi-channel EEG-based sample similarity knowledge distillation device provided according to an embodiment of the present application is shown.

[0020] The above drawings include the following reference numerals:

[0021] 102, processor; 104, memory; 106, transmission device; 108, input and output devices. DETAILED DESCRIPTION

[0022] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0023] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.

[0024] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequential order. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present application described here. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0025] For the convenience of description, some nouns or terms involved in the embodiments of the present application are explained below:

[0026] Multi-channel EEG: refers to EEG signals collected simultaneously through multiple electrodes. These signals come from electrodes at different locations in the brain and are used to record the brain's electrical activity. Multi-channel EEG can provide richer brain information for more accurate analysis and processing;

[0027] Sample similarity: refers to the technology of calculating similarity or distance between data samples. In this invention, sample similarity is used to measure the similarity of samples between the pre-trained teacher network and the student network to enhance the effect of knowledge distillation;

[0028] Knowledge distillation: A model compression technique that simplifies the model structure and improves inference efficiency by transferring the knowledge of a large teacher model to a smaller student model. The predictions or intermediate representations provided by the teacher model are used to train the student model, reducing computational and storage requirements while maintaining high performance.

[0029] As introduced in the background technology, the traditional pre-trained teacher network and student network models in the prior art lack consideration of the sample similarity in the two networks, resulting in the inability to capture the intrinsic structure and relationship between the input samples, thereby resulting in a decrease in the expressive ability of the student network. In addition, the whole-brain multi-channel EEG model relies on more channels and has high computational complexity; the traditional multi-channel EEG knowledge distillation ignores the intrinsic similarity relationship between input samples, resulting in insufficient expressive ability of the student network during the learning process; the efficiency of multi-channel EEG knowledge distillation is insufficient, resulting in the difficulty of the student network to maintain excellent performance with fewer EEG channels during the learning process. In order to solve the problem that the traditional EEG data knowledge distillation in the prior art ignores the intrinsic similarity relationship between input samples, resulting in insufficient expressive ability of the student network during the learning process, the embodiments of the present application provide a method for sample similarity knowledge distillation based on multi-channel EEG, a device for sample similarity knowledge distillation based on multi-channel EEG, a computer-readable storage medium, and a computer program product.

[0030] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the accompanying drawings in the embodiments of the present invention.

[0031] The method embodiments provided in the embodiments of the present application can be executed in a mobile terminal, a computer terminal or a similar computing device. Taking running on a mobile terminal as an example, Figure 1 1 is a hardware structure block diagram of a mobile terminal of a multi-channel EEG sample similarity knowledge distillation method according to an embodiment of the present invention. Figure 1 As shown, the mobile terminal may include one or more ( Figure 1 Only one is shown in the figure) a processor 102 (the processor 102 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA) and a memory 104 for storing data, wherein the mobile terminal may also include a transmission device 106 and an input / output device 108 for communication functions. It can be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the mobile terminal. Figure 1 More or fewer components as shown, or with Figure 1 Different configurations are shown.

[0032] The memory 104 can be used to store computer programs, for example, software programs and modules of application software, such as the computer program corresponding to the sample similarity knowledge distillation method based on multi-channel EEG in the embodiment of the present invention. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, that is, the above method is implemented. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include a memory remotely arranged relative to the processor 102, and these remote memories can be connected to the mobile terminal via a network. Examples of the above-mentioned network include but are not limited to the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof. The transmission device 106 is used to receive or send data via a network. The above-mentioned specific examples of the network may include a wireless network provided by a communication provider of the mobile terminal. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, referred to as NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 may be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0033] In this embodiment, a multi-channel EEG sample similarity knowledge distillation method running on a mobile terminal, a computer terminal or a similar computing device is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although the logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0034] Figure 2 is a flow chart of a method for sample similarity knowledge distillation based on multi-channel EEG according to an embodiment of the present application. Figure 2 As shown, the method comprises the following steps:

[0035] Step S201, the first calculation step, inputs the sample whole-brain EEG data into the pre-trained teacher network, so as to use the above-mentioned pre-trained teacher network to analyze and process the above-mentioned sample whole-brain EEG data, and obtain a first similarity matrix, a first output vector and a soft label. The above-mentioned pre-trained teacher network is a convergent network model obtained by inputting multiple groups of sample training data into a graph convolutional neural network for training. Each group of the above-mentioned sample training data includes sample whole-brain EEG data, and sample EEG labels. The above-mentioned first similarity matrix is ​​obtained by the weighted cosine similarity calculation of the above-mentioned pre-trained teacher network in the last layer of the graph convolutional neural network. The above-mentioned first output vector is the original output vector output by the above-mentioned pre-trained teacher network without being processed by the activation function. The above-mentioned soft label is the label value obtained after the above-mentioned first output vector is processed by the above-mentioned activation function. The above-mentioned label value is used to indicate whether the staff is in a fatigue state.

[0036] Specifically, the weight parameters of the teacher network are optimized in the training set training, and the overfitting is evaluated by comparing the model accuracy of the training and verification processes to obtain a high-precision and low-overfitting pre-trained teacher network. The sample whole-brain EEG data is input into the pre-trained teacher network. The sample whole-brain EEG data is analyzed and processed using the pre-trained teacher network. The pre-trained teacher network obtains a first similarity matrix, a first output vector, and a soft label. The pre-trained teacher network is trained by a graph convolutional neural network (GCN), which can process multiple sets of sample training data, including whole-brain EEG data and corresponding EEG labels. The first similarity matrix reflects the similarity between the sample EEG data, which is helpful for subsequent analysis. The first output vector is the original output of the network, and the soft label is the label value processed by the activation function, which is used to indicate whether the staff is in a state of fatigue.

[0037] Step S202, the second calculation step, sequentially inputs the first input data in multiple groups of current first training data into the student network, so as to use the above-mentioned student network to calculate the similarity matrix of the above-mentioned current first training data, and obtain a second similarity matrix, a second output vector and a student label, wherein the above-mentioned current first training data is any group of first training data in the first training data set, and the above-mentioned first training data set includes multiple groups of the above-mentioned first training data, and each group of the above-mentioned first training data includes the above-mentioned first input data and the first output data, and the above-mentioned first input data is the sample prefrontal three-channel EEG data, and the above-mentioned first output data is the sample EEG label corresponding to the above-mentioned sample prefrontal three-channel EEG data, and the above-mentioned second output vector is the original output vector output by the above-mentioned pre-trained teacher network without being processed by the activation function, and the above-mentioned student label is the label value obtained after the above-mentioned second output vector is processed by the above-mentioned activation function, and the above-mentioned sample prefrontal three-channel EEG data is the data extracted from the above-mentioned sample whole-brain EEG data.

[0038] Specifically, by designing a distillation network structure based on sample similarity, it is ensured that the student network can achieve excellent performance when using fewer EEG channels, solving the problem of insufficient expression ability of the student network in traditional methods. Therefore, the first input data (sample prefrontal three-channel EEG data) in the current first training data is input into the student network. The student network is used to calculate the similarity matrix of the current first training data. The second similarity matrix, the second output vector and the student label are obtained from the student network. The student network learns how to identify fatigue status by processing the sample prefrontal three-channel EEG data. The second similarity matrix and the second output vector are the outputs of the student network for subsequent verification and optimization. A knowledge distillation strategy based on sample similarity is introduced to strengthen the training process of the student network by constructing a sample similarity matrix, so that the student network can better capture the intrinsic structure and similarity relationship between input samples, thereby improving its expression ability.

[0039] Step S203, a verification step, compares and verifies all the above student labels with the corresponding above soft labels to obtain the accuracy of the above student network.

[0040] Specifically, the student labels output by the student network are compared with the soft labels of the corresponding pre-trained teacher network to obtain the accuracy of the student network. This is used to evaluate the performance of the student network and ensure that it can accurately identify fatigue states. Figure 3 As shown in the figure, the EEG signals are processed in two ways: high-density EEG (whole brain) EEG data and low-density EEG data. The high-density EEG EEG data is input to the teacher network, and the low-density EEG EEG data (frontal three-channel EEG data) is input to the student network. It is hoped that the student network can achieve the same performance as the teacher network by combining low-density EEG data with the distillation learning. This will achieve lightweight use in reality. The final effect is: using low-density EEG to achieve the classification of fatigue and non-fatigue.

[0041] Step S204, when the accuracy does not meet the accuracy requirement, the first similarity matrix, the second similarity matrix, the first output vector and the second output vector are input into the overall loss function of the student network for optimization, and the first calculation step, the second calculation step and the verification step are repeated at least once in sequence until the accuracy meets the accuracy requirement, and a lightweight student network is obtained, and the overall loss function is a loss function composed of a mean square error loss function, a cross entropy loss function and a KL divergence loss function.

[0042] Specifically, if the accuracy of the student network does not meet the requirements, the first similarity matrix, the second similarity matrix, the first output vector and the second output vector are input into the overall loss function of the student network for optimization. Repeat the first calculation step, the second calculation step and the verification step at least once until the accuracy meets the requirements. Improve the performance of the student network by optimizing the loss function (including the mean square error loss function, the cross entropy loss function and the KL divergence loss function). Repeat the training and verification process until a satisfactory accuracy is achieved to ensure that the student network can accurately identify the fatigue state.

[0043] Step S205, obtaining the three-channel EEG data of the prefrontal lobe of the above-mentioned staff in real time, and inputting the three-channel EEG data of the prefrontal lobe into the above-mentioned lightweight student network to obtain a true label, and the above-mentioned true label is 1 or 0.

[0044] Specifically, the three-channel EEG data of the frontal lobe of the staff is obtained in real time. These data are input into the lightweight student network. The true label (1 or 0) is obtained. The EEG data of the staff is monitored in real time to identify the fatigue state in time. The lightweight student network can quickly process the data and give the results.

[0045] Step S206, when the true label is 1, a prompt message is issued, wherein the prompt message is used to prompt the staff member that the staff member is currently in the fatigue state.

[0046] Specifically, if the true label is 1 (indicating that the worker is in a fatigued state), a prompt message is issued. The prompt message can remind the worker to take a break and avoid accidents or reduced efficiency due to fatigue. This real-time feedback helps improve the work efficiency and safety of the worker.

[0047] In this embodiment, first, the first calculation step is to input the sample whole-brain EEG data into the pre-trained teacher network, so as to use the above-mentioned pre-trained teacher network to analyze and process the above-mentioned sample whole-brain EEG data to obtain a first similarity matrix, a first output vector and a soft label. The above-mentioned pre-trained teacher network is a convergent network model obtained by inputting multiple groups of sample training data into the graph convolutional neural network for training. Each group of the above-mentioned sample training data includes sample whole-brain EEG data and sample EEG labels. The above-mentioned first similarity matrix is ​​obtained by the above-mentioned pre-trained teacher network in the last layer of the graph convolutional neural network by weighted cosine similarity calculation. The above-mentioned first output vector is the original output vector output by the above-mentioned pre-trained teacher network without being processed by the activation function. The soft label is the label value obtained after the first output vector is processed by the activation function, and the label value is used to indicate whether the staff is in a fatigue state; then, in the second calculation step, the first input data in the multiple sets of current first training data are sequentially input into the student network, so as to use the student network to calculate the similarity matrix of the current first training data, and obtain the second similarity matrix, the second output vector and the student label, the current first training data is any set of first training data in the first training data set, the first training data set includes multiple sets of the first training data, each set of the first training data includes the first input data and the first output data, and the first input data is the sample prefrontal three-channel EEG data , the first output data is the sample EEG label corresponding to the sample prefrontal three-channel EEG data, the second output vector is the original output vector output by the pre-trained teacher network without being processed by the activation function, the student label is the label value obtained after the second output vector is processed by the activation function, and the sample prefrontal three-channel EEG data is the data extracted from the sample whole-brain EEG data; then, a verification step is performed to compare and verify all the student labels with the corresponding soft labels to obtain the accuracy of the student network; then, if the accuracy does not meet the accuracy requirement, the first similarity matrix, the second similarity matrix, the first output vector and the second output vector are compared and verified. The overall loss function input into the student network is optimized, and the first calculation step, the second calculation step and the verification step are repeatedly performed at least once in sequence until the accuracy meets the accuracy requirement, and a lightweight student network is obtained. The overall loss function is a loss function composed of a mean square error loss function, a cross entropy loss function and a KL divergence loss function. After that, the three-channel EEG data of the prefrontal lobe of the staff member is obtained in real time, and the three-channel EEG data of the prefrontal lobe is input into the lightweight student network to obtain a true label, and the true label is 1 or 0. Finally, when the true label is 1, a prompt message is issued, and the prompt message is used to prompt the staff member that he is currently in the fatigue state.The present invention proposes a lightweight processing method for EEG signals based on knowledge distillation, by inputting high-density (whole brain) EEG data into a teacher network, and low-density (three channels of the frontal lobe) EEG data into a student network, and using the knowledge of the teacher network to guide the training of the student network, so that the student network can achieve performance close to that of the teacher network while relying only on low-density EEG data. Finally, the student network can achieve results comparable to the performance of the teacher network by using low-density EEG data combined with knowledge distillation learning. Thus, lightweight use in reality is achieved. The final effect is to use low-density EEG data to achieve the classification of fatigue and non-fatigue. The present application solves the problem in the prior art that traditional EEG data knowledge distillation ignores the intrinsic similarity relationship between input samples, resulting in insufficient expression ability of the student network during the learning process.

[0048] In order to enable those skilled in the art to more clearly understand the technical solution of the present application, the implementation process of the multi-channel EEG-based sample similarity knowledge distillation method of the present application will be described in detail below in combination with specific embodiments.

[0049] In order to improve the data quality of the pre-trained teacher network, in an optional implementation manner, before the above step S201, the method further includes:

[0050] Step S301, using a bandpass filtering method to filter the original sample whole-brain EEG data to filter out low-frequency noise and high-frequency noise, and using a notch filter to eliminate common frequency noise, thereby obtaining first sample data;

[0051] Step S302, normalizing the first sample data to obtain the sample whole-brain EEG data.

[0052] In the above embodiment, the sample whole-brain EEG data is cut into a training set and a validation set as the network input data of the teacher network. Prior to this, the original sample whole-brain EEG data needs to be pre-processed for noise reduction, de-aliasing and normalization. Specifically, low-frequency and high-frequency noise are filtered out by a 0.1-50Hz bandpass filter, and common frequency noise is eliminated by a 50Hz notch filter. In order to eliminate the amplitude difference caused by different EEG channels, the EEG signal is adjusted to a data distribution with a mean of 0 and a standard deviation of 1 by Z-score normalization. A suitable bandpass filter is selected, usually a digital filter, such as a Butterworth filter, a Chebyshev filter or an elliptical filter. The cutoff frequency of the filter is set to filter out low-frequency noise (such as frequency components less than 0.5Hz, which are usually related to power supply frequency interference) and high-frequency noise (such as frequency components higher than 40Hz, which may be related to myoelectric interference). The filter is applied to the original EEG data to retain the main frequency range of brain activity (usually between 1Hz and 40Hz). Identify common frequency noise, such as power frequency (50Hz or 60Hz) and its harmonics, design notch filters to specifically filter out these frequencies, and apply the notch filter to the bandpass filtered data to eliminate the common frequency noise. By filtering out low-frequency and high-frequency noise, the quality of the EEG signal can be significantly improved and interference during the analysis process can be reduced. The notch filter specifically targets common frequency noise and can effectively eliminate these periodic interferences and improve the signal-to-noise ratio of the signal. By retaining the frequency range of 1Hz to 40Hz, it can ensure that useful information related to cognitive and emotional states in the EEG signal is retained. Normalization can eliminate the impact of different dimensions and magnitudes on data comparison, so that data from different channels or different individuals can be directly compared. Many machine learning algorithms are sensitive to the scale of the data, and normalization can improve the performance and convergence speed of these algorithms.

[0053] In order to improve the training data quality of the student network, in an optional implementation manner, before the above step S202, the method further includes:

[0054] Step S401, using a bandpass filtering method to filter the original second training data to remove low-frequency noise and high-frequency noise, and using a notch filter to eliminate common frequency noise, to obtain second sample data;

[0055] Step S402, normalizing the second sample data to obtain the sample prefrontal three-channel EEG data.

[0056] In the above embodiment, the three-channel EEG data of the frontal lobe is extracted from the original EEG data, and is cut into a training set and a validation set as the input data of the student network. Prior to this, the three-channel EEG data of the frontal lobe needs to be preprocessed with the same noise reduction, de-aliasing and normalization as the sample whole-brain EEG data, and the three-layer graph neural network is trained. The band-pass filter allows signals within a certain frequency range to pass while blocking signals of other frequencies. In EEG signal processing, this usually means retaining frequencies related to EEG activity (such as 1-40Hz) while filtering out low-frequency noise (such as signals less than 1Hz, which may be related to physiological rhythms) and high-frequency noise (such as signals exceeding 40Hz, which may be related to power supply interference). Select an appropriate filter design (such as Butterworth filter, Chebyshev filter, etc.) and set the cutoff frequency to ensure that only the EEG signal of interest is retained. Notch filters are used to eliminate signals of specific frequencies, which are usually power supply frequencies (such as 50Hz or 60Hz) and their harmonics. Signals of these frequencies are called common frequency noise. To design a notch filter, determine the specific frequencies that need to be eliminated, and set the bandwidth of the filter to ensure that only the noise of these specific frequencies is eliminated without affecting signals of other frequencies. By bandpass filtering and notch filtering, the noise in EEG signals can be significantly reduced, the signal-to-noise ratio (SNR) of the signal can be improved, and subsequent signal analysis can be more accurate. After filtering, EEG signals related to cognitive and neural activities are retained, providing high-quality data for further analysis. Normalization is the scaling of data to a specific range, usually 0 to 1, or -1 to 1. For EEG data, this usually involves subtracting the mean of the signal of each channel and dividing it by its standard deviation so that the data of each channel has zero mean and unit variance. Normalization makes EEG data between different channels and individuals comparable because they are all scaled to the same scale. Normalization can reduce errors in numerical calculations, especially when processing large-scale data, and can avoid numerical overflow or underflow problems.

[0057] In order to improve the ability to understand and classify whole-brain EEG data, in an optional implementation, the above step S201 includes:

[0058] Step S2011, using the first formula in the above-mentioned pre-trained teacher network to perform matrix calculation of the multi-layer graph convolutional neural network, to obtain the teacher network output matrix corresponding to the whole-brain EEG data of each of the above-mentioned samples, the above-mentioned first formula is Among them, H (l) represents the feature matrix of the above graph convolutional neural network at layer l, H (l+1) It represents the output matrix of the above graph convolutional neural network at the lth layer, and also represents the input matrix of the above graph convolutional neural network at the l+1th layer. is the connection matrix, for The degree matrix, W (l) is a learnable matrix, the teacher network output matrix is ​​the output matrix of the last layer of the graph convolutional neural network, and one sample whole-brain EEG data corresponds to one teacher network output matrix;

[0059] Step S2012, calculating the weighted cosine similarity according to the teacher network output matrix corresponding to each of the above sample whole-brain EEG data to obtain the above first similarity matrix;

[0060] Step S2013, performing convolution calculation using a layer graph convolutional neural network in the pre-trained teacher network to obtain the first output vector and the soft label.

[0061] In the above embodiment, a three-layer graph neural network is pre-trained, and the batch size is usually a power of 2, and this paper selects 16. The calculation process for sample X1 in a layer of graph neural network is as follows:

[0062] Among them, H (l) represents the feature matrix of the lth layer. When l = 0, H (l) is the input sample X1. is the connection matrix, for The degree matrix, W (l) is a learnable matrix. After calculation of the three-layer graph network, the output matrix can be obtained For batchsize, the weighted cosine similarity between samples in the last layer of the graph network is calculated internally. For the output corresponding to sample X1 and sample X2 and output The above similarity calculation is performed for all samples in the batchsize and multiplied by the learnable weights to obtain the weighted similarity matrix of the teacher network. The cosine similarity between each pair of samples is calculated, which is a method to measure the angular difference between vectors. Convolution calculation: In the pre-trained teacher network, the graph convolution layer is used to further extract features from the data. The output vectors of each sample are obtained, which contain the feature representation after the convolution operation. According to the output vectors, soft labels are generated, which represent the probability of the sample belonging to each category. Improved the understanding and classification ability of whole-brain EEG data.

[0063] In order to emphasize the intrinsic similarity relationship between input samples and improve the expression ability of the student network during the learning process, in an optional implementation, the above step S2012 includes:

[0064] Step S20121, corresponding to each of the above sample whole-brain EEG data, calculate the corresponding mean vector of the above teacher network output matrix, the above mean vector is a vector formed by the mean of each row element of the above teacher network output matrix;

[0065] Step S20122, calculating the weighted cosine similarity between the mean vectors corresponding to the whole-brain EEG data of each sample according to the second formula, the second formula is: represents the mean vector of the first sample whole-brain EEG data, represents the mean vector of the second sample whole-brain EEG data, The first sample whole-brain EEG data is one of any two of the above sample whole-brain EEG data, the second sample whole-brain EEG data is the above sample whole-brain EEG data of any two of the above sample whole-brain EEG data except the first sample whole-brain EEG data, W is a weight vector, W = [w1, w2…, w n ];

[0066] Step S20123, generating the first similarity matrix according to the weighted cosine similarities corresponding to all the above-mentioned sample whole-brain EEG data, the elements of the first similarity matrix are the weighted cosine similarities, the rows and columns of the first similarity matrix represent the above-mentioned sample whole-brain EEG data, and the first similarity matrix is ​​a symmetric matrix.

[0067] In the above embodiment, for the output matrix of the whole-brain EEG data of each sample, the mean of each row element is calculated to form a new vector. This vector is called the mean vector, which represents the feature representation of the original EEG data at the network output level, and can convert the original high-dimensional EEG data into a more abstract feature representation. These features may better capture the key information in the data. Calculating the mean vector can reduce the dimension of the data, making the subsequent similarity calculation more efficient. Using the second formula Calculating the weighted cosine similarity between the mean vectors corresponding to the whole-brain EEG data of two samples can be used to adjust the importance of different features in the similarity calculation. The numerator in the formula is the sum of the weighted products of the corresponding elements of the two vectors, and the denominator is the product of the weighted moduli of the two vectors. This calculation method can quantify the similarity between two vectors. Through weighted cosine similarity, we can quantify the similarity between two samples, which is very useful for subsequent tasks such as classification and clustering. For all sample whole-brain EEG data, a similarity matrix is ​​constructed based on the calculated weighted cosine similarity. The rows and columns of this matrix correspond to different samples, and the elements in the matrix are the similarity values ​​between the samples. Due to the symmetry of cosine similarity, this matrix is ​​symmetric. It should be noted that the method for generating the second similarity matrix for the student network is the same as the method for generating the first similarity matrix for the pre-trained teacher network, which will not be repeated here.

[0068] In order to reduce the difference between the student network and the teacher network and improve the generalization ability of the student network, in an optional implementation, before the above step S204, the method further includes:

[0069] Step S501, based on the mean square error loss function, the cross entropy loss function and the KL divergence loss function, the overall loss function is determined. The overall loss function is expressed as: y is the above student label, Loss CE () represents the cross entropy loss function, KL() is the above KL divergence loss function, Loss SK is the above mean square error loss function, z T is the first output vector mentioned above, z S For the second output vector, T and α are the temperature coefficient and the balancing weight respectively.

[0070] In the above embodiment, the similarity matrix of the teacher network and the student network is simultaneously input into the root mean square loss function Loss SK Perform similarity loss calculation. Minimize the mean square error loss function to narrow the gap between the student network and the teacher network. Where MSE is the mean square error loss function, (l, l ′ ) is the teacher network layer l corresponding to the student network layer l ′ layer, and is the similarity matrix of the teacher network at layer l and the student network at layer l ′ The weighted cosine similarity matrix of the layer is calculated in the last layer of the pre-trained teacher network and the student network. The mean square error loss is combined with the classic Hinton knowledge distillation loss to obtain the overall loss function for training the student network:

[0071] Among them, y is the student label, Loss CE is the cross entropy loss function, KL is the KL divergence loss, z T and z S are the logits vectors output by the teacher network and the student network, respectively. T and α are the temperature coefficient and balance weight, respectively, which are selected as 1.5 and 0.95. The composite loss function can help the model achieve a balance in multiple aspects, thereby improving the generalization ability of the model.

[0072] In order to improve work efficiency, in an optional implementation manner, after the above step S205, the method further includes:

[0073] Step S601: when the true label is 0, it is determined that the staff member is currently in a non-fatigue state.

[0074] In the above embodiment, when the above true label is 0, it means that the above staff is currently in a non-fatigue state monitored by the multi-channel EEG student network. By identifying the non-fatigue state, the system can prevent the staff from being overly fatigued, thereby reducing work accidents and improving productivity. Of course, the system can also provide feedback to the staff to let them understand their fatigue state, so that they can take appropriate rest or adjust their work rhythm.

[0075] It should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and that, although a logical order is shown in the flowcharts, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0076] The embodiment of the present application also provides a sample similarity knowledge distillation device based on multi-channel EEG. It should be noted that the sample similarity knowledge distillation device based on multi-channel EEG in the embodiment of the present application can be used to execute the sample similarity knowledge distillation method based on multi-channel EEG provided in the embodiment of the present application. The device is used to implement the above-mentioned embodiments and preferred implementation modes, and the descriptions that have been made will not be repeated. As used below, the term "module" can be a combination of software and / or hardware that implements a predetermined function. Although the device described in the following embodiments is preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceived.

[0077] The following is an introduction to the multi-channel EEG-based sample similarity knowledge distillation device provided in an embodiment of the present application.

[0078] Figure 4is a structural block diagram of a multi-channel EEG sample similarity knowledge distillation device according to an embodiment of the present application. Figure 4 As shown, the device comprises:

[0079] The first calculation unit 10 is used to execute the first calculation step, input the sample whole-brain EEG data into the pre-trained teacher network, and use the above-mentioned pre-trained teacher network to analyze and process the above-mentioned sample whole-brain EEG data to obtain a first similarity matrix, a first output vector and a soft label. The above-mentioned pre-trained teacher network is a convergent network model obtained by inputting multiple groups of sample training data into a graph convolutional neural network for training. Each group of the above-mentioned sample training data includes sample whole-brain EEG data and sample EEG labels. The above-mentioned first similarity matrix is ​​obtained by the above-mentioned pre-trained teacher network performing weighted cosine similarity calculation on the last layer of the graph convolutional neural network. The above-mentioned first output vector is the original output vector output by the above-mentioned pre-trained teacher network without being processed by the activation function. The above-mentioned soft label is the label value obtained after the above-mentioned first output vector is processed by the above-mentioned activation function. The above-mentioned label value is used to indicate whether the staff is in a fatigue state.

[0080] Specifically, the weight parameters of the teacher network are optimized in the training set training, and the overfitting is evaluated by comparing the model accuracy of the training and verification processes to obtain a high-precision and low-overfitting pre-trained teacher network. The sample whole-brain EEG data is input into the pre-trained teacher network. The sample whole-brain EEG data is analyzed and processed using the pre-trained teacher network. The pre-trained teacher network obtains a first similarity matrix, a first output vector, and a soft label. The pre-trained teacher network is trained by a graph convolutional neural network (GCN), which can process multiple sets of sample training data, including whole-brain EEG data and corresponding EEG labels. The first similarity matrix reflects the similarity between the sample EEG data, which is helpful for subsequent analysis. The first output vector is the original output of the network, and the soft label is the label value processed by the activation function, which is used to indicate whether the staff is in a state of fatigue.

[0081] The second calculation unit 20 is used to execute the second calculation step, inputting the first input data in multiple groups of current first training data into the student network in sequence, so as to use the above-mentioned student network to calculate the similarity matrix of the above-mentioned current first training data, and obtain the second similarity matrix, the second output vector and the student label, the above-mentioned current first training data is any group of first training data in the first training data set, the above-mentioned first training data set includes multiple groups of the above-mentioned first training data, each group of the above-mentioned first training data includes the above-mentioned first input data and the first output data, the above-mentioned first input data is the sample prefrontal three-channel EEG data, the above-mentioned first output data is the sample EEG label corresponding to the above-mentioned sample prefrontal three-channel EEG data, the above-mentioned second output vector is the original output vector output by the above-mentioned pre-trained teacher network without being processed by the activation function, the above-mentioned student label is the label value obtained after the above-mentioned second output vector is processed by the above-mentioned activation function, and the above-mentioned sample prefrontal three-channel EEG data is the data extracted from the above-mentioned sample whole-brain EEG data.

[0082] Specifically, by designing a distillation network structure based on sample similarity, it is ensured that the student network can achieve excellent performance when using fewer EEG channels, solving the problem of insufficient expression ability of the student network in traditional methods. Therefore, the first input data (sample prefrontal three-channel EEG data) in the current first training data is input into the student network. The student network is used to calculate the similarity matrix of the current first training data. The second similarity matrix, the second output vector and the student label are obtained from the student network. The student network learns how to identify fatigue status by processing the sample prefrontal three-channel EEG data. The second similarity matrix and the second output vector are the outputs of the student network for subsequent verification and optimization. A knowledge distillation strategy based on sample similarity is introduced to strengthen the training process of the student network by constructing a sample similarity matrix, so that the student network can better capture the intrinsic structure and similarity relationship between input samples, thereby improving its expression ability.

[0083] The verification unit 30 is used to perform a verification step, and compare and verify all the above-mentioned student labels with the corresponding above-mentioned soft labels to obtain the accuracy of the above-mentioned student network.

[0084] Specifically, the student labels output by the student network are compared with the soft labels of the corresponding pre-trained teacher network to obtain the accuracy of the student network. This is used to evaluate the performance of the student network and ensure that it can accurately identify fatigue states. Figure 3As shown in the figure, the EEG signals are processed in two ways: high-density EEG (whole brain) EEG data and low-density EEG data. The high-density EEG EEG data is input to the teacher network, and the low-density EEG EEG data (frontal three-channel EEG data) is input to the student network. It is hoped that the student network can achieve the same performance as the teacher network by combining low-density EEG data with the distillation learning. This will achieve lightweight use in reality. The final effect is: using low-density EEG to achieve the classification of fatigue and non-fatigue.

[0085] The iterative optimization unit 40 is used to input the first similarity matrix, the second similarity matrix, the first output vector and the second output vector into the overall loss function of the student network for optimization when the accuracy does not meet the accuracy requirement, and repeat the first calculation step, the second calculation step and the verification step at least once in sequence until the accuracy meets the accuracy requirement, thereby obtaining a lightweight student network. The overall loss function is a loss function composed of a mean square error loss function, a cross entropy loss function and a KL divergence loss function.

[0086] Specifically, if the accuracy of the student network does not meet the requirements, the first similarity matrix, the second similarity matrix, the first output vector and the second output vector are input into the overall loss function of the student network for optimization. Repeat the first calculation step, the second calculation step and the verification step at least once until the accuracy meets the requirements. Improve the performance of the student network by optimizing the loss function (including the mean square error loss function, the cross entropy loss function and the KL divergence loss function). Repeat the training and verification process until a satisfactory accuracy is achieved to ensure that the student network can accurately identify the fatigue state.

[0087] The input unit 50 is used to obtain the three-channel EEG data of the prefrontal lobe of the above-mentioned staff in real time, and input the three-channel EEG data of the prefrontal lobe into the above-mentioned lightweight student network to obtain a true label, and the above-mentioned true label is 1 or 0.

[0088] Specifically, the three-channel EEG data of the frontal lobe of the staff is obtained in real time. These data are input into the lightweight student network. The true label (1 or 0) is obtained. The EEG data of the staff is monitored in real time to identify the fatigue state in time. The lightweight student network can quickly process the data and give the results.

[0089] The prompt unit 60 is used to issue a prompt message when the true label is 1, and the prompt message is used to prompt the staff member that he is currently in the fatigue state.

[0090] Specifically, if the true label is 1 (indicating that the worker is in a fatigued state), a prompt message is issued. The prompt message can remind the worker to take a break and avoid accidents or reduced efficiency due to fatigue. This real-time feedback helps improve the work efficiency and safety of the worker.

[0091] In this embodiment, high-density (whole brain) EEG data is input into the teacher network, and low-density (three channels of the prefrontal lobe) EEG data is input into the student network. The knowledge of the teacher network is used to guide the training of the student network, so that the student network can achieve performance close to that of the teacher network while relying only on low-density EEG data. Finally, the student network can achieve the same performance as the teacher network by using low-density EEG data combined with knowledge distillation learning. Thereby achieving lightweight use in reality. The final effect is to use low-density EEG data to realize the classification of fatigue and non-fatigue. The present application solves the problem that the traditional EEG data knowledge distillation in the prior art ignores the intrinsic similarity relationship between input samples, resulting in insufficient expression ability of the student network during the learning process.

[0092] In order to improve the data quality of the pre-trained teacher network, in an optional implementation manner, the device further includes:

[0093] A first filtering processing unit is used to filter the original sample whole-brain EEG data by using a bandpass filtering method to filter out low-frequency noise and high-frequency noise, and to eliminate common frequency noise by using a notch filter before inputting the sample whole-brain EEG data into a pre-trained teacher network to analyze and process the sample whole-brain EEG data by using the pre-trained teacher network to obtain a first similarity matrix, a first output vector and a soft label, thereby obtaining first sample data;

[0094] The first normalization processing unit is used to perform normalization processing on the first sample data to obtain the sample whole-brain EEG data.

[0095] In the above embodiment, the sample whole-brain EEG data is cut into a training set and a validation set as the network input data of the teacher network. Prior to this, the original sample whole-brain EEG data needs to be pre-processed for noise reduction, de-aliasing and normalization. Specifically, low-frequency and high-frequency noise are filtered out by a 0.1-50Hz bandpass filter, and common frequency noise is eliminated by a 50Hz notch filter. In order to eliminate the amplitude difference caused by different EEG channels, the EEG signal is adjusted to a data distribution with a mean of 0 and a standard deviation of 1 by Z-score normalization. A suitable bandpass filter is selected, usually a digital filter, such as a Butterworth filter, a Chebyshev filter or an elliptical filter. The cutoff frequency of the filter is set to filter out low-frequency noise (such as frequency components less than 0.5Hz, which are usually related to power supply frequency interference) and high-frequency noise (such as frequency components higher than 40Hz, which may be related to myoelectric interference). The filter is applied to the original EEG data to retain the main frequency range of brain activity (usually between 1Hz and 40Hz). Identify common frequency noise, such as power frequency (50Hz or 60Hz) and its harmonics, design notch filters to specifically filter out these frequencies, and apply the notch filter to the bandpass filtered data to eliminate the common frequency noise. By filtering out low-frequency and high-frequency noise, the quality of the EEG signal can be significantly improved and interference during the analysis process can be reduced. The notch filter specifically targets common frequency noise and can effectively eliminate these periodic interferences and improve the signal-to-noise ratio of the signal. By retaining the frequency range of 1Hz to 40Hz, it can ensure that useful information related to cognitive and emotional states in the EEG signal is retained. Normalization can eliminate the impact of different dimensions and magnitudes on data comparison, so that data from different channels or different individuals can be directly compared. Many machine learning algorithms are sensitive to the scale of the data, and normalization can improve the performance and convergence speed of these algorithms.

[0096] In order to improve the training data quality of the student network, in an optional implementation manner, the device further includes:

[0097] A second filtering processing unit, before sequentially inputting the first input data in the multiple groups of current first training data into the student network to calculate the similarity matrix of the current first training data using the student network to obtain a second similarity matrix, a second output vector and a student label, uses a bandpass filtering method to filter the original second training data to filter out low-frequency noise and high-frequency noise, and uses a notch filter to eliminate common frequency noise to obtain second sample data;

[0098] The second normalization processing unit is used to perform normalization processing on the second sample data to obtain the sample prefrontal lobe three-channel EEG data.

[0099] In the above embodiment, the three-channel EEG data of the frontal lobe is extracted from the original EEG data, and is cut into a training set and a validation set as the input data of the student network. Prior to this, the three-channel EEG data of the frontal lobe needs to be preprocessed with the same noise reduction, de-aliasing and normalization as the sample whole-brain EEG data, and the three-layer graph neural network is trained. The band-pass filter allows signals within a certain frequency range to pass while blocking signals of other frequencies. In EEG signal processing, this usually means retaining frequencies related to EEG activity (such as 1-40Hz) while filtering out low-frequency noise (such as signals less than 1Hz, which may be related to physiological rhythms) and high-frequency noise (such as signals exceeding 40Hz, which may be related to power supply interference). Select an appropriate filter design (such as Butterworth filter, Chebyshev filter, etc.) and set the cutoff frequency to ensure that only the EEG signal of interest is retained. Notch filters are used to eliminate signals of specific frequencies, which are usually power supply frequencies (such as 50Hz or 60Hz) and their harmonics. Signals of these frequencies are called common frequency noise. To design a notch filter, determine the specific frequencies that need to be eliminated, and set the bandwidth of the filter to ensure that only the noise of these specific frequencies is eliminated without affecting signals of other frequencies. By bandpass filtering and notch filtering, the noise in EEG signals can be significantly reduced, the signal-to-noise ratio (SNR) of the signal can be improved, and subsequent signal analysis can be more accurate. After filtering, EEG signals related to cognitive and neural activities are retained, providing high-quality data for further analysis. Normalization is the scaling of data to a specific range, usually 0 to 1, or -1 to 1. For EEG data, this usually involves subtracting the mean of the signal of each channel and dividing it by its standard deviation so that the data of each channel has zero mean and unit variance. Normalization makes EEG data between different channels and individuals comparable because they are all scaled to the same scale. Normalization can reduce errors in numerical calculations, especially when processing large-scale data, and can avoid numerical overflow or underflow problems.

[0100] In order to improve the ability to understand and classify whole-brain EEG data, in an optional implementation, the first computing unit includes:

[0101] The first calculation module is used to perform matrix calculation of the multi-layer graph convolutional neural network using the first formula in the above-mentioned pre-trained teacher network to obtain the teacher network output matrix corresponding to the whole-brain EEG data of each of the above-mentioned samples. The above-mentioned first formula is Among them, H (l) represents the feature matrix of the above graph convolutional neural network at layer l, H (l +1)It represents the output matrix of the above graph convolutional neural network at the lth layer, and also represents the input matrix of the above graph convolutional neural network at the l+1th layer. is the connection matrix, for The degree matrix, W (l) is a learnable matrix, the teacher network output matrix is ​​the output matrix of the last layer of the graph convolutional neural network, and one sample whole-brain EEG data corresponds to one teacher network output matrix;

[0102] A second calculation module is used to calculate the weighted cosine similarity according to the teacher network output matrix corresponding to each of the above sample whole-brain EEG data to obtain the above first similarity matrix;

[0103] The third computing module is used to perform convolution calculation using a layer graph convolutional neural network in the pre-trained teacher network to obtain the first output vector and the soft label.

[0104] In the above embodiment, a three-layer graph neural network is pre-trained, and the batch size is usually a power of 2, and this paper selects 16. The calculation process for sample X1 in a layer of graph neural network is as follows:

[0105] Among them, H (l) represents the feature matrix of the lth layer. When l = 0, H (l) is the input sample X1. is the connection matrix, for The degree matrix, W (l) is a learnable matrix. After calculation of the three-layer graph network, the output matrix can be obtained For batchsize, the weighted cosine similarity between samples in the last layer of the graph network is calculated internally. For the output corresponding to sample X1 and sample X2 and output The above similarity calculation is performed for all samples in the batchsize and multiplied by the learnable weights to obtain the weighted similarity matrix of the teacher network. The cosine similarity between each pair of samples is calculated, which is a method to measure the angular difference between vectors. Convolution calculation: In the pre-trained teacher network, the graph convolution layer is used to further extract features from the data. The output vectors of each sample are obtained, which contain the feature representation after the convolution operation. According to the output vectors, soft labels are generated, which represent the probability of the sample belonging to each category. Improved the understanding and classification ability of whole-brain EEG data.

[0106] In order to attach importance to the intrinsic similarity relationship between input samples and improve the expression ability of the student network in the learning process, in an optional implementation, the second calculation module includes:

[0107] A first calculation submodule calculates a mean vector of the teacher network output matrix corresponding to each of the whole-brain EEG data of the sample, wherein the mean vector is a vector formed by the mean of each row element of the teacher network output matrix;

[0108] The second calculation submodule is used to calculate the weighted cosine similarity between the above mean vectors corresponding to the whole brain EEG data of each sample according to the second formula. The above second formula is: represents the mean vector of the first sample whole-brain EEG data, represents the mean vector of the second sample whole-brain EEG data, The first sample whole-brain EEG data is one of any two of the above sample whole-brain EEG data, the second sample whole-brain EEG data is the above sample whole-brain EEG data of any two of the above sample whole-brain EEG data except the first sample whole-brain EEG data, W is a weight vector, W = [w1, w2…, w n ];

[0109] A generating submodule is used to generate the first similarity matrix according to the weighted cosine similarities corresponding to all the above-mentioned sample whole-brain EEG data, wherein the elements of the first similarity matrix are the weighted cosine similarities, the rows and columns of the first similarity matrix represent the above-mentioned sample whole-brain EEG data, and the first similarity matrix is ​​a symmetric matrix.

[0110] In the above embodiment, for the output matrix of the whole-brain EEG data of each sample, the mean of each row element is calculated to form a new vector. This vector is called the mean vector, which represents the feature representation of the original EEG data at the network output level, and can convert the original high-dimensional EEG data into a more abstract feature representation. These features may better capture the key information in the data. Calculating the mean vector can reduce the dimension of the data, making the subsequent similarity calculation more efficient. Using the second formula Calculating the weighted cosine similarity between the mean vectors corresponding to the whole-brain EEG data of two samples can be used to adjust the importance of different features in the similarity calculation. The numerator in the formula is the sum of the weighted products of the corresponding elements of the two vectors, and the denominator is the product of the weighted moduli of the two vectors. This calculation method can quantify the similarity between two vectors. Through weighted cosine similarity, we can quantify the similarity between two samples, which is very useful for subsequent tasks such as classification and clustering. For all sample whole-brain EEG data, a similarity matrix is ​​constructed based on the calculated weighted cosine similarity. The rows and columns of this matrix correspond to different samples, and the elements in the matrix are the similarity values ​​between the samples. Due to the symmetry of cosine similarity, this matrix is ​​symmetric. It should be noted that the method for generating the second similarity matrix for the student network is the same as the method for generating the first similarity matrix for the pre-trained teacher network, which will not be repeated here.

[0111] In order to reduce the difference between the student network and the teacher network and improve the generalization ability of the student network, in an optional implementation, the device further includes:

[0112] A determination unit is used to determine the overall loss function based on the mean square error loss function, the cross entropy loss function and the KL divergence loss function before inputting the first similarity matrix, the second similarity matrix, the first output vector and the second output vector into the overall loss function of the student network for optimization, wherein the overall loss function is expressed as For the above student labels, Loss CE () represents the cross entropy loss function, KL() is the above KL divergence loss function, Loss SK is the above mean square error loss function, z T is the first output vector mentioned above, z S For the second output vector, T and α are the temperature coefficient and the balancing weight respectively.

[0113] In the above embodiment, the similarity matrix of the teacher network and the student network is simultaneously input into the root mean square loss function Loss SK Perform similarity loss calculation. Minimize the mean square error loss function to narrow the gap between the student network and the teacher network. Where MSE is the mean square error loss function, (l, l ′ ) is the teacher network layer l corresponding to the student network layer l ′ layer, and is the similarity matrix of the teacher network at layer l and the student network at layer l ′The weighted cosine similarity matrix of the layer is calculated in the last layer of the pre-trained teacher network and the student network. The mean square error loss is combined with the classic Hinton knowledge distillation loss to obtain the overall loss function for training the student network: Among them, y is the student label, Loss CE is the cross entropy loss function, KL is the KL divergence loss, z T and z S are the logits vectors output by the teacher network and the student network, respectively. T and α are the temperature coefficient and balance weight, respectively, which are selected as 1.5 and 0.95. The composite loss function can help the model achieve a balance in multiple aspects, thereby improving the generalization ability of the model.

[0114] In order to improve work efficiency, in an optional implementation manner, the device further includes:

[0115] The determination unit is used to obtain the three-channel EEG data of the prefrontal lobe of the above-mentioned staff in real time, and input the three-channel EEG data of the prefrontal lobe into the above-mentioned lightweight student network, and after obtaining the true label, when the above-mentioned true label is 0, determine that the above-mentioned staff is currently in a non-fatigue state.

[0116] In the above embodiment, when the above true label is 0, it means that the above staff is currently in a non-fatigue state monitored by the multi-channel EEG student network. By identifying the non-fatigue state, the system can prevent the staff from being overly fatigued, thereby reducing work accidents and improving productivity. Of course, the system can also provide feedback to the staff to let them understand their fatigue state, so that they can take appropriate rest or adjust their work rhythm.

[0117] The above-mentioned multi-channel EEG-based sample similarity knowledge distillation device includes a processor and a memory. The above-mentioned units are stored in the memory as program units, and the processor executes the above-mentioned program units stored in the memory to realize the corresponding functions. The above-mentioned modules are all located in the same processor; or, the above-mentioned modules are located in different processors in any combination.

[0118] The processor includes a kernel, which calls the corresponding program unit from the memory. One or more kernels can be set, and the kernel parameters can be adjusted to solve the problem that the traditional EEG data knowledge distillation in the prior art ignores the inherent similarity relationship between input samples, resulting in insufficient expression ability of the student network during the learning process.

[0119] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM, and the memory includes at least one memory chip.

[0120] An embodiment of the present invention provides a computer-readable storage medium, which includes a stored program, wherein when the program is running, the device where the computer-readable storage medium is located is controlled to execute the sample similarity knowledge distillation method based on multi-channel EEG.

[0121] Specifically, the sample similarity knowledge distillation method based on multi-channel EEG includes:

[0122] Step S201, the first calculation step, inputting the sample whole-brain EEG data into the pre-trained teacher network, so as to use the pre-trained teacher network to analyze and process the sample whole-brain EEG data to obtain a first similarity matrix, a first output vector and a soft label, the pre-trained teacher network is a convergent network model obtained by inputting multiple groups of sample training data into a graph convolutional neural network for training, each group of the sample training data includes sample whole-brain EEG data, and sample EEG labels, the first similarity matrix is ​​obtained by the pre-trained teacher network performing weighted cosine similarity calculation on the last layer of the graph convolutional neural network, the first output vector is the original output vector output by the pre-trained teacher network without being processed by the activation function, the soft label is the label value obtained after the first output vector is processed by the activation function, and the label value is used to indicate whether the staff is in a fatigue state;

[0123] Step S202, a second calculation step, sequentially inputting the first input data in multiple groups of current first training data into the student network, so as to use the above-mentioned student network to calculate the similarity matrix of the above-mentioned current first training data, and obtain a second similarity matrix, a second output vector and a student label, wherein the above-mentioned current first training data is any group of first training data in the first training data set, and the above-mentioned first training data set includes multiple groups of the above-mentioned first training data, and each group of the above-mentioned first training data includes the above-mentioned first input data and the first output data, and the above-mentioned first input data is the sample prefrontal three-channel EEG data, and the above-mentioned first output data is the sample EEG label corresponding to the above-mentioned sample prefrontal three-channel EEG data, and the above-mentioned second output vector is the original output vector output by the above-mentioned pre-trained teacher network without being processed by the activation function, and the above-mentioned student label is the label value obtained after the above-mentioned second output vector is processed by the above-mentioned activation function, and the above-mentioned sample prefrontal three-channel EEG data is the data extracted from the above-mentioned sample whole-brain EEG data;

[0124] Step S203, a verification step, comparing and verifying all the above student labels with the corresponding above soft labels to obtain the accuracy of the above student network;

[0125] Step S204, when the accuracy does not meet the accuracy requirement, the first similarity matrix, the second similarity matrix, the first output vector and the second output vector are input into the overall loss function of the student network for optimization, and the first calculation step, the second calculation step and the verification step are sequentially repeated at least once until the accuracy meets the accuracy requirement, and a lightweight student network is obtained, and the overall loss function is a loss function composed of a mean square error loss function, a cross entropy loss function and a KL divergence loss function;

[0126] Step S205, obtaining the three-channel EEG data of the prefrontal lobe of the staff member in real time, and inputting the three-channel EEG data of the prefrontal lobe into the lightweight student network to obtain a true label, where the true label is 1 or 0;

[0127] Step S206, when the true label is 1, a prompt message is issued, and the prompt message is used to prompt the staff member that he is currently in the fatigue state.

[0128] An embodiment of the present invention provides a processor, which is used to run a program, wherein the program executes the above-mentioned sample similarity knowledge distillation method based on multi-channel EEG when running.

[0129] The embodiment of the present invention provides a knowledge distillation system based on the similarity of multi-channel EEG samples. The knowledge distillation system based on the similarity of multi-channel EEG samples includes a processor, a memory, and a program stored in the memory and executable on the processor. When the processor executes the program, at least the following steps are implemented:

[0130] Step S201, the first calculation step, inputting the sample whole-brain EEG data into the pre-trained teacher network, so as to use the pre-trained teacher network to analyze and process the sample whole-brain EEG data to obtain a first similarity matrix, a first output vector and a soft label, the pre-trained teacher network is a convergent network model obtained by inputting multiple groups of sample training data into a graph convolutional neural network for training, each group of the sample training data includes sample whole-brain EEG data, and sample EEG labels, the first similarity matrix is ​​obtained by the pre-trained teacher network performing weighted cosine similarity calculation on the last layer of the graph convolutional neural network, the first output vector is the original output vector output by the pre-trained teacher network without being processed by the activation function, the soft label is the label value obtained after the first output vector is processed by the activation function, and the label value is used to indicate whether the staff is in a fatigue state;

[0131] Step S202, a second calculation step, sequentially inputting the first input data in multiple groups of current first training data into the student network, so as to use the above-mentioned student network to calculate the similarity matrix of the above-mentioned current first training data, and obtain a second similarity matrix, a second output vector and a student label, wherein the above-mentioned current first training data is any group of first training data in the first training data set, and the above-mentioned first training data set includes multiple groups of the above-mentioned first training data, and each group of the above-mentioned first training data includes the above-mentioned first input data and the first output data, and the above-mentioned first input data is the sample prefrontal three-channel EEG data, and the above-mentioned first output data is the sample EEG label corresponding to the above-mentioned sample prefrontal three-channel EEG data, and the above-mentioned second output vector is the original output vector output by the above-mentioned pre-trained teacher network without being processed by the activation function, and the above-mentioned student label is the label value obtained after the above-mentioned second output vector is processed by the above-mentioned activation function, and the above-mentioned sample prefrontal three-channel EEG data is the data extracted from the above-mentioned sample whole-brain EEG data;

[0132] Step S203, a verification step, comparing and verifying all the above student labels with the corresponding above soft labels to obtain the accuracy of the above student network;

[0133] Step S204, when the accuracy does not meet the accuracy requirement, the first similarity matrix, the second similarity matrix, the first output vector and the second output vector are input into the overall loss function of the student network for optimization, and the first calculation step, the second calculation step and the verification step are sequentially repeated at least once until the accuracy meets the accuracy requirement, and a lightweight student network is obtained, and the overall loss function is a loss function composed of a mean square error loss function, a cross entropy loss function and a KL divergence loss function;

[0134] Step S205, obtaining the three-channel EEG data of the prefrontal lobe of the staff member in real time, and inputting the three-channel EEG data of the prefrontal lobe into the lightweight student network to obtain a true label, where the true label is 1 or 0;

[0135] Step S206, when the true label is 1, a prompt message is issued, and the prompt message is used to prompt the staff member that he is currently in the fatigue state.

[0136] The present application also provides a computer program product, which, when executed on a data processing device, is suitable for executing a program for initializing at least the following method steps:

[0137] Step S201, the first calculation step, inputting the sample whole-brain EEG data into the pre-trained teacher network, so as to use the pre-trained teacher network to analyze and process the sample whole-brain EEG data to obtain a first similarity matrix, a first output vector and a soft label, the pre-trained teacher network is a convergent network model obtained by inputting multiple groups of sample training data into a graph convolutional neural network for training, each group of the sample training data includes sample whole-brain EEG data, and sample EEG labels, the first similarity matrix is ​​obtained by the pre-trained teacher network performing weighted cosine similarity calculation on the last layer of the graph convolutional neural network, the first output vector is the original output vector output by the pre-trained teacher network without being processed by the activation function, the soft label is the label value obtained after the first output vector is processed by the activation function, and the label value is used to indicate whether the staff is in a fatigue state;

[0138] Step S202, a second calculation step, sequentially inputting the first input data in multiple groups of current first training data into the student network, so as to use the above-mentioned student network to calculate the similarity matrix of the above-mentioned current first training data, and obtain a second similarity matrix, a second output vector and a student label, wherein the above-mentioned current first training data is any group of first training data in the first training data set, and the above-mentioned first training data set includes multiple groups of the above-mentioned first training data, and each group of the above-mentioned first training data includes the above-mentioned first input data and the first output data, and the above-mentioned first input data is the sample prefrontal three-channel EEG data, and the above-mentioned first output data is the sample EEG label corresponding to the above-mentioned sample prefrontal three-channel EEG data, and the above-mentioned second output vector is the original output vector output by the above-mentioned pre-trained teacher network without being processed by the activation function, and the above-mentioned student label is the label value obtained after the above-mentioned second output vector is processed by the above-mentioned activation function, and the above-mentioned sample prefrontal three-channel EEG data is the data extracted from the above-mentioned sample whole-brain EEG data;

[0139] Step S203, a verification step, comparing and verifying all the above student labels with the corresponding above soft labels to obtain the accuracy of the above student network;

[0140] Step S204, when the accuracy does not meet the accuracy requirement, the first similarity matrix, the second similarity matrix, the first output vector and the second output vector are input into the overall loss function of the student network for optimization, and the first calculation step, the second calculation step and the verification step are sequentially repeated at least once until the accuracy meets the accuracy requirement, and a lightweight student network is obtained, and the overall loss function is a loss function composed of a mean square error loss function, a cross entropy loss function and a KL divergence loss function;

[0141] Step S205, obtaining the three-channel EEG data of the prefrontal lobe of the staff member in real time, and inputting the three-channel EEG data of the prefrontal lobe into the lightweight student network to obtain a true label, where the true label is 1 or 0;

[0142] Step S206, when the true label is 1, a prompt message is issued, and the prompt message is used to prompt the staff member that he is currently in the fatigue state.

[0143] Obviously, those skilled in the art should understand that the above modules or steps of the present invention can be implemented by a general computing device, they can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices, they can be implemented by a program code executable by a computing device, so that they can be stored in a storage device and executed by the computing device, and in some cases, the steps shown or described can be executed in a different order than here, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. Thus, the present invention is not limited to any specific combination of hardware and software.

[0144] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0145] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0146] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0147] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0148] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0149] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0150] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0151] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0152] From the above description, it can be seen that the above embodiments of the present application achieve the following technical effects:

[0153] 1) The sample similarity knowledge distillation method based on multi-channel EEG of the present application, by inputting high-density (whole brain) EEG data into the teacher network, and low-density (three channels of the prefrontal lobe) EEG data into the student network, uses the knowledge of the teacher network to guide the training of the student network, so that the student network can achieve performance close to that of the teacher network when relying only on low-density EEG data. Finally, the student network can achieve the same performance as the teacher network by using low-density EEG data combined with knowledge distillation learning. Thereby achieving lightweight use in reality. The final effect is to use low-density EEG data to realize the classification of fatigue and non-fatigue. The present application solves the problem in the prior art that traditional EEG data knowledge distillation ignores the intrinsic similarity relationship between input samples, resulting in insufficient expression ability of the student network during the learning process.

[0154] 2) The sample similarity knowledge distillation device based on multi-channel EEG of the present application inputs high-density (whole brain) EEG data into the teacher network and low-density (three channels of the prefrontal lobe) EEG data into the student network, and uses the knowledge of the teacher network to guide the training of the student network, so that the student network can achieve performance close to that of the teacher network while relying only on low-density EEG data. Finally, the student network can achieve the same performance as the teacher network by using low-density EEG data combined with knowledge distillation learning. Thereby achieving lightweight use in reality. The final effect is to use low-density EEG data to realize the classification of fatigue and non-fatigue. The present application solves the problem in the prior art that traditional EEG data knowledge distillation ignores the intrinsic similarity relationship between input samples, resulting in insufficient expression ability of the student network during the learning process.

[0155] The above description is only the preferred embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A sample similarity knowledge distillation method based on multi-channel EEG, characterized in that: include: The first calculation step is to input the sample whole-brain EEG data into the pre-trained teacher network, so as to use the pre-trained teacher network to analyze and process the sample whole-brain EEG data to obtain a first similarity matrix, a first output vector and a soft label, wherein the pre-trained teacher network is a convergent network model obtained by inputting multiple groups of sample training data into a graph convolutional neural network for training, and each group of the sample training data includes sample whole-brain EEG data and sample EEG labels, and the first similarity matrix is ​​obtained by the pre-trained teacher network performing weighted cosine similarity calculation on the last layer of the graph convolutional neural network, and the first output vector is the original output vector output by the pre-trained teacher network without being processed by an activation function, and the soft label is a label value obtained after the first output vector is processed by the activation function, and the label value is used to indicate whether the staff is in a fatigue state; A second calculation step is to input the first input data in multiple groups of current first training data into the student network in sequence, so as to use the student network to analyze the first input data to obtain a second similarity matrix, a second output vector and a student label, wherein the current first training data is any group of first training data in the first training data set, the first training data set includes multiple groups of the first training data, each group of the first training data includes the first input data and the first output data, the first input data is sample prefrontal three-channel EEG data, the first output data is the sample EEG label corresponding to the sample prefrontal three-channel EEG data, the second output vector is the original output vector output by the pre-trained teacher network without being processed by the activation function, the student label is the label value obtained after the second output vector is processed by the activation function, and the sample prefrontal three-channel EEG data is data extracted from the sample whole-brain EEG data; A verification step, comparing and verifying all the student labels with the corresponding soft labels to obtain the accuracy of the student network; In the case where the accuracy does not meet the accuracy requirement, the first similarity matrix, the second similarity matrix, the first output vector and the second output vector are input into the overall loss function of the student network for optimization, and the first calculation step, the second calculation step and the verification step are sequentially repeated at least once until the accuracy meets the accuracy requirement, and a lightweight student network is obtained, and the overall loss function is a loss function composed of a mean square error loss function, a cross entropy loss function and a KL divergence loss function; Acquire the three-channel EEG data of the frontal lobe of the staff member in real time, and input the three-channel EEG data of the frontal lobe into the lightweight student network to obtain a true label, where the true label is 1 or 0; When the true label is 1, a prompt message is issued, and the prompt message is used to prompt that the staff is currently in the fatigue state.

2. The method according to claim 1, characterized in that Before inputting the sample whole-brain EEG data into the pre-trained teacher network to analyze and process the sample whole-brain EEG data using the pre-trained teacher network to obtain a first similarity matrix, a first output vector and a soft label, the method further includes: The original sample whole-brain EEG data is filtered by a bandpass filtering method to remove low-frequency noise and high-frequency noise, and a notch filter is used to eliminate common frequency noise to obtain the first sample data; The first sample data is normalized to obtain the sample whole-brain EEG data.

3. The method according to claim 1, characterized in that Before sequentially inputting first input data in the multiple sets of current first training data into the student network to analyze the first input data using the student network to obtain a second similarity matrix, a second output vector and a student label, the method further includes: The original second training data is filtered by a bandpass filtering method to remove low-frequency noise and high-frequency noise, and a notch filter is used to eliminate common frequency noise to obtain second sample data; The second sample data is normalized to obtain the sample prefrontal three-channel EEG data.

4. The method according to claim 1, characterized in that: Inputting the sample whole-brain EEG data into the pre-trained teacher network, and using the pre-trained teacher network to analyze and process the sample whole-brain EEG data to obtain a first similarity matrix, a first output vector and a soft label, including: The matrix calculation of the multi-layer graph convolutional neural network is performed using the first formula in the pre-trained teacher network to obtain the teacher network output matrix corresponding to the whole-brain EEG data of each sample. The first formula is Among them, H (l) represents the feature matrix of the graph convolutional neural network in layer l, H (l+1) represents the output matrix of the graph convolutional neural network described in the lth layer, and also represents the input matrix of the graph convolutional neural network described in the l+1th layer, is the connection matrix, for The degree matrix, W (l) is a learnable matrix, the teacher network output matrix is ​​the output matrix of the last layer of the graph convolutional neural network, and one sample whole-brain EEG data corresponds to one teacher network output matrix; Calculate the weighted cosine similarity according to the teacher network output matrix corresponding to each of the sample whole-brain EEG data to obtain the first similarity matrix; A layer graph convolutional neural network is used to perform convolution calculations in the pre-trained teacher network to obtain the first output vector and the soft label.

5. The method according to claim 1, characterized in that Calculating the weighted cosine similarity according to the teacher network output matrix corresponding to each of the sample whole-brain EEG data to obtain the first similarity matrix includes: Corresponding to each of the sample whole-brain EEG data, a mean vector of the corresponding teacher network output matrix is ​​calculated, wherein the mean vector is a vector formed by the mean of each row element of the teacher network output matrix; The weighted cosine similarity between the mean vectors corresponding to the whole brain EEG data of each sample is calculated according to the second formula. The second formula is: represents the mean vector of the first sample whole-brain EEG data, represents the mean vector of the second sample whole-brain EEG data, The first sample whole-brain EEG data is one of any two sample whole-brain EEG data, the second sample whole-brain EEG data is the sample whole-brain EEG data of any two sample whole-brain EEG data except the first sample whole-brain EEG data, W is a weight vector, W=[w1, w2…, w n ]; The first similarity matrix is ​​generated according to the weighted cosine similarities corresponding to all the sample whole-brain EEG data, the elements of the first similarity matrix are the weighted cosine similarities, the rows and columns of the first similarity matrix represent the sample whole-brain EEG data, and the first similarity matrix is ​​a symmetric matrix.

6. The method according to claim 1, characterized in that Before inputting the first similarity matrix, the second similarity matrix, the first output vector, and the second output vector into the overall loss function of the student network for optimization, the method further includes: The overall loss function is determined based on the mean square error loss function, the cross entropy loss function and the KL divergence loss function. The overall loss function is expressed as follows: y is the student label, Loss CE () represents the cross entropy loss function, KL() is the KL divergence loss function, Loss SK is the mean square error loss function, z T is the first output vector, z S For the second output vector, T and α are the temperature coefficient and the balancing weight respectively.

7. The method according to claim 1, characterized in that After acquiring the three-channel EEG data of the frontal lobe of the staff member in real time and inputting the three-channel EEG data of the frontal lobe into the lightweight student network to obtain the real label, the method further includes: When the true label is 0, it is determined that the worker is currently in a non-fatigue state.

8. A sample similarity knowledge distillation device based on multi-channel EEG, characterized in that: The device comprises: A first calculation unit is used to execute a first calculation step, inputting the sample whole-brain EEG data into a pre-trained teacher network, so as to analyze and process the sample whole-brain EEG data using the pre-trained teacher network to obtain a first similarity matrix, a first output vector and a soft label, wherein the pre-trained teacher network is a convergent network model obtained by inputting multiple groups of sample training data into a graph convolutional neural network for training, and each group of the sample training data includes sample whole-brain EEG data and a sample EEG label, the first similarity matrix is ​​obtained by the pre-trained teacher network performing weighted cosine similarity calculation on the last layer of the graph convolutional neural network, the first output vector is an original output vector output by the pre-trained teacher network without being processed by an activation function, and the soft label is a label value obtained after the first output vector is processed by the activation function, and the label value is used to indicate whether the staff is in a fatigue state; A second calculation unit is used to execute a second calculation step, inputting the first input data in multiple groups of current first training data into the student network in sequence, so as to use the student network to analyze the first input data to obtain a second similarity matrix, a second output vector and a student label, wherein the current first training data is any group of first training data in the first training data set, the first training data set includes multiple groups of the first training data, each group of the first training data includes the first input data and the first output data, the first input data is sample prefrontal three-channel EEG data, the first output data is the sample EEG label corresponding to the sample prefrontal three-channel EEG data, the second output vector is the original output vector output by the pre-trained teacher network without being processed by the activation function, the student label is the label value obtained after the second output vector is processed by the activation function, and the sample prefrontal three-channel EEG data is data extracted from the sample whole-brain EEG data; A verification unit, used to perform a verification step, comparing and verifying all the student labels with the corresponding soft labels to obtain the accuracy of the student network; An optimization iteration unit, used for inputting the first similarity matrix, the second similarity matrix, the first output vector and the second output vector into the overall loss function of the student network for optimization when the accuracy does not meet the accuracy requirement, and repeating the first calculation step, the second calculation step and the verification step at least once in sequence until the accuracy meets the accuracy requirement, thereby obtaining a lightweight student network, and the overall loss function is a loss function composed of a mean square error loss function, a cross entropy loss function and a KL divergence loss function; An input unit, used for acquiring the three-channel EEG data of the frontal lobe of the staff member in real time, and inputting the three-channel EEG data of the frontal lobe into the lightweight student network to obtain a true label, wherein the true label is 1 or 0; A prompt unit is used to issue a prompt message when the real label is 1, and the prompt message is used to prompt that the staff is currently in the fatigue state.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored program, wherein when the program is executed, the device where the computer-readable storage medium is located is controlled to execute the method according to any one of claims 1 to 7.

10. A computer program product comprising computer instructions, characterized in that When the computer instructions are executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images