An emotion recognition method based on friction nanogenerator electric signal

Abstract

The application discloses an emotion recognition method based on a friction nanometer generator electric signal, which adopts a friction nanometer generator unit for sensing the change of facial expressions, the friction nanometer generator unit has the characteristics of self-power supply and does not need an external power supply, and has the advantages of low cost, small size and easy processing of output signals. The friction nanometer generator unit is pasted on the face, and the change of facial expressions can be sensed, the change of the face is converted into an electric signal for emotion analysis, and the working efficiency in the face is not affected. The application pastes two friction nanometer generator units on the inner and outer corners of a single eye of a subject, can fully reflect the change of emotions, and proposes a network structure based on LSTM and attention mechanism, extracts features and classifies four basic emotions, and can meet common emotion analysis and emotion recognition.

Description

Technical Field

[0001] The present invention belongs to the technical field of emotion recognition, and specifically relates to an emotion recognition method based on electrical signals from a triboelectric nanogenerator (TENG). Background Art

[0002] Emotions play a crucial role in influencing human behavior and decision-making. Accurately identifying emotional states is not only fundamental to understanding human psychology but also a key driver for the development of applications such as mental health monitoring, human-computer interaction, and intelligent systems. However, traditional emotion recognition methods rely on complex biosignal acquisition equipment, such as electroencephalogram (EEG), galvanic skin response (GSR) sensors, and cameras. These methods suffer from high cost, complex equipment, and poor portability. The highly invasive signal acquisition process can also easily cause the wearer's emotional fluctuations to interfere with experimental results. Summary of the Invention

[0003] The present invention aims to address the shortcomings of existing technologies and provides an emotion recognition method based on electrical signals from triboelectric nanogenerators (FNGs). This method uses FNG units to sense changes in facial expressions and constructs an emotion recognition neural network model. Using this trained neural network model, the method can identify emotions based on facial expression data collected using the FNG units. This method is energy-efficient and highly efficient, meeting common emotion analysis tasks.

[0004] In order to achieve the above technical objectives, the present invention adopts the following technical solutions:

[0005] An emotion recognition method based on triboelectric nanogenerator electrical signals, which uses a triboelectric nanogenerator unit to sense changes in facial expressions and convert the changes in facial expressions into electrical signals;

[0006] The structure of the friction nanogenerator unit is as follows: it includes a positive charge friction layer, a negative charge friction layer, an intermediate rebound support portion, a first thin film electrode layer, a second thin film electrode layer and an outer packaging film, the positive charge friction layer and the negative charge friction layer are arranged opposite and parallel to each other, and the intermediate rebound support portion is arranged between the positive charge friction layer and the negative charge friction layer; the first thin film electrode layer is provided on the outer surface of the negative charge friction layer, and the second thin film electrode layer is provided on the outer surface of the positive charge friction layer, and the first thin film electrode layer and the second thin film electrode layer are respectively led out of the outer packaging film through wires for connecting to an external circuit;

[0007] Two triboelectric nanogenerator units were attached to the inner and outer corners of the subject's left (or right) eye, respectively, and connected to an electrical signal acquisition device to collect signal data from the subject under different given emotions. The data was preprocessed to construct a dataset.

[0008] Then, an emotion recognition neural network model is built and the model is trained and tested using the dataset;

[0009] Finally, the trained emotion recognition neural network model can be used to perform emotion recognition on the facial expression change data collected by the friction nanogenerator unit.

[0010] In the above technical solution, the positive charge friction layer adopts nylon film, the negative charge friction layer adopts PDMS film, the middle rebound support part adopts sponge, and the middle rebound support part is located at the outer edge between the positive charge friction layer and the negative charge friction layer.

[0011] In the above technical solution, both the first thin film electrode layer and the second thin film electrode layer are made of double-sided copper tape.

[0012] In the above technical solution, the outer packaging film is made of PVC film.

[0013] In the above technical solution, the effective contact area of ​​a single triboelectric nanogenerator unit is set to 1×2 cm 2 .

[0014] In the above technical solution, the electrical signal acquisition device adopts an electrometer.

[0015] In the above technical solution, the method of collecting the signal data of the subject under different given emotions and preprocessing the data to construct the data set is as follows:

[0016] (1) Emotional data collection: The subjects were in a separate room, the sampling frequency was set to 300 points per second, and the data type collected was voltage data. The subjects made facial expressions according to the given emotions, and each expression lasted for 3 seconds. Four types of emotions were collected at one time, namely happiness, sadness, anger, and surprise, and continuous voltage waveform data was obtained;

[0017] (2) Data preprocessing: Use fast Fourier transform to reduce the noise of the collected voltage waveform data, and resample to reduce the sample dimension. Set the interval sampling to sample one point every two points. Finally, divide the data and split the waveform data into separate samples. Set the sample length to 225. For samples with a length less than 225, fill them with 0. For samples with a length greater than 225, cut them to keep the data sample dimension consistent.

[0018] (3) Constructing the data set: Collect 4 types of emotions and repeat each emotion 25 times, for a total of 100 samples; use 80% of the data as the training set and the remaining 20% ​​as the test set for training and testing.

[0019] In the above technical solution, the emotion recognition neural network model is composed of a bidirectional LSTM layer, a convolutional neural network layer and an Attention layer.

[0020] In the above technical solution, during model training, relevant parameters are continuously adjusted to achieve the optimal structure. Early stopping and learning rate adjustment strategies are used. Each adjustment is made on a single parameter basis, and the loss is then observed. The hyperparameters adjusted primarily include the number of network layers, network structure, and the number of neurons in the hidden layer.

[0021] The present invention has the following advantages and beneficial effects:

[0022] This invention designs a triboelectric nanogenerator unit made of PDMS and nylon film, with sponge as a spacer and PVC film as the encapsulation. This triboelectric nanogenerator unit is self-powered, requiring no external power source, and is low-cost, compact, and its output signal is easily processed. When attached to a person's face, this unit can sense changes in facial expression and convert these changes into electrical signals for emotional analysis. Its efficiency remains unaffected even when the face is obscured or in dark environments.

[0023] The present invention adheres two friction nanogenerator units to the inner and outer corners of a subject's eyes, respectively, which can fully reflect emotional changes, and connects the two friction nanogenerator units in series, thereby increasing the output voltage.

[0024] This paper proposes a novel emotion recognition system, which uses friction nanogenerator sensors to extract different emotional signals, and proposes a network structure based on LSTM and attention mechanism to extract and classify the features of four basic emotions, which can meet the needs of common sentiment analysis and emotion recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 It is a wearing schematic diagram of the present invention.

[0026] Figure 2 Schematic diagram of the working principle of the friction nanogenerator unit.

[0027] Figure 3 This is the structural diagram of the friction nanogenerator unit.

[0028] Figure 4 Graphs of voltage and current output for triboelectric nanogenerator units with different areas.

[0029] Figure 5 The voltage waveforms corresponding to different emotions.

[0030] Figure 6 This is a flow chart of bidirectional LSTM data operation.

[0031] Figure 7 This is a diagram of the bidirectional LSTM network structure combined with the attention mechanism.

[0032] Figure 8 Accuracy diagram of different network structures.

[0033] For ordinary technicians in this field, other relevant drawings can be obtained based on the above drawings without any creative work. DETAILED DESCRIPTION

[0034] The technical solution of the present invention is further described below with reference to specific embodiments.

[0035] An emotion recognition method based on triboelectric nanogenerator electrical signals uses a triboelectric nanogenerator unit to sense changes in facial expressions and convert them into electrical signals. Figure 1 .

[0036] For details, see the attached Figure 2 and attached Figure 3The triboelectric nanogenerator unit comprises a positively charged friction layer, a negatively charged friction layer, an intermediate rebound support portion, a first thin-film electrode layer, a second thin-film electrode layer, and an outer film. The positively charged friction layer and the negatively charged friction layer are arranged opposite and parallel to each other. The intermediate rebound support portion is disposed between the positively charged friction layer and the negatively charged friction layer to provide elastic support for the positively charged friction layer and the negatively charged friction layer. In the absence of external forces, the positively charged friction layer and the negatively charged friction layer are spaced apart. Under external forces, the positively charged friction layer and the negatively charged friction layer can compress the intermediate rebound support portion, reducing the distance between the positively charged friction layer and allowing them to contact each other. When the positively charged friction layer and the negatively charged friction layer contact, electrons are transferred, causing the positively charged friction layer to become positively charged and the negatively charged friction layer to become negatively charged. In this embodiment, the positive-charge friction layer is made of nylon film, the negative-charge friction layer is made of PDMS film, and the intermediate rebound support portion is made of sponge. The intermediate rebound support portion is located at the outer edge between the positive-charge friction layer and the negative-charge friction layer, so that the intermediate regions of the positive-charge friction layer and the negative-charge friction layer can fully contact each other. The first thin-film electrode layer is provided on the outer surface of the negative-charge friction layer, and the second thin-film electrode layer is provided on the outer surface of the positive-charge friction layer. Preferably, both the first and second thin-film electrode layers are made of double-sided copper tape. The outer packaging film is made of PVC film, and the first and second thin-film electrode layers are respectively led out of the outer packaging film via wires for connection to an external circuit.

[0037] The triboelectric nanogenerator unit converts mechanical energy into electrical energy by using the triboelectric effect and electrostatic induction effect. When two surfaces of different materials rub or come into contact and separate, charge transfer occurs due to the different attraction of the materials to electrons. One material loses electrons and becomes positively charged, while the other material gains electrons and becomes negatively charged, thereby generating a current signal in the external circuit. The charge transfer process of the triboelectric nanogenerator unit is as follows: Figure 2 Shown: In Figure 2In the first stage, there is no charge transfer in the static state, and the nylon film and PDMS film induce corresponding positive and negative charges on the copper electrode respectively; in the second stage, when the sponge is squeezed, the distance between the nylon film and the PDMS film decreases, an induced potential difference is formed between the two electrodes, and the current flows from the first thin film electrode layer on the PDMS side to the second thin film electrode layer on the nylon side; in the third stage, the PDMS film and the nylon film continue to approach each other until they contact each other. At this time, the potential difference caused by the positive and negative charges offsetting the friction charge gradually disappears, the current gradually decreases to zero, and the charge on the thin film electrode layer also decreases to almost zero; then in the fourth stage, the nylon film moves away from the PDMS film, and the two charged node films begin to separate, and an induced potential difference is re-formed between the two electrodes, electrons flow back, and the current flows from the second thin film electrode layer on the nylon side to the first thin film electrode layer on the PDMS side, forming a current to balance the electrostatic field until the first stage is restored, and thus a cycle is completed.

[0038] The present invention uses the aforementioned triboelectric nanogenerator unit to sense the changing characteristics of facial expressions for emotional analysis. Changes in facial expression are closely associated with noticeable facial muscle activity, particularly around the eyes, which is most prominent between the eyebrows and at the outer corners of the eyes. Furthermore, due to the symmetry of the facial structure, the left and right eyes exhibit the same emotional changes, so the movement of the muscles around a single eye is sufficient for emotional analysis. Therefore, in one embodiment, two triboelectric nanogenerator units were attached to the inner and outer corners of the subject's left (or right) eye, respectively.

[0039] Furthermore, the present invention conducted a detailed test on the performance of the triboelectric nanogenerator unit. During the test, the performance of triboelectric nanogenerator units of different sizes was evaluated, and it was finally determined that the effective contact area of ​​a single triboelectric nanogenerator unit was 1×2cm 2 It should be noted that this size refers only to the effective friction contact area between the positive and negative friction layers of the triboelectric nanogenerator unit during operation, rather than the final overall size of the entire triboelectric nanogenerator. The following tests the effect of different triboelectric nanogenerator areas on output performance:

[0040] A single triboelectric nanogenerator unit was fixed to one end of a linear motor and made periodic contact and separation motions under the drive of the linear motor. The output voltage and output current of triboelectric nanogenerator units with different areas were tested at an operating frequency of 2 Hz. 2 , 1×2cm 2 , 2×2cm 2 3×3cm 2 The output voltage and output current under conditions such as Figure 4As shown in the figure, as the unit area of ​​the triboelectric nanogenerator increases, the output open-circuit voltage increases from 0.43V to 4.3V, and the short-circuit current increases from 7.16nA to 54.9nA. Therefore, increasing the unit area of ​​the triboelectric nanogenerator can obtain a higher voltage. However, due to the limitations of the effective signal acquisition area of ​​the face and wearing comfort, the unit area of ​​the triboelectric nanogenerator is finally selected to be 1×2cm 2 Under these conditions, the open-circuit voltage of the friction nanogenerator unit is 0.84V, the short-circuit current is 6.8nA, and the output signal can effectively represent emotional changes.

[0041] The two triboelectric nanogenerator units attached to the inner and outer canthi of the subject's left (or right) eye are connected in series and then connected to an electrical signal acquisition device to achieve signal acquisition (the series connection here means connecting the first thin film electrode layers of the two triboelectric nanogenerator units as a first output terminal, connecting the second thin film electrode layers of the two triboelectric nanogenerator units as a second output terminal, and then connecting the first output terminal and the second output terminal to the positive and negative signal input terminals of the electrical signal acquisition device, respectively). Furthermore, in this embodiment, the electrical signal acquisition device uses an electrometer.

[0042] Next, the above signal collection method is used to collect signal data of the subjects under different given emotions, and the data is preprocessed to construct a data set. The specific implementation method is as follows:

[0043] (1) Emotional data collection: The subjects were in a separate room, the sampling frequency was set to 300 points per second, and the data type collected was voltage data. The subjects made facial expressions according to the given emotions, and each expression lasted for 3 seconds. Four types of emotions were collected at one time, namely happiness, sadness, anger, and surprise, and continuous voltage waveform data was obtained (see Appendix). Figure 5 ).

[0044] (2) Preprocess the data: Use fast Fourier transform to reduce the noise of the collected voltage waveform data, and resample to reduce the sample dimension. Set the interval sampling to sample one point every two points, and finally divide the data to split the waveform data into separate samples. Set the sample length to 225, fill the samples with a length of less than 225 with 0, and cut the samples with a length of more than 225 to keep the data sample dimension consistent.

[0045] (3) Constructing a data set: In order to analyze different emotional data, this embodiment collected four types of emotions, each of which was repeated 25 times, for a total of 100 samples; 80% of the data was used as a training set, and the remaining 20% ​​was used as a test set for training and testing.

[0046] Next, we build an emotion recognition neural network model and use the above dataset to train and test the model. The specific implementation is as follows:

[0047] The main body of the neural network is bidirectional LSTM. The data operation method is shown in the attached Figure 6 When processing data, bidirectional LSTM can access data streams from two directions simultaneously, enhancing the ability to mine data associations over long time intervals and automatically extract input data features.

[0048] When training the model, the relevant parameters are continuously adjusted to optimize the structure. The emotion recognition neural network model consists of a bidirectional LSTM layer, a convolutional neural network layer, and an Attention layer. The network structure is shown in the attached figure. Figure 7 The model uses early stopping and a learning rate adjustment strategy. Each time, only one parameter is adjusted and the loss is observed. The hyperparameters adjusted primarily include the number of network layers, network structure, and the number of hidden layer neurons.

[0049] Number of network layers: The emotion recognition neural network model constructed in this embodiment is a two-layer bidirectional LSTM designed based on the LSTM basic unit. The model training rounds are determined by the early stopping method, and the learning rate uses a dynamic learning rate adjustment strategy.

[0050] Network structure: The emotion recognition neural network model constructed in this embodiment is a two-layer bidirectional LSTM combined with attention mechanism based on the LSTM basic unit design. This embodiment evaluates the two-layer bidirectional, two-layer bidirectional combined attention mechanism, three-layer bidirectional combined attention mechanism, two-layer unidirectional, two-layer unidirectional combined attention mechanism, and three-layer unidirectional combined attention mechanism. The evaluation method is to use the same data to train different networks. The evaluation results are as follows: Figure 8 As shown. Figure 8 As can be seen from the figure, the two-layer unidirectional network has the lowest accuracy, likely due to its simple structure and inability to extract effective information from the data. The two-layer bidirectional attention network used in this system has the highest accuracy. However, as the number of network layers increases, the model accuracy improves little and is prone to overfitting. Taking all factors into consideration, this system selects the most appropriate two-layer bidirectional attention network model for sentiment classification.

[0051] Dynamic learning rate adjustment strategy: At the end of each training run during model compilation, the monitoring metric is checked to see if it is better than the best metric recorded. If so, the best metric is updated and the wait counter is reset. If not, the wait counter is incremented. During neural network training, if validation set performance does not improve for five consecutive training cycles (patience = 5), the learning rate is adjusted to 50% of the current value (factor = 0.5) and the wait counter is reset. This cycle is repeated until the lower limit of the learning rate is reached.

[0052] Early stopping: Use the early stopping method to return the optimal number of training cycles for the model. For different network structures, when the accuracy of the validation set does not change within 10 training cycles, the model's classification performance has reached its optimal level, and training is stopped at this point. In this experiment, the upper limit of epochs is set to 300.

[0053] Finally, after the emotion recognition neural network model is trained, the model can be applied to the facial expression change data collected by the friction nanogenerator unit to perform emotion recognition, that is, automatically identify one of the four basic emotions: happiness, sadness, anger and surprise.

[0054] The above description is only used to illustrate the present invention and is not intended to limit the present invention to the structure and scope of use shown and described. Therefore, all corresponding modifications and equivalents made within the spirit and principles of the present invention fall within the scope of the patent applied for by the present invention.

Claims

1. A method for emotion recognition based on electrical signals from triboelectric nanogenerators, characterized by: A triboelectric nanogenerator unit is used to sense changes in facial expressions and convert them into electrical signals. The structure of the friction nanogenerator unit is as follows: it includes a positive charge friction layer, a negative charge friction layer, an intermediate rebound support portion, a first thin film electrode layer, a second thin film electrode layer and an outer packaging film, the positive charge friction layer and the negative charge friction layer are arranged opposite and parallel to each other, and the intermediate rebound support portion is arranged between the positive charge friction layer and the negative charge friction layer; the first thin film electrode layer is arranged on the outer surface of the negative charge friction layer, and the second thin film electrode layer is arranged on the outer surface of the positive charge friction layer, and the first thin film electrode layer and the second thin film electrode layer are respectively led out of the outer packaging film through wires for connecting to an external circuit; Two triboelectric nanogenerator units were attached to the inner and outer corners of the subject's eyes, respectively, and connected to an electrical signal acquisition device to collect electrical signal data under different given emotions. The electrical signal data was preprocessed to construct a data set. The method of collecting the electrical signal data of the subject under different given emotions and preprocessing the electrical signal data to construct a data set is as follows: (1) Emotional data collection: The subjects were in a separate room, the sampling frequency was set to 300 points per second, and the data type collected was voltage data. The subjects made facial expressions according to the given emotions, and each facial expression lasted for 3 seconds. Four types of emotions were collected at one time, namely happiness, sadness, anger, and surprise, and continuous voltage waveform data was obtained; (2) Data preprocessing: Use fast Fourier transform to reduce the noise of the collected voltage waveform data, and resample to reduce the sample dimension. Set the interval sampling to sample one point every two points. Finally, divide the data and split the voltage waveform data into separate samples. Set the sample length to 225. For samples with a length less than 225, fill them with 0. For samples with a length greater than 225, cut them to keep the sample dimension consistent. (3) Constructing a data set: Collect 4 types of emotions and repeat each emotion 25 times, for a total of 100 samples; use 80% of the samples as the training set and the remaining 20% ​​as the test set for training and testing; Then, an emotion recognition neural network model is constructed and the dataset is used to train and test the emotion recognition neural network model; Finally, by applying the trained emotion recognition neural network model, emotions can be recognized by converting the electrical signals converted from changes in facial expressions collected by the friction nanogenerator unit.

2. The emotion recognition method based on the electrical signal of the triboelectric nanogenerator according to claim 1 is characterized in that: The positive charge friction layer is made of nylon film, the negative charge friction layer is made of PDMS film, the middle rebound support part is made of sponge, and the middle rebound support part is located at the outer edge between the positive charge friction layer and the negative charge friction layer.

3. The emotion recognition method based on triboelectric nanogenerator electrical signals according to claim 1, characterized in that: The first thin film electrode layer and the second thin film electrode layer both use double-sided copper tape.

4. The emotion recognition method based on triboelectric nanogenerator electrical signals according to claim 1, characterized in that: The outer packaging film is a PVC film.

5. The emotion recognition method based on triboelectric nanogenerator electrical signals according to claim 1, characterized in that: The effective friction area of ​​a single triboelectric nanogenerator unit is set to 1×2 cm 2 .

6. The emotion recognition method based on triboelectric nanogenerator electrical signals according to claim 1, characterized in that: The electrical signal acquisition device adopts an electrometer.

7. The emotion recognition method based on triboelectric nanogenerator electrical signals according to claim 1, characterized in that: The emotion recognition neural network model consists of a bidirectional LSTM layer, a convolutional neural network layer and an Attention layer.

Images