Emotion recognition method based on electric signals of friction nanometer generator

Abstract

The invention discloses an emotion recognition method based on an electric signal of a friction nanometer generator, the method adopts a friction nanometer generator unit for sensing the change condition of facial expression of a human face, the friction nanometer generator unit has the characteristic of self power supply, does not need an external power supply, and is low in cost and small in size, and the output signal is easy to process. The friction nanometer generator unit is pasted to the face, the change condition of the facial expression of the face can be sensed, the change condition of the facial expression of the face is converted into an electric signal for emotion analysis, and the working efficiency is not affected when the face is shielded and in the dark environment. According to the invention, the two friction nano-generator units are respectively pasted at the inner and outer canthus of a single eye of a subject, emotion changes can be fully reflected, a network structure is provided based on LSTM and an attention mechanism, feature extraction and classification are carried out on four basic emotions, and common emotion analysis and emotion recognition can be satisfied.

Description

Technical Field

[0001] The present invention belongs to the technical field of emotion recognition, and in particular relates to an emotion recognition method based on electrical signals of a triboelectric nanogenerator (TENG). Background Art

[0002] Emotions play a vital role in influencing human behavior and decision-making. Accurately identifying emotional states is not only the basis for understanding human psychology, but also an important driving factor for the development of applications such as mental health monitoring, human-computer interaction, and intelligent systems. However, traditional emotion recognition methods rely on complex biological signal acquisition equipment, such as electroencephalogram (EEG), galvanic skin response (GSR) sensors, and cameras, which have the problems of high cost, complex equipment, and poor portability. The highly invasive signal acquisition process can also easily cause the wearer's emotional fluctuations to interfere with the experimental results. Summary of the invention

[0003] The purpose of the present invention is to solve the deficiencies of the prior art and to provide an emotion recognition method based on the electrical signal of a friction nanogenerator. The method uses a friction nanogenerator unit to sense the changes in facial expressions of a person, and constructs an emotion recognition neural network model. By using the trained emotion recognition neural network model, the facial expression change data of a person collected by the friction nanogenerator unit can be used for emotion recognition. It can meet common emotion analysis tasks and has the characteristics of energy saving and high efficiency.

[0004] In order to achieve the above technical objectives, the present invention adopts the following technical solutions:

[0005] An emotion recognition method based on triboelectric nanogenerator electrical signals, the method uses a triboelectric nanogenerator unit to sense changes in facial expressions of a person and convert the changes in facial expressions of a person into electrical signals;

[0006] The structure of the friction nanogenerator unit is as follows: it includes a positive charge friction layer, a negative charge friction layer, an intermediate rebound support part, a first thin film electrode layer, a second thin film electrode layer and an outer packaging film, the positive charge friction layer and the negative charge friction layer are arranged oppositely and parallel to each other, and the intermediate rebound support part is arranged between the positive charge friction layer and the negative charge friction layer; the first thin film electrode layer is arranged on the outer surface of the negative charge friction layer, the second thin film electrode layer is arranged on the outer surface of the positive charge friction layer, and the first thin film electrode layer and the second thin film electrode layer are respectively led out of the outer packaging film through wires for connecting to an external circuit;

[0007] Two friction nanogenerator units are respectively attached to the inner and outer corners of the subject's left eye (or right eye), connected to an electrical signal acquisition device, and the signal data of the subject under different given emotions are collected. The data are preprocessed to construct a data set.

[0008] Then, an emotion recognition neural network model is constructed and the model is trained and tested using the dataset;

[0009] Finally, the trained emotion recognition neural network model can be used to perform emotion recognition on the facial expression change data collected by the friction nanogenerator unit.

[0010] In the above technical solution, the positive charge friction layer adopts nylon film, the negative charge friction layer adopts PDMS film, the middle rebound support part adopts sponge, and the middle rebound support part is located at the outer edge between the positive charge friction layer and the negative charge friction layer.

[0011] In the above technical solution, both the first thin film electrode layer and the second thin film electrode layer adopt double-sided copper tape.

[0012] In the above technical solution, the outer packaging film is made of PVC film.

[0013] In the above technical solution, the effective contact area of ​​a single friction nanogenerator unit is set to 1×2 cm 2 .

[0014] In the above technical solution, the electrical signal acquisition device adopts an electrometer.

[0015] In the above technical solution, the method of collecting the signal data of the subject under different given emotions and preprocessing the data to construct the data set is as follows:

[0016] (1) Emotional data collection: The subjects were in a separate room, the sampling frequency was set to 300 points per second, and the data collected was voltage data. The subjects made facial expressions according to the given emotions, and each expression lasted for 3 seconds. Four types of emotions were collected at one time, namely happiness, sadness, anger, and surprise, and continuous voltage waveform data was obtained;

[0017] (2) Preprocessing the data: Use fast Fourier transform to reduce the noise of the collected voltage waveform data, and resample to reduce the sample dimension. Set the interval sampling to sample one point every two points, and finally divide the data to split the waveform data into separate samples. Set the sample length to 225, fill the samples with a length less than 225 with 0, and cut the samples with a length greater than 225 to keep the data sample dimension consistent;

[0018] (3) Constructing a data set: Collect 4 types of emotions and repeat each emotion 25 times, for a total of 100 samples; use 80% of the data as the training set and the remaining 20% ​​as the test set for training and testing.

[0019] In the above technical solution, the emotion recognition neural network model is composed of a bidirectional LSTM layer, a convolutional neural network layer and an Attention layer.

[0020] In the above technical solution, when training the model, the relevant parameters are continuously adjusted to optimize the structure, and the early stopping method and learning rate adjustment strategy are used. Each time the adjustment is made, only one parameter is adjusted, and then the loss changes are observed. The adjusted hyperparameters mainly include the number of network layers, network structure, number of neurons in the hidden layer, etc.

[0021] The present invention has the following advantages and beneficial effects:

[0022] The present invention designs a friction nanogenerator unit made of PDMS film and nylon film materials, and uses sponge as a spacer material and is encapsulated with a PVC film. The friction nanogenerator unit has the characteristics of self-power supply and does not require an external power supply, and has low cost, small size, and easy processing of output signals. The friction nanogenerator unit is attached to the face of a person, and can sense the changes in facial expressions of the person, and convert the changes in the facial expressions of the person into electrical signals for emotional analysis, and the working efficiency is not affected when the face is blocked or in a dark environment.

[0023] The present invention adheres two friction nanogenerator units to the inner and outer corners of a subject's eye, respectively, which can fully reflect emotional changes, and connects the two friction nanogenerator units in series, thereby increasing the output voltage.

[0024] The present invention proposes a novel emotion recognition system, which uses friction nanogenerator sensors to extract different emotion signals, and proposes a network structure based on LSTM and attention mechanism to extract and classify the features of four basic emotions, which can meet the needs of common sentiment analysis and emotion recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 It is a wearing schematic diagram of the present invention.

[0026] Figure 2 This is a diagram of the working principle of the friction nanogenerator unit.

[0027] Figure 3 This is the structural diagram of the friction nanogenerator unit.

[0028] Figure 4 Graphs of voltage and current output for friction nanogenerator units with different areas.

[0029] Figure 5 The voltage waveforms corresponding to different emotions.

[0030] Figure 6 This is a bidirectional LSTM data operation flow chart.

[0031] Figure 7 This is a diagram of the bidirectional LSTM network structure combined with the attention mechanism.

[0032] Figure 8 Accuracy diagram of different network structures.

[0033] For ordinary technicians in this field, other relevant drawings can be obtained based on the above drawings without any creative work. DETAILED DESCRIPTION

[0034] The technical solution of the present invention is further described below in conjunction with specific embodiments.

[0035] An emotion recognition method based on triboelectric nanogenerator electrical signals, the method uses a triboelectric nanogenerator unit to sense changes in facial expressions and convert the changes in facial expressions into electrical signals. Figure 1 .

[0036] For details, see the attached Figure 2 and attached Figure 3The structure of the friction nanogenerator unit is as follows: it includes a positive charge friction layer, a negative charge friction layer, an intermediate rebound support part, a first thin film electrode layer, a second thin film electrode layer and an encapsulation outer film. The positive charge friction layer and the negative charge friction layer are arranged oppositely and parallel to each other, and the intermediate rebound support part is arranged between the positive charge friction layer and the negative charge friction layer, and is used to provide elastic support for the positive charge friction layer and the negative charge friction layer, so that the positive charge friction layer and the negative charge friction layer are in an interval state when not affected by external force, and when affected by external force, the positive charge friction layer and the negative charge friction layer can compress the intermediate rebound support part, so that the interval between the positive charge friction layer and the negative charge friction layer is reduced and can contact each other; when the positive charge friction layer and the negative charge friction layer contact, electrons are transferred, the positive charge friction layer will be positively charged, and the negative charge friction layer will be negatively charged. In this embodiment, the positive charge friction layer is made of nylon film, the negative charge friction layer is made of PDMS film, the middle rebound support part is made of sponge, and the middle rebound support part is located at the outer edge between the positive charge friction layer and the negative charge friction layer, so that the middle area of ​​the positive charge friction layer and the negative charge friction layer can fully contact. The first thin film electrode layer is provided with the outer surface of the negative charge friction layer, and the second thin film electrode layer is provided with the outer surface of the positive charge friction layer. Preferably, the first thin film electrode layer and the second thin film electrode layer are both made of double-sided copper tape; the packaging outer film is made of PVC film, and the first thin film electrode layer and the second thin film electrode layer are respectively led out of the packaging outer film through wires for connecting to an external circuit.

[0037] The friction nanogenerator unit converts mechanical energy into electrical energy by using the frictional electrification effect and the electrostatic induction effect. When two surfaces of different materials are rubbed or contacted and separated, charge transfer occurs due to the different attraction of the materials to electrons. One material loses electrons and becomes positively charged, while the other material gains electrons and becomes negatively charged, thereby generating a current signal in the external circuit. The charge transfer process of the friction nanogenerator unit is shown in the figure below. Figure 2 Shown: In Figure 2In the first stage, there is no charge transfer in the static state, and the nylon film and the PDMS film induce corresponding positive and negative charges on the copper electrode respectively; in the second stage, when the sponge is squeezed, the distance between the nylon film and the PDMS film decreases, an induced potential difference is formed between the two electrodes, and the current flows from the first thin film electrode layer on the PDMS side to the second thin film electrode layer on the nylon side; in the third stage, the PDMS film and the nylon film continue to approach each other until they touch each other. At this time, the potential difference caused by the offset of the friction charge by the positive and negative charges gradually disappears, the current gradually decreases to zero, and the charge on the thin film electrode layer also decreases to almost zero; then in the fourth stage, the nylon film moves away from the PDMS film, and the two charged node films begin to separate, and an induced potential difference is formed between the two electrodes again, the electrons flow back, and the current flows from the second thin film electrode layer on the nylon side to the first thin film electrode layer on the PDMS side, forming a current to balance the electrostatic field until the first stage is restored, and thus a cycle is completed.

[0038] The present invention uses the above-mentioned friction nanogenerator unit to sense the changing characteristics of facial expressions of human faces for emotional analysis. The changes in facial expressions of human faces are closely related to the obvious facial muscle activities, especially the muscle activities around the eyes, which are most prominent at the center of the eyebrows and the outer corners of the eyes; in addition, since the facial structure is symmetrical, the left and right eyes are consistent in emotional changes, so the movements generated by the muscles around a single eye are sufficient for emotional analysis. Therefore, in an embodiment, two friction nanogenerator units are respectively attached to the inner and outer corners of the left (or right) eye of the subject.

[0039] Furthermore, the present invention has carried out a detailed test on the performance of the friction nanogenerator unit. During the test, by evaluating the performance of the friction nanogenerator units of different sizes, it is finally determined that the effective contact area of ​​a single friction nanogenerator unit is 1×2 cm 2 It should be pointed out that this size refers only to the effective friction contact area between the positive and negative friction layers of the triboelectric nanogenerator unit when it is working, rather than the final overall size of the entire triboelectric nanogenerator. The following tests the effect of different triboelectric nanogenerator areas on output performance:

[0040] A single friction nanogenerator unit was fixed at one end of a linear motor and made periodic contact and separation movements under the drive of the linear motor. The output voltage and output current of friction nanogenerator units with different areas were tested at an operating frequency of 2 Hz. 2 , 1×2cm 2 , 2×2cm 2 , 3×3cm 2 The output voltage and output current under the conditions, such as Figure 4As shown in the figure, as the unit area of ​​the friction nanogenerator increases, the output open circuit voltage increases from 0.43V to 4.3V, and the short circuit current increases from 7.16nA to 54.9nA. Therefore, increasing the unit area of ​​the friction nanogenerator can obtain a larger voltage. However, due to the limitations of the effective signal collection area of ​​the face and the wearing comfort, the unit area of ​​the friction nanogenerator is finally selected to be 1×2cm 2 Under this condition, the open circuit voltage of the friction nanogenerator unit is 0.84V, the short circuit current is 6.8nA, and the output signal can effectively represent emotional changes.

[0041] The two friction nanogenerator units attached to the inner and outer corners of the left eye (or right eye) of the subject are connected in series and then connected to an electrical signal acquisition device to realize signal acquisition (the series connection here means connecting the first thin film electrode layers of the two friction nanogenerator units as the first output end, connecting the second thin film electrode layers of the two friction nanogenerator units as the second output end, and then correspondingly connecting the first output end and the second output end to the positive and negative signal input ends of the electrical signal acquisition device). Further, in this embodiment, the electrical signal acquisition device adopts an electrometer.

[0042] Next, the above signal collection method is used to collect signal data of the subjects under different given emotions, and the data is preprocessed to construct a data set. The specific implementation method is as follows:

[0043] (1) Emotional data collection: The subjects were in a separate room, the sampling frequency was set to 300 points per second, and the data collected was voltage data. The subjects made facial expressions according to the given emotions, and each expression lasted for 3 seconds. Four types of emotions were collected at one time, namely happiness, sadness, anger, and surprise, thereby obtaining continuous voltage waveform data (see Appendix). Figure 5 ).

[0044] (2) Preprocess the data: Use fast Fourier transform to reduce the noise of the collected voltage waveform data, and resample to reduce the sample dimension. Set the interval sampling to sample one point every two points, and finally divide the data to split the waveform data into separate samples. Set the sample length to 225, fill the samples with 0 for the length less than 225, and cut the samples with a length greater than 225 to keep the data sample dimension consistent.

[0045] (3) Constructing a data set: In order to analyze different emotional data, this embodiment collects four types of emotions, each of which is repeated 25 times, for a total of 100 samples; 80% of the data is used as a training set, and the remaining 20% ​​is used as a test set for training and testing.

[0046] Next, we build an emotion recognition neural network model and use the above dataset to train and test the model. The specific implementation is as follows:

[0047] The main body of the neural network is a bidirectional LSTM. The data operation method is shown in the attached Figure 6 When processing data, bidirectional LSTM can access data streams from two directions simultaneously, enhancing the ability to mine data associations over long time intervals and automatically extract input data features.

[0048] When training the model, the relevant parameters are continuously adjusted to optimize the structure. The emotion recognition neural network model is composed of a bidirectional LSTM layer, a convolutional neural network layer and an Attention layer. The network structure composition is shown in the attached Figure 7 The model uses early stopping and learning rate adjustment strategies. Each time, only one parameter is adjusted, and then the loss changes are observed. The adjusted hyperparameters mainly include the number of network layers, network structure, number of neurons in the hidden layer, etc.

[0049] Number of network layers: The emotion recognition neural network model constructed in this embodiment is a two-layer bidirectional LSTM designed based on the LSTM basic unit. The model training rounds are determined by the early stopping method, and the learning rate uses a dynamic learning rate adjustment strategy.

[0050] Network structure: The emotion recognition neural network model constructed in this embodiment is a two-layer bidirectional LSTM combined with attention mechanism based on the LSTM basic unit design. This embodiment evaluates two-layer bidirectional, two-layer bidirectional combined with attention mechanism, three-layer bidirectional combined with attention mechanism, two-layer unidirectional, two-layer unidirectional combined with attention mechanism, and three-layer unidirectional combined with attention mechanism. The evaluation method is to use the same data to train different networks. The evaluation results are shown in Figure 2. Figure 8 As shown. Figure 8 It can be seen that the two-layer unidirectional network has the lowest accuracy, which may be due to the simple network structure that cannot extract effective information from the data. The network with two layers of bidirectional attention mechanism used in this system has the highest accuracy. As the number of network model layers increases, the model accuracy does not improve much and is prone to overfitting. Considering comprehensively, this system selects the most appropriate two-layer bidirectional attention mechanism network model for emotion classification.

[0051] Dynamic learning rate adjustment strategy: At the end of each training in the model compilation process, check whether the monitoring indicator is better than the best indicator recorded. If so, update the best indicator and reset the wait count. If not, increase the wait count. During the neural network training process, when the validation set performance has not improved for five consecutive training cycles (patience = 5), adjust the learning rate to 50% of the current value (factor = 0.5) and reset the wait counter. Repeat this until the lower limit of the set learning rate is reached.

[0052] Early stopping method: Use the early stopping method to return the optimal number of training times for the model. For different network structures, when the accuracy of the validation set does not change within 10 training cycles, it means that the classification effect of the model has reached the optimal level, and training is stopped at this time. In this experiment, the upper limit of epoch is set to 300.

[0053] Finally, after the emotion recognition neural network model is trained, the model can be applied to perform emotion recognition on the facial expression change data collected by the friction nanogenerator unit, that is, automatically identifying one of the four basic emotions of happiness, sadness, anger and surprise.

[0054] The above description is only used to illustrate the present invention, and is not intended to limit the present invention to the structure and scope of use shown and described. Therefore, all corresponding modifications and equivalents made within the spirit and principle of the present invention belong to the patent scope applied for by the present invention.

Claims

1. An emotion recognition method based on electrical signals of a triboelectric nanogenerator, characterized in that: The method uses a friction nanogenerator unit to sense changes in facial expressions of a person and converts the changes in facial expressions of a person into electrical signals; The structure of the friction nanogenerator unit is as follows: it includes a positive charge friction layer, a negative charge friction layer, an intermediate rebound support part, a first thin film electrode layer, a second thin film electrode layer and an outer packaging film, the positive charge friction layer and the negative charge friction layer are arranged oppositely and parallel to each other, and the intermediate rebound support part is arranged between the positive charge friction layer and the negative charge friction layer; the first thin film electrode layer is arranged on the outer surface of the negative charge friction layer, the second thin film electrode layer is arranged on the outer surface of the positive charge friction layer, and the first thin film electrode layer and the second thin film electrode layer are respectively led out of the outer packaging film through wires for connecting to an external circuit; Two friction nanogenerator units were respectively attached to the inner and outer corners of a subject's eyes, and connected to an electrical signal acquisition device to collect signal data of the subject under different given emotions, and the data was preprocessed to construct a data set; Then, an emotion recognition neural network model is constructed and the model is trained and tested using the dataset; Finally, the trained emotion recognition neural network model can be used to perform emotion recognition on the facial expression change data collected by the friction nanogenerator unit.

2. The emotion recognition method based on the electrical signal of the triboelectric nanogenerator according to claim 1 is characterized in that: The positive charge friction layer is made of nylon film, the negative charge friction layer is made of PDMS film, the middle rebound support part is made of sponge, and the middle rebound support part is located at the outer edge between the positive charge friction layer and the negative charge friction layer.

3. The emotion recognition method based on the electrical signal of the triboelectric nanogenerator according to claim 1 is characterized in that: The first thin film electrode layer and the second thin film electrode layer both use double-sided copper tape.

4. The emotion recognition method based on the electrical signal of the triboelectric nanogenerator according to claim 1 is characterized in that: The packaging outer film is made of PVC film.

5. The emotion recognition method based on the electrical signal of the triboelectric nanogenerator according to claim 1 is characterized in that: The effective friction area of ​​a single triboelectric nanogenerator unit is set to 1×2 cm 2 .

6. The emotion recognition method based on the electrical signal of the triboelectric nanogenerator according to claim 1 is characterized in that: The electrical signal acquisition device adopts an electrometer.

7. The emotion recognition method based on the electrical signal of the triboelectric nanogenerator according to claim 1 is characterized in that: The method of collecting the signal data of the subject under different given emotions and preprocessing the data to construct the data set is as follows: (1) Emotional data collection: The subjects were in a separate room, the sampling frequency was set to 300 points per second, and the data collected was voltage data. The subjects made facial expressions according to the given emotions, and each expression lasted for 3 seconds. Four types of emotions were collected at one time, namely happiness, sadness, anger, and surprise, and continuous voltage waveform data was obtained; (2) Preprocessing the data: Use fast Fourier transform to reduce the noise of the collected voltage waveform data, and resample to reduce the sample dimension. Set the interval sampling to sample one point every two points, and finally divide the data to split the waveform data into separate samples. Set the sample length to 225, fill the samples with 0 for the length less than 225, and cut the samples with a length greater than 225 to keep the data sample dimension consistent; (3) Constructing a data set: Collect 4 types of emotions and repeat each emotion 25 times, for a total of 100 samples; use 80% of the data as the training set and the remaining 20% ​​as the test set for training and testing.

8. The emotion recognition method based on the electrical signal of the triboelectric nanogenerator according to claim 1 is characterized in that: The emotion recognition neural network model consists of a bidirectional LSTM layer, a convolutional neural network layer and an Attention layer.

9. The emotion recognition method based on the electrical signal of the triboelectric nanogenerator according to claim 8 is characterized in that: When training the model, continuously adjust relevant parameters to optimize the structure, use early stopping method and learning rate adjustment strategy, and adjust only one parameter each time, and then observe the change in loss. The adjusted hyperparameters include the number of network layers, network structure, and the number of neurons in the hidden layer.

Images