Emotion recognition method based on deep learning algorithm and early warning system

Abstract

The invention discloses a deep learning algorithm-based emotion recognition method and early warning system, the early warning system comprises an intelligent bracelet, a camera, an information transmission module and a terminal module, the camera is arranged in front of a cockpit or an office table and is used for monitoring user expressions; in the emotion recognition process, physiological data are firstly collected, then data preprocessing, feature extraction and model training are carried out, and emotion recognition is carried out by collecting face data; and finally, pushing an emotion judgment result. According to the invention, through an advanced biosensing technology and a machine learning algorithm, the bracelet is utilized to monitor individual physiological indexes such as heart rate, blood pressure, skin conductance and the like in real time, and the indexes are closely related to emotional states. Through accurate action analysis, the system can recognize the behavior mode of an individual in a specific situation, and meanwhile, the face of the individual is detected and analyzed in real time through a camera, so that the emotional state of the individual is judged.

Description

Technical Field

[0001] The present invention relates to monitoring individual physiological indicators, and in particular to a method and early warning system for emotion recognition based on a deep learning algorithm. Background Art

[0002] With the rapid development of industrialization and urbanization, people are facing increasing pressure in work and life, which often comes from a fast-paced lifestyle, workplace competition, economic pressure, and interpersonal relationships. Emotional instability not only affects the mental health of individuals, but also may have a negative impact on decision-making if the person is a bus driver, a pilot of a civil aircraft, a ship driver, or a stock / financial derivatives trader, which in turn affects property losses, public safety, and social stability. Therefore, it is necessary to develop a detection device that can not only help individuals manage their emotions and reduce decision-making errors caused by emotional fluctuations, but also promote social harmony and stability at a broader social level by reducing public hazards and property losses caused by emotional instability.

[0003] With the development of smart wearable devices, the commonly used detection equipment is the smart bracelet. The use of smart bracelets is becoming more and more widespread. Common smart bracelets mostly tie the display panel to the user's wrist through a strap and record various physiological parameters of the user through multiple built-in sensors.

[0004] After searching, a Chinese patent with publication number CN 114451642 B discloses a smart bracelet, which adds circulation convex points to cooperate with a wristband, so that when the wristband exchanges air, the circulation convex points can be used to evenly disperse the exchanged air, thereby accelerating the drying of residual sweat on the user's wrist, and utilizing the air drying and heat absorption of sweat to bring a cool feeling to the user, thereby improving the user's experience. However, when the smart bracelet is worn, sweat on the wrist can easily enter the first circulation channel and is difficult to clean, and can easily breed bacteria. Summary of the invention

[0005] In order to achieve the purpose of the present invention, the technical solution adopted by the present invention is:

[0006] An early warning system for emotion recognition, the early warning system comprising a smart bracelet, a camera, an information transmission module and a terminal module, wherein the camera is arranged in front of a cockpit or in front of a desk to monitor user expressions;

[0007] Furthermore, the smart bracelet includes a shell and a display screen installed on the shell, a wristband is fixedly connected to one side of the shell, a connecting part for fixing the wristband is fixedly connected to the other side of the shell, a accommodating cavity is provided on the lower surface of the shell, an air supply member is arranged in the accommodating cavity, a circulation groove is provided on the lower surface of the wristband, a protrusion contacting the wrist is formed on the side of the lower surface of the wristband, an air passage connected to the air outlet end of the air supply member is arranged inside the protrusion, an air hole connected to the air passage is provided on the inner wall of the circulation groove, and the circulation groove cooperates with the protrusion to prevent sweat from entering the air hole.

[0008] Furthermore, the connecting part includes a connecting block fixedly connected to a side of the shell away from the wristband, one side of the connecting block is provided with a socket for inserting the wristband, the inner wall of the socket is slidably connected with a pressing block, and the pressing block can be lowered to fix the wristband and cut off the airway.

[0009] Furthermore, a through hole is provided on the upper surface of the connecting block, and a connecting rod is arranged in the through hole, the bottom end of the connecting rod is fixedly connected to the upper surface of the pressing block, the top end of the connecting rod is fixedly connected to the limiting strip, the upper surface of the connecting block is provided with an insertion hole, the lower surface of the limiting strip is fixedly connected with a fixing rod adapted to the insertion hole, a mounting hole is provided on the front side of the fixing rod, and a locking block is slidably connected to the inner wall of the mounting hole, a locking groove adapted to the locking block is provided on the inner wall of the insertion hole, a spring is fixedly connected to the inner wall of the mounting hole, one end of the spring is fixedly connected to the locking block, the inner wall of the locking groove is slidably connected to a push rod, the inner wall of the locking groove is provided with a limiting groove, and the inner wall of the limiting groove is slidably connected to the limiting block, and the limiting block is fixedly connected to the push rod.

[0010] Furthermore, the socket passes through the inner wall of the accommodating cavity so that the wristband can extend into the accommodating cavity, and the inner wall of the accommodating cavity is fixedly connected with a guide plate, and a bending portion is formed at one end of the guide plate close to the connecting block.

[0011] Furthermore, the air supply component includes an air bag fixedly connected to the top wall of the accommodating cavity, an air inlet is opened on the surface of the air bag, and an air inlet check valve is fixedly connected to the inner wall of the air inlet, a connecting hole is opened on the surface of the outer shell, an air outlet pipe is rotatably connected to the surface of the air bag, one end of the wristband is fixedly connected to a connecting pipe connected to the airway, the connecting pipe is rotatably connected to one end of the air outlet pipe, and the surface of the connecting pipe is fixedly connected to the air outlet check valve.

[0012] Furthermore, a movable plate is fixedly connected to the lower surface of the airbag, and the movable plate is slidably connected to the inner wall of the accommodating cavity. A fold is provided on the surface of the airbag, and an electromagnet is fixedly connected to the inner top wall of the accommodating cavity, and the electromagnet can adsorb the movable plate when energized.

[0013] Furthermore, a microcontroller and a battery are respectively provided on the upper surface of the guide plate, the inner wall of the accommodating cavity is fixedly connected to a base plate, the bottom of the base plate is fixedly connected to a piezoelectric generator that can be contacted by the wrist, the inner top wall of the accommodating cavity is fixedly connected to a position sensor, the bottom of the base plate is provided with a mounting opening, and the inner wall of the mounting opening is fixedly connected to a humidity sensor.

[0014] Furthermore, a skin galvanic response sensor and a photoelectric volumetric pulse wave sensor that fit the wrist are fixedly connected to the bottom of the base plate, and a microphone is fixedly connected to the upper surface of the shell.

[0015] A method for emotion recognition based on deep learning algorithm, comprising the following steps:

[0016] S1. Collect physiological data;

[0017] S2. Data preprocessing;

[0018] S3. Feature extraction;

[0019] S4. Model training;

[0020] S5. Emotion recognition by collecting face data;

[0021] S6. Push the emotion determination result.

[0022] Compared with the prior art, this patent application has the following advantages:

[0023] 1. The present invention uses advanced biosensor technology and machine learning algorithms to use a wristband to monitor an individual's physiological indicators in real time, such as heart rate, blood pressure, skin conductance, etc. These indicators are closely related to emotional state. Through precise motion analysis, the system can identify the behavior pattern of an individual in a specific situation, and at the same time, the camera can detect and analyze the individual's face in real time to determine his or her emotional state.

[0024] 2. Real-time monitoring and accurate identification of the user's emotional state can be achieved through high-precision biosensors and motion recognition technology. Secondly, the instant feedback mechanism provided by the bracelet enables users to respond quickly to emotional fluctuations and effectively reduce decision-making errors caused by emotional instability. Personalized emotional management suggestions further help users regulate their emotions and maintain mental health.

[0025] 3. In the present invention, by providing a circulation groove and a protrusion, and providing an air hole on the inner wall of the circulation groove, sweat from the wrist can be prevented from entering the air hole. By providing a single wristband and fixing it with a connecting part, there is no need to punch a hole on the wristband, thereby preventing sweat from entering and being difficult to clean. By providing an air supply member, an air duct and an air hole, the air circulation speed at the wrist position is increased, the evaporation of sweat is accelerated, and the comfort is improved. By providing a circulation groove, the space between the wristband and the wrist is increased to allow better air circulation, further increasing the evaporation speed of sweat, thereby enabling the smart bracelet to solve the problem that wristbands in the prior art are difficult to clean and easy to breed bacteria.

[0026] 4. In the present invention, the fixing rod is driven to move downward by pressing the limit bar downward until the locking block corresponds to the locking groove, and the spring resets to push the locking block to insert into the locking groove to lock the fixing rod. At the same time, the limit bar drives the connecting rod to move downward, and the connecting rod drives the pressing block to squeeze the wristband inserted into the socket to fix it so that the smart bracelet can be worn. At the same time, the pressing of the wristband by the pressing block can cut off the airway, so that the airflow only circulates in the wristband around the wrist, thereby increasing the air flow at the wrist position and improving the air drying effect.

[0027] 5. In the present invention, by providing a piezoelectric generator, during normal hand movements, the movement of the wrist will generate pressure on the piezoelectric generator, causing it to generate voltage and store it in a battery. By providing a humidity sensor and a position sensor to cooperate with each other to control the on and off of the electromagnet, the airbag can be exhausted and inflated, and the air in the airbag can be sent out to generate air flow at the wrist, so that the smart bracelet has an energy-saving effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 It is a schematic diagram of the three-dimensional structure of the present invention;

[0029] Figure 2 The first is a schematic diagram of the three-dimensional structure of the housing of the present invention;

[0030] Figure 3 The second schematic diagram of the three-dimensional structure of the housing of the present invention;

[0031] Figure 4 It is a schematic cross-sectional structural diagram of the housing of the present invention;

[0032] Figure 5 It is a schematic diagram of the front cross-section structure of the housing of the present invention;

[0033] Figure 6 It is a schematic cross-sectional structural diagram of the connecting block of the present invention;

[0034] Figure 7 For the present invention Figure 6 A schematic diagram of the enlarged structure at A in the middle;

[0035] Figure 8 It is a schematic cross-sectional structural diagram of a fixing rod of the present invention;

[0036] Fig. 9 is a schematic cross-sectional view of the wristband of the present invention;

[0037] Fig.10 For the present invention Fig. 9 A schematic diagram of the enlarged structure at B in the middle;

[0038] Fig.11 It is a schematic diagram of the three-dimensional structure of the airbag of the present invention.

[0039] The reference numerals are as follows:

[0040] 1. Shell; 2. Display screen; 3. Wristband; 4. Connecting part; 401. Connecting block; 402. Socket; 403. Pressing block; 404. Connecting rod; 405. Limiting strip; 406. Fixing rod; 407. Locking block; 408. Spring; 409. Ejector rod; 410. Limiting block; 5. Accommodating chamber; 6. Air supply part; 601. Air bag; 602. Inlet check valve; 603. Connecting hole; 604. Exhaust pipe; 605. Connecting pipe; 606 , air outlet check valve; 607, moving plate; 608, fold; 609, electromagnet; 610, microcontroller; 611, battery; 612, bottom plate; 613, piezoelectric generator; 614, position sensor; 615, humidity sensor; 7, flow slot; 8, protrusion; 9, airway; 10, pore; 11, guide plate; 12, bending part; 13, skin galvanic response sensor; 14, photoelectric volumetric pulse wave sensor; 15, microphone. DETAILED DESCRIPTION

[0041] The following is combined with Figure 1-11 The present invention is further described with examples:

[0042] Example 1

[0043] An early warning system for emotion recognition, the early warning system comprising a smart bracelet, a camera, an information transmission module and a terminal module, wherein the camera is arranged in front of a cockpit or in front of a desk to monitor user expressions;

[0044] like Figure 1-11As shown, the smart bracelet includes a shell 1 and a display screen 2 installed on the shell 1, a wristband 3 is fixedly connected to one side of the shell 1, and a connecting portion 4 for fixing the wristband 3 is fixedly connected to the other side of the shell 1. A accommodating cavity 5 is provided on the lower surface of the shell 1, and an air supply member 6 is arranged in the accommodating cavity 5. A circulation groove 7 is provided on the lower surface of the wristband 3, and a protrusion 8 contacting the wrist is formed on the side of the lower surface of the wristband 3. An air passage 9 connected to the air outlet end of the air supply member 6 is arranged inside the protrusion 8, and an air hole 10 connected to the air passage 9 is provided on the inner wall of the circulation groove 7. The circulation groove 7 cooperates with the protrusion 8 to prevent sweat from entering the air hole 10.

[0045] Specifically, by providing the circulation groove 7 and the protrusion 8, and providing the air hole 10 on the inner wall of the circulation groove 7, sweat from the wrist can be prevented from entering the air hole 10. By providing a single wristband 3 and fixing it with the connecting part 4, there is no need to punch holes in the wristband 3, thereby preventing sweat from entering and being difficult to clean. By providing the air supply part 6, the air duct 9 and the air hole 10, the air circulation speed at the wrist position is increased, the evaporation of sweat is accelerated, and the comfort is improved. By providing the circulation groove 7, the space between the wristband 3 and the wrist is increased to facilitate better air circulation, further increasing the evaporation speed of sweat, thereby enabling the smart bracelet to solve the problem that the wristband 3 in the prior art is difficult to clean and easy to breed bacteria.

[0046] The connecting portion 4 includes a connecting block 401 fixedly connected to a side of the housing 1 away from the wristband 3. A socket 402 for inserting the wristband 3 is provided on one side of the connecting block 401. A pressing block 403 is slidably connected to the inner wall of the socket 402. The pressing block 403 can be lowered to fix the wristband 3 and cut off the airway 9.

[0047] A through hole is provided on the upper surface of the connecting block 401, and a connecting rod 404 is arranged in the through hole, the bottom end of the connecting rod 404 is fixedly connected to the upper surface of the pressing block 403, and the top of the connecting rod 404 is fixedly connected to the limiting strip 405. An insertion hole is provided on the upper surface of the connecting block 401, and a fixing rod 406 adapted to the insertion hole is fixedly connected to the lower surface of the limiting strip 405. A mounting hole is provided on the front of the fixing rod 406, and a locking block 407 is slidably connected to the inner wall of the mounting hole, and a locking groove adapted to the locking block 407 is provided on the inner wall of the insertion hole, a spring 408 is fixedly connected to the inner wall of the mounting hole, one end of the spring 408 is fixedly connected to the locking block 407, a push rod 409 is slidably connected to the inner wall of the locking groove, a limiting groove is provided on the inner wall of the locking groove, and a limiting block 410 is slidably connected to the inner wall of the limiting groove, and the limiting block 410 is fixedly connected to the push rod 409.

[0048] Specifically, by pressing down the limit bar 405, the fixing rod 406 moves downward until the locking block 407 corresponds to the locking groove, and the spring 408 resets and pushes the locking block 407 to insert into the locking groove to lock the fixing rod 406. At the same time, the limit bar 405 drives the connecting rod 404 to move downward, and the connecting rod 404 drives the pressing block 403 to squeeze the wristband 3 inserted into the socket 402 to fix it so that the smart bracelet can be worn. At the same time, the pressing of the wristband 3 by the pressing block 403 can cut off the airway 9, so that the airflow only circulates in the wristband 3 around the wrist, thereby increasing the air flow at the wrist position and improving the air drying effect.

[0049] The socket 402 penetrates the inner wall of the accommodating cavity 5 so that the wristband 3 can extend into the accommodating cavity 5 . The inner wall of the accommodating cavity 5 is fixedly connected with a guide plate 11 . A bending portion 12 is formed at one end of the guide plate 11 close to the connecting block 401 .

[0050] Specifically, by inserting the wristband 3 into the accommodating cavity 5 along the insertion opening 402 , the exposed length of the wristband 3 can be adjusted according to the size of the user's wrist, and by providing the guide plate 11 , the wristband 3 is located at the bottom of the accommodating cavity 5 .

[0051] The air supply component 6 includes an air bag 601 fixedly connected to the inner top wall of the accommodating chamber 5, an air inlet is provided on the surface of the air bag 601, and an air inlet check valve 602 is fixedly connected to the inner wall of the air inlet, a connecting hole 603 is provided on the surface of the outer shell 1, an air outlet pipe 604 is rotatably connected to the surface of the air bag 601, one end of the wristband 3 is fixedly connected to a connecting pipe 605 connected to the airway 9, the connecting pipe 605 is rotatably connected to one end of the air outlet pipe 604, and an air outlet check valve 606 is fixedly connected to the surface of the connecting pipe 605.

[0052] The lower surface of the airbag 601 is fixedly connected with a movable plate 607, and the movable plate 607 is slidably connected to the inner wall of the accommodating chamber 5. The surface of the airbag 601 is provided with a fold 608, and the inner top wall of the accommodating chamber 5 is fixedly connected with an electromagnet 609, which can adsorb the movable plate 607 when energized.

[0053] A microcontroller 610 and a battery 611 are respectively provided on the upper surface of the guide plate 11, the inner wall of the accommodating cavity 5 is fixedly connected to a bottom plate 612, the bottom of the bottom plate 612 is fixedly connected to a piezoelectric generator 613 that can be contacted by the wrist, the inner top wall of the accommodating cavity 5 is fixedly connected to a position sensor 614, a mounting opening is opened at the bottom of the bottom plate 612, and a humidity sensor 615 is fixedly connected to the inner wall of the mounting opening.

[0054] Specifically, by setting the piezoelectric generator 613, during normal hand movements, the movement of the wrist will generate pressure on the piezoelectric generator 613, causing it to generate voltage and store it in the battery 611. By setting the humidity sensor 615 and the position sensor 614 to cooperate with each other to control the on and off of the electromagnet 609, the airbag 601 can be exhausted and inflated, and the air in the airbag 601 can be sent out and air flow can be generated at the wrist, so that the smart bracelet has an energy-saving effect. Among them, the humidity sensor 615 is used to sense the humidity of the wrist, and the position sensor 614 is used to detect the position of the moving plate 607. When the humidity is When the sensor 615 and the position sensor 614 simultaneously meet the power-on condition for the electromagnet 609, the battery 611 powers the electromagnet 609. At this time, the electromagnet 609 adsorbs the movable plate 607 to compress the airbag 601 and discharge the air in the airbag 601. When the humidity sensor 615 meets the power-on condition and the position sensor 614 meets the power-off condition, the electromagnet 609 is powered off. At this time, the movable plate 607 moves downward under its own gravity, the airbag 601 is reset and air is replenished. When the humidity sensor 615 and the position sensor 614 simultaneously meet the power-off condition, the electromagnet 609 is powered off and no longer powered.

[0055] The bottom of the bottom plate 612 is fixedly connected with a skin galvanic response sensor 13 and a photoelectric volume pulse wave sensor 14 that fit the wrist, and the upper surface of the housing 1 is fixedly connected with a microphone 15.

[0056] Specifically, the wearer's emotions and sleep quality are monitored by setting a skin galvanic response sensor 13, the user's heart rate and blood oxygen saturation are monitored by setting a photoplethysmogram sensor 14, and the microphone 15 is used for calls.

[0057] Working principle: When the smart bracelet is used, the user first wears the smart bracelet on the wrist, inserts one end of the wristband 3 into the receiving groove from the socket 402 and extends it inward at the bottom of the receiving cavity 5 under the cooperation of the curved portion 12 and the guide plate 11 until the wristband 3 is appropriately tight, presses down the limit bar 405, the limit bar 405 drives the fixing rod 406 to move downward, the fixing rod 406 drives the locking block 407 to move downward, until the locking block 407 corresponds to the locking groove, the spring 408 resets and pushes the locking block 407 to insert into the locking groove, locks the fixing rod 406, and at the same time the limit bar 405 drives the connecting rod 40 4 moves downward, the connecting rod 404 drives the pressing block 403 to squeeze the wristband 3 inserted into the socket 402 to fix it so that the user can wear the smart bracelet. At this time, the pressing block 403 can cut off the airway 9 by pressing the wristband 3, so that the airflow only flows in the wristband 3 around the wrist, thereby increasing the air flow at the wrist position. During normal hand movements, the movement of the wrist will generate pressure on the piezoelectric power generation sheet 613, causing it to generate voltage and store it in the battery 611. The humidity sensor 615 senses the humidity of the wrist, and the position sensor 614 detects the position of the moving plate 607. When the humidity sensor 61 5 and position sensor 614 simultaneously meet the power-on condition for electromagnet 609, battery 611 powers electromagnet 609, at this time, electromagnet 609 attracts movable plate 607, compresses airbag 601, and exhausts air in airbag 601, and the exhausted air passes through outlet pipe 604, connecting pipe 605, air passage 9 in sequence, and then ejects from air hole 10 into flow slot 7, and then is discharged outward from the gap between protrusion 8 and wrist, thereby increasing the air circulation speed at wrist position, accelerating sweat evaporation, and improving comfort, utilizing flow slot 7 and protrusion 8, and setting air hole 10 on the inner wall of flow slot 7, to prevent wrist Sweat enters the pores 10, and the circulation slots 7 are used to increase the space between the wristband 3 and the wrist to allow better air circulation, further increasing the rate of sweat evaporation. When the humidity sensor 615 meets the power-on condition and the position sensor 614 meets the power-off condition, the electromagnet 609 is powered off. At this time, the movable plate 607 moves downward under its own gravity, the airbag 601 is reset and air is replenished. When the humidity sensor 615 and the position sensor 614 meet the power-off condition at the same time, the electromagnet 609 is powered off and no longer powered on, thereby enabling the smart bracelet to solve the problem that the wristband 3 in the prior art is difficult to clean and easy to breed bacteria.

[0058] Example 2

[0059] A method for emotion recognition based on deep learning algorithm, comprising the following steps:

[0060] S1. Collecting Physiological Data

[0061] PPG physiological signal data is collected by the photoplethysmography sensor 14. Signal data under various emotional states are collected as much as possible, such as relaxation, anxiety, anger, tension, etc., to establish a rich training data set.

[0062] S2. Data Preprocessing

[0063] The PPG physiological signal data is bandpass filtered to remove motion artifacts and noise. The bandpass filtered PPG physiological signal is then normalized so that data of different dimensions can be used in the same model. The long-term physiological signal is divided into a fixed-length time window (3 seconds) for feature extraction.

[0064] S3. Feature extraction

[0065] S31. Extract time domain features:

[0066] Extract the beats per minute (BPM) from the PPG physiological signal. Extract the amplitude (RMS) of the sound.

[0067] S32. Calculate the RR interval:

[0068] First, data is collected from PPG physiological signals. These data are usually expressed in the form of time series, recording the time point of each heartbeat.

[0069] Determine the R wave: In the PPG physiological signal, the R wave corresponds to the point of maximum change in blood volume.

[0070] Calculate the RR interval: Once the position of the R wave is determined, the time difference between two consecutive R waves can be calculated. If the R wave of the nth heartbeat occurs at time t n , the R wave of the n+1th heartbeat occurs at time t n+1 ,

[0071] Then during the nth RR period RR n It can be calculated as:

[0072] RR n =t n+1 -t n

[0073] The above process is repeated for all heart beats during the entire recording period to obtain a sequence of RR intervals.

[0074] Arrange all calculated RR intervals into a list or array for further HRV analysis.

[0075] S33. Calculate the standard deviation of normal RR intervals and the square root of the mean of the squares of the RR interval differences

[0076] SDNN is the standard deviation of all normal RR intervals. It reflects the heart rate variability of the entire time series. Formula:

[0077]

[0078] Where RRi is the i-th RR interval, is the average value of RR intervals, and N is the total number of RR intervals.

[0079] RMSSD is the square root of the mean of the squares of the differences between consecutive RR intervals and reflects the variability between adjacent heart beats.

[0080]

[0081] Where RRi and RRi+1 are consecutive RR intervals. GSR feature: Calculate the average value of skin conductance.

[0082] S33. Calculate the average skin conductance

[0083] First, the skin conductance values ​​are continuously recorded over a period of time using the galvanic skin response sensor 13. These data are usually obtained in the form of a time series, recording the skin conductance readings at each time point.

[0084] Determine a time window, 1 or 2 minutes, over which to calculate the average. This time window should be long enough to reflect stable skin conductance levels, but short enough to capture rapid changes.

[0085] Before calculating the average value, the collected data were screened to exclude outliers.

[0086] The specific method is to draw a box plot to show the distribution of data. The box plot can show the median, quartiles, upper and lower limits of the data. Outliers usually exceed the upper and lower limits of the box plot. If there are data points outside the whiskers of the box plot, they are outliers.

[0087] For all skin conductance readings within the selected time window, calculate their average. The formula is as follows:

[0088]

[0089] Among them, GSR i is the skin conductance reading at the ith time point, and N is the total number of data points in the time window.

[0090] S34. Calculate frequency domain features

[0091] The HRV signal is collected by the photoplethysmography sensor 14, and the spectrum analysis is performed on the HRV signal to extract the low frequency (LF) and high frequency (HF) components for evaluating the activities of the sympathetic and parasympathetic nerves.

[0092] For signals generally in the range of 0.04-0.15 Hz, they are extracted as low frequency (LF) components, and for signals generally in the range of 0.15-0.4 Hz, they are extracted as high frequency (LF) components, and then the LF / HF ratio is calculated.

[0093] The extracted LF component is used to assess the relative activity of the sympathetic and parasympathetic nerves. Higher power values ​​of the LF component may indicate that the sympathetic nerves are relatively active. The extracted LF component is used to assess the relative activity of the parasympathetic nerves (vagus nerve). Higher power values ​​of the LF component may indicate an increase in parasympathetic nerve activity. The ratio of LF to HF power (LF / HF) is used to assess the relative balance of sympathetic and parasympathetic nerve activity. An increase in the LF / HF ratio may indicate an increase in sympathetic nerve activity relative to parasympathetic nerve activity. A decrease in the LF / HF ratio may indicate an increase in parasympathetic nerve activity relative to sympathetic nerve activity.

[0094] S4. Model training

[0095] Select the long short-term memory network (LSTM) to train the model. During the training process, select a physiological signal sequence of length T:

[0096] X = [x1, x2, ..., x T ]

[0097] Each X t is the physiological data vector at time step t (every 3 seconds). The physiological data vector X at each time step t There are multiple features. X t The expression is:

[0098] X t =[RMS, BPM, SDNN, RMSSD, GSR]

[0099] RMS sound amplitude, BPM heart beats per minute, SDNN standard deviation of heart rate variation, RMSSD root mean square of heart rate variation, GSR average skin conductance.

[0100] S41: Initialization:

[0101] Initialize the hidden state h of the previous time step t-1 and cell status C t-1 At time step 1, h0 and C0 are usually initialized to zero vectors.

[0102] S42: Forget gate calculation:

[0103] The input of the forget gate includes the physiological data X of the current time step t and the hidden state h of the previous time step t-1 , the parameter of the forget gate is the weight matrix W f and the bias vector b f ,

[0104] The calculation formula of the forget gate is: f t =σ(W f ·[x t ,h t-1 ]+b f )

[0105] Here, σ is the sigmoid activation function, which outputs a value between 0 and 1 indicating the degree of forgetting.

[0106] S43: Input gate calculation:

[0107] The input gate consists of two parts: a sigmoid layer for input modulation and a tanh layer for input candidate values. The parameter of the input gate is the weight matrix W i , W c and the bias vector b i and b c .

[0108] The calculation formula of the input gate is:

[0109]

[0110] S44: Update unit status:

[0111] Update the cell state according to the results of the forget gate and the input gate:

[0112] Here, * indicates element-wise multiplication.

[0113] S45: Output gate calculation:

[0114] The input data X of the current time step t and the hidden state h at the previous time step t-1 Combined, then multiplied with the weight matrix W0, and added with the bias b0. t Usually between 0 and 1, which means it can control the amount of information flowing, similar to a valve. Use the sigmoid activation function σ to get the activation value O of the output gate t

[0115] o t =σ(W o ·[x t,h t-1 ]+b o )

[0116] S46: Update hidden status:

[0117] The activation value of the output gate is O t Determines the final hidden state h t Middle cell status C t The output ratio of O is obtained by element-by-element multiplication. t The output of the activation function applied to the cell state usually uses the hyperbolic tangent function tanh to convert the cell state to a more tractable value.

[0118] h t =O t *tanh(C t )

[0119] S47: Loop to the next time step:

[0120] h t and C t As the initial hidden state and unit state of the next time step. Then, according to the result of the output gate, the current emotion is classified as excitement, quiet, angry, etc. At the same time, the data is transmitted to the early warning system in real time through the Internet, so that supervisors can intervene in time.

[0121] S5. Emotion recognition by collecting face data

[0122] Facial data is collected through the camera, and key frames are extracted from the video based on the shot video to obtain multiple face pictures. The artificial intelligence MaskR-CNN algorithm is used for instance segmentation. Multiple pre-prepared face pictures are used as training data sets, and the pre-trained convolutional neural network ResNet is used as the backbone network to extract feature maps.

[0123] The results of compressing the length and width twice, three times, four times, and five times are taken out, namely p1, p2, p3, and p4 to construct the feature pyramid structure.

[0124] The candidate regions are generated through RPN, and classification and bounding box regression are performed to determine whether the captured content contains the target, and the suggestion box is adjusted to achieve pixel-level object segmentation.

[0125] The candidate region is generated using the following formula:

[0126] a x，y =(x+w a ·r·cos(θ),y+h a ·r·sin(θ))

[0127] Among them, W a and h a are the width and height of the candidate region, r and θ are the scale and rotation angle respectively.

[0128] Use the labelme package in Python to annotate the obtained pixel-level images. After completing the annotation of all images, use the labelme2coco package in Python to convert the labelme format dataset into COCO format to obtain the training model.

[0129] Then, based on the generated training model, real-time face detection is performed on the images extracted from the video to generate COCO format images, thereby achieving pixel-level object segmentation.

[0130] For the detected face region, the model's classification head is used to perform emotion classification. The classification head usually outputs the probability of each emotion category, and selects the category with the highest probability (greater than 80%) as the emotion of the face.

[0131] As users use it, more data is collected and the model is updated regularly to improve personalized emotion recognition capabilities.

[0132] S6. Push emotion determination results

[0133] When the terminal module detects the emotion result, on the one hand, it is pushed to the bracelet in real time through the information transmission module to remind the wearer of the current emotion;

[0134] On the other hand, by pushing the information to the early warning center in real time through the information transmission module, the supervisor can understand the person’s emotional state in real time.

[0135] When the detected emotions are excitement or anger, the bracelet will remind the wearer to calm down. At the same time, the monitoring center will use early warning messages to remind supervisors whether necessary measures need to be taken.

[0136] The embodiments of the present invention disclose preferred embodiments, but are not limited thereto. A person skilled in the art can easily understand the spirit of the present invention based on the above embodiments and make different extensions and changes. However, as long as they do not deviate from the spirit of the present invention, they are all within the protection scope of the present invention.

Claims

1. An early warning system for emotion recognition, characterized in that: The early warning system comprises a smart bracelet, a camera, an information transmission module and a terminal module, wherein the camera is arranged in front of the cockpit or in front of the desk for monitoring the user's expression; the smart bracelet comprises a shell (1) and a display screen (2) mounted on the shell (1), characterized in that a wristband (3) is fixedly connected to one side of the shell (1), a connecting portion (4) for fixing the wristband (3) is fixedly connected to the other side of the shell (1), and a lower surface of the shell (1) is provided with An accommodating cavity (5) is provided inside the accommodating cavity (5), a flow groove (7) is provided on the lower surface of the wristband (3), a protrusion (8) in contact with the wrist is formed on the side of the lower surface of the wristband (3), an air passage (9) connected to the air outlet end of the air supply member (6) is provided inside the protrusion (8), an air hole (10) connected to the air passage (9) is provided on the inner wall of the flow groove (7), and the flow groove (7) cooperates with the protrusion (8) to prevent sweat from entering the air hole (10).

2. A smart bracelet as claimed in claim 1, characterized in that: The connecting portion (4) comprises a connecting block (401) fixedly connected to a side of the housing (1) away from the wristband (3); a socket (402) for inserting the wristband (3) is provided on one side of the connecting block (401); a pressing block (403) is slidably connected to the inner wall of the socket (402); the pressing block (403) can be lowered to fix the wristband (3) and cut off the airway (9).

3. A method for emotion recognition based on deep learning algorithm, characterized in that: The steps include: S1. Collect physiological data; S2. Data preprocessing; S3. Feature extraction; S4. Model training; S5. Emotion recognition by collecting face data; S6. Push the emotion determination result.

Images