An active grasping hand function training system based on multi-dimensional emotion recognition

Abstract

The present invention proposes an active gripping hand function training system based on multi-dimensional emotion recognition; it includes a rehabilitation training ball, a core processing module, a training interaction module and a multi-source data acquisition module. The core processing module performs emotion recognition on pulse wave data and electrocardiogram data respectively, and fuses the emotion recognition results corresponding to facial expression, pulse wave and electrocardiogram to obtain a comprehensive emotion recognition result. The rehabilitation training ball is used for the user to hold and detect the user's gripping force. The training interaction module is used to guide the user to perform hand gripping, rotation and movement training. The present invention comprehensively judges the user's emotional changes during hand training through facial expression, electrocardiogram and pulse wave signals, and when the user has negative emotions, reduces the training difficulty by reducing the required threshold of hand training gripping force and the threshold of rotation and movement amplitude, thereby alleviating the user's resistance and prompting the user to train more actively and proactively.

Description

Technical Field

[0001] The present invention belongs to the technical field of human-computer interaction training equipment, and in particular relates to an active grasping hand function training system based on multi-dimensional emotion recognition. Background Art

[0002] There are many deficiencies in the technology and application of rehabilitation equipment on the current market. For example, many rehabilitation exercise equipment lacks real-time individual monitoring and dynamic adjustment functions, which makes it difficult to reasonably adjust the equipment according to the user's actual rehabilitation progress, which can easily cause the user to overuse or operate improperly, thereby bringing the risk of secondary injury; in addition, the mechanical design of some equipment is complex and the operation is cumbersome, which brings additional burdens to users and caregivers; at the same time, the accuracy of the sensor feedback system is insufficient, and it cannot reflect the user's physical condition in real time and accurately, which increases the uncertainty and potential safety hazards during use.

[0003] Some rehabilitation equipment on the market, such as pneumatic gloves, rely on air pumps to force users to perform rehabilitation training. Although this can improve rehabilitation efficiency to a certain extent, its forced mechanical action may cause hand injuries to the user. In addition, such equipment often lacks fun, and the rehabilitation process is monotonous and repetitive, which can easily cause users to resist during rehabilitation training and reduce the enthusiasm and sustainability of rehabilitation training. At the same time, many rehabilitation devices have a single function and only train a certain part of the fingers or wrist. They lack comprehensiveness and cannot meet the multi-dimensional rehabilitation needs of users.

[0004] Therefore, the market urgently needs an innovative rehabilitation device that can combine real-time monitoring, intelligent dynamic adjustment, multi-dimensional comprehensive training and interesting interaction to meet the actual rehabilitation needs of users with hand dysfunction, improve rehabilitation efficiency and safety, reduce the psychological burden of users, and thus comprehensively improve the quality of life of users. Summary of the invention

[0005] The purpose of the present invention is to provide an active grasping hand function training system based on multi-dimensional emotion recognition.

[0006] The present invention provides an active grasping hand function training system based on multi-dimensional emotion recognition, comprising a rehabilitation training ball, a core processing module, a training interaction module and a multi-source data acquisition module. The multi-source data acquisition module comprises an image emotion recognition module, an electrocardiogram acquisition module and a photoelectric pulse wave sensor. The image emotion recognition module is used to collect the user's facial image and identify the emotion type corresponding to the expression. The electrocardiogram acquisition module is used to collect the user's electrocardiogram data. The photoelectric pulse wave sensor is used to collect the user's pulse wave data.

[0007] The core processing module performs emotion recognition on pulse wave data and ECG data respectively, and integrates the emotion recognition results corresponding to facial expression, pulse wave and ECG to obtain the final comprehensive emotion recognition result. The rehabilitation training ball is used for the user to hold and detect the user's grip strength; the training interaction module is used to guide the user to perform hand grip, rotation and movement training; the core processing module is used to detect the user's grip strength according to the comprehensive emotion recognition results. Adjust the difficulty of hand grasping, moving, and rotating training to prevent users from maintaining negative emotions.

[0008] Preferably, the rehabilitation training ball comprises a shell, a posture sensor installed in the shell, and a pressure detection module. The shell is provided with a gripping groove corresponding to a human finger. The shell comprises an upper hemisphere shell, a middle base, and a lower hemisphere power supply box which are stacked in sequence to form a hollow spherical structure. The posture sensor is used to detect the posture data of the rehabilitation training ball and obtain the rotation amplitude of the user's hand. The pressure detection module is used to detect the pressure on the end of the gripping groove.

[0009] Preferably, the pressure detection module adopts a spring pressing collection structure. The gripping groove includes five finger accommodating grooves, and the ends of the five finger accommodating grooves are provided with mounting holes. A spring pressing collection structure is fixed in each mounting hole. The spring pressing collection structure includes a fixing seat, a pressing block, a spring and a pressure sensor. The fixing seat is fixed in the mounting hole, the pressing block is arranged on the outside of the fixing seat, and a spring and a pressure sensor are arranged between the pressing block and the fixing seat.

[0010] Preferably, the rehabilitation training ball further comprises a training ball controller and a power supply module. The training ball controller and the posture sensor are installed in the middle base. The power supply module is installed in the power supply box of the lower hemisphere.

[0011] Preferably, the training interaction module includes a training ball tracking module and a touch screen. The touch screen is used to display a screen that guides the user's hand grasping, rotation and movement. The training ball tracking module includes a servo gimbal and a digital camera installed on the servo gimbal. The digital camera is used to capture the image of the user's hand holding the rehabilitation training ball. The core processing module detects the position of the rehabilitation training ball in the image, and adjusts the direction of the digital camera in real time through the servo gimbal so that the digital camera continues to capture the position of the rehabilitation training ball. The core processing module determines the movement amplitude of the user's hand based on the position change of the rehabilitation training ball.

[0012] Preferably, the training interaction module further comprises an audio player, which is used to play background music.

[0013] Preferably, the training interaction module further comprises a distance measurement module, which is used to detect the distance between the user's body and the touch screen.

[0014] Preferably, the core processing module measures the comprehensive emotion recognition result. When the emotion is positive, the training difficulty is reduced; the way to reduce the training difficulty includes lowering the required threshold of grip force and reducing the threshold of the amplitude of hand movement and rotation. When the emotion is negative, the difficulty of training is increased; ways to increase the difficulty of training include increasing the required threshold of grip force and increasing the threshold of the amplitude of hand movement and rotation.

[0015] Preferably, the core processing module performs emotion recognition on the ECG signal features extracted based on the ECG data and the pulse wave signal features extracted based on the pulse wave data respectively through the random forest algorithm to obtain the ECG emotion type vector , pulse wave emotion type vector The image emotion recognition module uses the YOLOv5 algorithm to perform emotion recognition on facial images and obtain the expression recognition emotion type vector .

[0016] As a preferred method, the process of fusing the emotion recognition results corresponding to facial expressions, pulse waves, and electrocardiograms is as follows:

[0017] (1) Concatenate ECG emotion type vectors , pulse wave emotion type vector , and obtain the feature vector . The feature vector is Processing to obtain the preliminary prediction vector .

[0018] (2) Emotional recognition vector and the initial prediction vector Input the gradient boosting tree algorithm together to obtain the emotion prediction results .

[0019] (3) Emotion prediction results Perform regularization-assisted correction to obtain comprehensive emotion recognition results .

[0020] Preferably, the process of extracting ECG signal features from ECG data by the core processing module is as follows:

[0021] (1) Based on the bilateral slope, the steepness of each sampling point in the ECG data is detected.

[0022] (2) Construct a dynamic window based on RR interval to extract R peak as follows:

[0023] a. Set the steepness M of each sampling point in the starting segment of the ECG data steep Compare with the preset initial threshold T0 to obtain the suspected R peak; the steepness M of each sampling point after the starting segment of the ECG data is calculated. steep Greater than or equal to the dynamic threshold T a The sampling point is taken as the R peak. Dynamic threshold T a It is set according to the steepness of multiple R peaks before the current sampling point.

[0024] b. Build a dynamic window.

[0025] Dynamic window width The expression is as follows:

[0026]

[0027] Wherein, RRavr represents the average value of multiple RR intervals before the current sampling point. The starting point of the dynamic window is the interval before the end point of the previous dynamic window. Distance.

[0028] c. When there are multiple R peaks in a dynamic window, only the local maximum value is retained to determine the position of the R peak in the dynamic window.

[0029] (3) Extract the P, Q, S, and T feature points based on the R peak position to obtain the ECG signal features.

[0030] Preferably, the process of calculating the steepness is:

[0031] (1) Calculate the slopes on both sides of the sampling point , .

[0032] (2) For the slope With slope The steepness M is calculated at sampling points with opposite signs. steep as follows:

[0033]

[0034] in, , are two weight coefficients.

[0035] As a preference, set the weight coefficient , And the process of initial threshold T0 is as follows:

[0036] Set the weight coefficient between 0.25 and 0.75 Multiple candidate values ​​of; weight coefficient =1− For each weight coefficient Candidate values ​​of the slope M are calculated for each sampling point. steep . Traverse all steepness M steep The maximum value M in steepMax , set the initial threshold T0=M steepMax / 3; take the maximum value M steepMax The corresponding weight coefficient , As the final weight coefficient , .

[0037] Preferably, the process of extracting pulse wave signal features from pulse wave data by the core processing module is as follows:

[0038] Set a moving window in the pulse wave data; extract the local maximum value LocMax in the moving window i ; If the local maximum value LocMax i If the following conditions are met, the local maximum value LocMax in the moving window is determined i The corresponding position is taken as the peak feature point.

[0039]

[0040] Among them, T b It represents the peak threshold of the main wave; ΔT represents the threshold allowable error.

[0041] A trough detection window is constructed with each peak feature point as the end point; the local minimum point of the amplitude in the trough detection window is used as the trough feature point. The width of the trough detection window is 1 / 10 of the distance between two adjacent peaks. Multiple time domain features and frequency domain features are extracted as pulse wave signal features based on the peak feature points and trough feature points of the main wave.

[0042] The beneficial effects of the present invention are as follows:

[0043] 1. The present invention comprehensively judges the emotional changes of the user during the hand training process through facial expressions, electrocardiograms, and pulse wave signals, and reduces the difficulty of training by reducing the required threshold of hand training grip force and the threshold of rotation and movement amplitude when the user has negative emotions, thereby reducing the user's resistance to rehabilitation training and prompting the user to train more actively and proactively.

[0044] 2. The present invention improves and uses a key feature point extraction algorithm based on bilateral slope. By constructing a dynamic window and screening the R peak position in the dynamic window, the R peak of the electrocardiogram signal is accurately detected. The detection method is verified in the MIT-BIH dataset. The results show that the average accuracy of identifying the R peak using the algorithm reaches 99.10%. Based on this, the present invention accurately extracts the key feature point PQRST feature point of the electrocardiogram signal, thereby improving the accuracy of electrocardiogram recognition of emotions.

[0045] 3. The rehabilitation training ball in the present invention serves as a detection device for finger grip force and hand rotation, as well as a positioning object for visually detecting hand movement, thereby improving the accuracy of visual hand movement detection, so that the present invention can accurately capture the user's training movements, so as to achieve accurate mapping between the training movements and the game screen. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 It is a system block diagram of the present invention.

[0047] Figure 2 It is a structural block diagram of the core device in the present invention.

[0048] Figure 3 It is a structural block diagram of the rehabilitation training ball in the present invention.

[0049] Figure 4 It is a structural block diagram of the core processing module of the present invention.

[0050] Figure 5 It is a three-dimensional schematic diagram of the rehabilitation training ball of the present invention.

[0051] Figure 6 The figure is a schematic diagram of the explosion of the rehabilitation training ball of the present invention.

[0052] Figure 7 This is a diagram showing the effect of preprocessing the electrocardiogram signal in step 2 of the present invention.

[0053] Figure 8 This is a diagram showing the effect of preprocessing the pulse wave signal in step 2 of the present invention.

[0054] Fig. 9 This is a schematic diagram of the R peak detection result of the central electrical signal in step 3 of the present invention.

[0055] Fig.10 This is a schematic diagram of the positioning results of the key characteristic points of the central electrical signal in step 3 of the present invention.

[0056] Fig.11 Schematic diagram of the pulse wave signal key feature point positioning result in step 4 of the present invention.

[0057] Fig.12This is an operational flow chart of performing interactive game-style hand training in step 8 of the present invention.

[0058] Fig.13 This is a schematic diagram of the finger coordination rehabilitation training game in step 8 of the present invention.

[0059] Fig.14 This is a schematic diagram of the finger force control rehabilitation training game in step 8 of the present invention.

[0060] Fig.15 This is a schematic diagram of the all-round wrist flexibility rehabilitation training game in step 8 of the present invention.

[0061] Fig.16 This is a schematic diagram of the arm flexibility rehabilitation training game in step 8 of the present invention.

[0062] Fig.17 This is a schematic diagram of the comprehensive collaborative rehabilitation training game in step 8 of the present invention.

[0063] In the figure, a-auxiliary equipment; b-core equipment; 1-gesture sensor; 2-pressure sensor; 3-training ball controller; 4-core controller; 5-photoelectric pulse wave sensor; 6-ECG acquisition module; 7-power supply module; 8-image emotion recognition module; 9-voice module; 10-digital camera; 11-distance measurement module; 12-servo gimbal; 13-touch screen; 14-audio player; 15-spring pressing acquisition structure; 16-pressure acquisition pressure dividing board; 17-lithium battery; 18-power control board; 19-expansion board; 20-PWM output port; 21-servo power supply port; 22-ECG acquisition module data return interface; 23-distance measurement module data return interface; 24-upper hemisphere shell; 25-middle base; 26-lower hemisphere power box. DETAILED DESCRIPTION

[0064] The present invention is further described below in conjunction with the accompanying drawings.

[0065] like Figure 1 and Figure 2 As shown, an active grasping hand function training system based on multi-dimensional emotion recognition includes a core device b and an auxiliary device a. The core device b includes a core processing module, a training interaction module and a multi-source data acquisition module. The core device b is used for data processing, algorithm analysis, and interaction with the user. The auxiliary device a is a rehabilitation training ball, which is used to obtain the user's finger pressure data and device posture data.

[0066] like Figure 4As shown, the core processing module includes a core controller 4, an expansion board 19 and a voice module 9. The expansion board 19 is connected to the core controller 4 through a female connector. The core controller 4 adopts Remi Pi. An analog-to-digital conversion module is integrated on the expansion board 19, and the model of the analog-to-digital conversion module is AD7606, which is used to collect pulse wave data. The expansion board 19 reserves a PWM output port 20 for driving the servo gimbal 12, a servo power supply port 21, an ECG acquisition module data return interface 22 and a distance measurement module data return interface 23. The voice module 9 specifically adopts an ASR voice recognition sensor to identify relevant voice instructions of the user to operate the training interaction module.

[0067] The training interaction module includes a training ball tracking module, a distance measurement module 11, a touch screen 13 and an audio player 14. The training ball tracking module includes a servo gimbal 12 and a digital camera 10 mounted on the servo gimbal 12. The digital camera 10 is used for image tracking and identification of the position of the rehabilitation training ball, and the servo gimbal 12 is used to assist the digital camera 10 in stable tracking. The distance measurement module 11 uses a Tof laser distance measurement sensor to measure the distance between the touch screen 13 and the user's body, and reminds the user to stay away from the screen appropriately to protect his eyesight when the distance reaches a certain threshold. The touch screen 13 is used for user touch control interaction. The audio player 14 is used to play background music.

[0068] The multi-source data acquisition module includes an image emotion recognition module 8, an ECG acquisition module 6, and a photoelectric pulse wave sensor 5. The image emotion recognition module 8 uses a MAIX-III AXera-Pi embedded edge AI device, which can collect the user's facial image in real time, capture the user's expression, and use the YOLOv5 algorithm for emotion analysis. The ECG acquisition module 6 is used to collect the user's ECG data for emotion analysis. The photoelectric pulse wave sensor 5 is used to collect the user's pulse wave data for emotion analysis.

[0069] The core processing module uses the pulse wave data measured by the photoelectric pulse wave sensor 5 and the ECG data measured by the ECG acquisition module 6 to perform emotion analysis, and communicates with the image emotion recognition module 8 through the serial port to obtain the user's facial expression and emotion data. The comprehensive emotion recognition result is obtained by combining the ECG data, pulse wave data and facial expression and emotion data Y final This function is designed to dynamically adjust the difficulty of rehabilitation training according to the user's current emotional state, reduce the user's resistance, and appropriately increase the difficulty when the user is in a good mood and maintains a positive mood for a long time, thereby optimizing and improving the user's rehabilitation experience. This highly integrated processing method not only improves the efficiency of the equipment, but also ensures the personalization and accuracy of rehabilitation training.

[0070] like Figure 3 , Figure 5and Figure 6 As shown, the rehabilitation training ball is held by the user to obtain the device posture data and finger pressure data, and also acts as a positioning object in the object tracking system. The rehabilitation training ball includes a shell, a posture sensor 1, a training ball controller 3, a power supply module 7, a spring pressing collection structure 15 and a pressure collection pressure dividing plate 16. The shell includes an upper hemisphere shell 24, a middle base 25 and a lower hemisphere power box 26 which are stacked in sequence to form a hollow spherical structure.

[0071] A gripping groove is provided on the upper hemispherical shell 24. The gripping groove is an inner concave structure designed with reference to ergonomics to meet the requirements of stable hand gripping, so as to facilitate stable grip and force application by the user. The gripping groove includes five finger-accommodating grooves, and the ends of the five finger-accommodating grooves (i.e., the positions corresponding to the fingertips) are provided with mounting holes. A spring pressing and collecting structure 15 is fixed in each mounting hole. The spring pressing and collecting structure 15 includes a fixing seat, a pressing block, a spring and a pressure sensor 2. The fixing seat is fixed in the mounting hole, and the pressing block is arranged on the outside of the fixing seat, and a spring and a pressure sensor 2 are provided between the pressing block and the fixing seat. The pressure sensor 2 specifically adopts an FSR flexible pressure sensor, which can detect the pressure of the corresponding finger of the user on the pressing block.

[0072] The middle base 25 is equipped with a training ball controller 3, a posture sensor 1 and a pressure collection voltage divider 16. The training ball controller 3 uses the ESP32-S3 development board. The model of the posture sensor 1 is MPU9250. The posture sensor 1 is used to obtain the acceleration, magnetic field, and angle data of the rehabilitation training ball to calculate the device posture. The pressure collection voltage divider 16 is used to cooperate with the potentiometer to divide the voltage signal output by the pressure sensor 2 and calibrate the initial pressure data.

[0073] The lower hemisphere power box 26 is equipped with a power supply module 7 for supplying power to the rehabilitation training ball. The power supply module 7 includes a lithium battery 17 and a power control board 18; the rated voltage and rated current of the lithium battery 17 are 7.4V and 2A respectively. The lithium battery 17 is used to supply power to the training ball controller 3 and the pressure collection voltage divider board 16. The power control board 18 is used to independently control the power on and off of the training ball controller 3 and the pressure collection voltage divider board 16.

[0074] Based on the above structure, the training ball controller 3 can collect the user's hand pressure data and posture data, and send the data to the core device b in the local area network environment via the websocket protocol.

[0075] The technical solution adopted by this embodiment to solve the technical problem includes the following steps:

[0076] Step 1: The wearable device collects multi-source data from the user.

[0077] The user faces the image emotion recognition module 8, holds the rehabilitation training ball in his hand, and wears the photoelectric pulse wave sensor 5 and the electrocardiogram acquisition module 6. The image emotion recognition module 8, the electrocardiogram acquisition module 6, and the photoelectric pulse wave sensor 5 respectively collect the user's facial expression image, electrocardiogram data, and pulse wave data during the rehabilitation training process, so as to analyze the user's emotional state later.

[0078] Step 2: Preprocessing of physiological signals.

[0079] Process the ECG signal and pulse wave signal for emotion analysis as follows:

[0080] Since ECG and PPG are non-stationary and weak signals, they are easily affected by power frequency interference, baseline drift, myoelectric interference and other noise. Power frequency interference is usually 50Hz or 60Hz, mainly from power supply and electrical equipment. Baseline drift has a frequency range of 0.05Hz to 1Hz, which is caused by breathing or changes in body position. Myoelectric interference has a frequency range of 30Hz to 300Hz, which comes from muscle activity.

[0081] In the case where the noise frequency is concentrated above 40 Hz, this embodiment uses a third-order Butterworth low-pass filter with a cutoff frequency of 40 Hz to remove high-frequency noise, and uses a median filter to eliminate baseline drift, and uses a maximum-minimum normalization method to normalize the data to the range of [0,1] to eliminate signal dimension differences. The formula is as follows:

[0082]

[0083] in, X norm For the processed data, X is the amplitude of the signal before processing, X min is the minimum value of the signal, X max is the maximum value of the signal. The effect of ECG signal preprocessing is as follows Figure 7 The pulse wave signal preprocessing effect is shown in Figure 8 shown.

[0084] The signal processing results show that the original signal has been significantly improved, especially the high-frequency noise and baseline drift problems have been effectively eliminated. The quality of the processed signal has been significantly improved, providing a more accurate and clear data basis for subsequent analysis. At the same time, the amplitude of the signal is adjusted to the range of 0 to 1, which not only improves the processing quality of the signal, but also retains the original waveform characteristics of the signal, ensuring the integrity of the original information. More importantly, the normalized signal is dimensionless, so that signals under different emotional states can be effectively compared, further enhancing the comparability of the data and the accuracy of the analysis.

[0085] Step 3: Extract ECG signal features.

[0086] The ECG signal data is simplified and key features are extracted, as follows:

[0087] The quality of feature extraction directly determines the recognition accuracy of the system. In order to improve the recognition accuracy, it is necessary to extract features that are both representative and stable from the processed signals. These features can accurately reflect the nature of the signal and are robust to environmental changes, ensuring high recognition accuracy under different circumstances.

[0088] The QRS wave is an extremely important waveform in the ECG signal, representing the process of ventricular depolarization. Since the ventricular muscle is thicker than the atrium, the amplitude of the QRS wave is higher than the waveform of other parts, becoming the most obvious feature of the ECG. In the time domain analysis of the ECG signal, extracting and identifying the QRS complex wave is one of the core tasks. Accurately detecting the QRS wave helps to quickly identify other feature points. Based on the waveform characteristics of the ECG, this embodiment improves and uses a key feature point extraction algorithm based on bilateral slope. The specific steps are as follows:

[0089] 3-1. Calculate the steepness based on the bilateral slope and initialize the threshold.

[0090] 3-1-1. Calculate the left slope of all sampling points of ECG data respectively M Left and the right slope M Right ; The duration of the QRS wave is usually 0.06 seconds to 0.10 seconds; the ventricular wall activation time (VAT, the time interval between the beginning of the QRS complex and the vertical line of the R wave peak) is usually less than 0.03 seconds. Therefore, the step interval for calculating the slope is selected to be 0.03 seconds to eliminate the possibility of the slope value being too low due to the different shapes of the QRS complex waveform. The expression for calculating the slope is as follows:

[0091]

[0092]

[0093]

[0094] in, EEG ( i ) represents the value of the current sampling point of the ECG signal; f s Indicates the sampling frequency of the signal; Δi represents the step interval for calculating the slope, M Left and M Right Respectively represent the left slope and right slope of the current sampling point.

[0095] 3-1-2. Due to individual differences, the rise and fall ratios of the QRS complex waveform are different. In order to eliminate this difference, the weight coefficient is introduced ω Left and ω Right Build steepness M steep The expression is as follows:

[0096]

[0097] The signs of the slopes on the left and right sides of the R peak are opposite, so the following relationship must also be satisfied:

[0098]

[0099] 3-1-3. Setting weight coefficient ω Left , ω Right , and the initial threshold T 0.

[0100] In the above steps, the steepness around the QRS complex wave will be very high, but a reasonable threshold needs to be determined to determine whether it is an R peak, and the current weight coefficient ω Left and ω Right It is also unknown. Therefore, it is necessary to use known data to initialize the threshold and weight. The heart rate of a normal person is 1-1.5 times per second, and at least two QRS waveforms are required for initialization. Therefore, the data of the first 4 seconds is used to calculate the initial threshold T 0 and weight coefficient ω Left and ω Right , the initialization steps are as follows:

[0101] (1) Data input: input ECG signal ECG(i) and sampling frequencyf s .

[0102] (2) Set the weight coefficient between 0.25 and 0.75 in steps of 0.01. ω Left multiple candidate values ​​(specifically 51 candidate values ​​in this embodiment); weight coefficient ω Right =1- ω Left For each weight coefficient ω Left Candidate values ​​are calculated for different sampling points within 4 seconds. i The steepness M steep (i).

[0103] (3) Traverse and search for all the steepness obtained in step (2) M steep The maximum value in (i) M steepMax。

[0104] (4) Setting the initial threshold T 0= M steepMax / 3; take the maximum value M steepMax The corresponding weight coefficient ω Left , ω Right As the final weight coefficient ω Left , ω Right .

[0105] 3-2. Construct a dynamic window based on RR interval to extract R peak.

[0106] 3-2-1. The steepness of the first 4 seconds of the ECG data M steep Greater than or equal to the initial threshold T 0 The sampling point is taken as the suspected R peak.

[0107] 3-2-2. Set a dynamic threshold for the sampling points after 4 seconds before the ECG data T a As a judgment condition.

[0108] In order to better adapt to the changes in ECG, the dynamic threshold T a Perform dynamic updates as follows:

[0109]

[0110] in, M RpeakAvrSteep It represents the average steepness of the last eight R peaks measured before the current sampling point; T a is the R peak threshold after dynamic update.

[0111] The steepness of the sampling points 4s before and after the ECG data M steep Greater than or equal to the dynamic threshold T a The sampling point is taken as the R peak.

[0112] 3-2-3. Construct dynamic windows and filter out R peaks in each dynamic window.

[0113] The RR interval refers to the interval between two adjacent R peaks. For normal adults, the RR interval is generally 0.6-1.0 seconds at rest. Physiology shows that there is a refractory period within 0.2 seconds after the R peak appears, that is, the R peak will not appear again during this period. Based on this, in order to detect the QRS waveform faster and more accurately, a dynamic window based on the RR interval needs to be constructed.

[0114] Each dynamic window width WIN 1 is calculated as follows:

[0115]

[0116]

[0117] in, RR avr It represents the average value of the 8 detected RR intervals before the current moment; RR n-i For the current moment n Previous i The duration of each RR interval; ΔR is the preset interval, and in this embodiment, the value is 0.48· fs , which is determined based on the normal range of the RR interval and the refractory period; when RR avr =0.6 fs , due to the existence of a refractory period of 0.2 seconds, the range of the next window should be 0.2 seconds-1.0 seconds. In order to make it have a certain fault tolerance, the range of the window is set to 0.12 seconds-1.08 seconds. RR avr =1· fs When the heart rate drops suddenly, the RR interval can be kept within the range of 0.52 seconds to 1.48 seconds.

[0118] The starting point of the dynamic window is before the end point of the previous dynamic window. ΔR Distance.

[0119] In the dynamic window range based on the RR interval, under normal and stable conditions, only one peak should be detected. However, when the individual's emotions change due to the influence of the external environment, the RR interval will also change significantly, resulting in multiple R peaks in the dynamic window; the steepness of multiple points in a QRS complex waveform can meet the dynamic threshold T a When the R peak usually corresponds to the point with the highest rate of ventricular depolarization, the position of the R peak in the dynamic window is determined by searching for the local maximum value. The specific process is as follows: from the first point that meets the dynamic threshold T a The point to the last point that meets the dynamic threshold T a A window is constructed from the points of the dynamic threshold and the local maximum search method is used to locate the R peak in the window. However, sometimes multiple local maxima meet the dynamic threshold. T a To solve this problem, the point with the largest steepness should be selected as the R peak feature point, which can ensure that each QRS complex wave is counted only once, thereby improving the accuracy of the analysis.

[0120] During the detection process, the subject may have abnormal contact with the acquisition device, resulting in a decrease in the amplitude of the ECG signal and a decrease in the steepness of the R peak, which makes it impossible to pass the originally set higher threshold detection. To adapt to this situation, the threshold needs to be adjusted quickly and a backtracking search is performed: if no R peak is detected in a dynamic window, the dynamic threshold is adjusted. T a After reducing by 50%, the steepness of each sampling point in the dynamic window is compared, and the steepness of the sampling points in the dynamic window is screened out to be greater than the updated dynamic threshold. T a The sampling point is taken as the R peak. The ECG signal R peak detection result is as follows Fig. 9 shown.

[0121] 3-3. Extract PQST feature points based on R peak position to obtain ECG signal features

[0122] The R peak can be used as a reference point for extracting other feature points. According to the physiological characteristics of other feature points, extreme values ​​can be found on both sides of the R peak. The feature points of the Q wave and the S wave are located on the left and right sides of the R peak, respectively, and appear as minimum values ​​in the ECG. Therefore, 0.48· fsThe window of the Q wave and the S wave is searched for the minimum value to determine the location of the characteristic points of the Q wave and the S wave (i.e., the Q peak and the S peak); the P wave usually appears between 0.08 seconds and 0.20 seconds on the left side of the R peak, and the T wave is located between 0.2 seconds and 0.4 seconds on the right side of the R peak; therefore, a window can be constructed within these intervals and the maximum value can be searched to determine the location of the characteristic points of the Q wave and the S wave (i.e., the P peak and the T peak). The results of locating the key characteristic points of the ECG signal are shown in the figure below. Fig.10 shown.

[0123] Through the above-mentioned key feature point extraction algorithm based on bilateral slope, the five key feature points are accurately detected, and the features are extracted based on the time amplitude information and time information between PQRST. The average power of four frequency bands of 0.02-0.06Hz, 0.06-0.15Hz, 0.15-0.4Hz, and 0.4-0.8Hz is extracted in the frequency domain. The feature extraction of ECG signal is shown in Table 1.

[0124] Table 1 ECG signal characteristics

[0125]

[0126] The ECG input features are composed of the ECG signal features extracted from each R peak and its corresponding P, Q, S, and T feature points.

[0127] Step 4: Extract pulse wave signal features.

[0128] The waveform structure of the pulse wave signal is simpler than that of the ECG signal, and the most important feature points are the peak point P and the trough point U of the main wave. In order to quickly extract these key feature points, this embodiment proposes an automatic detection algorithm based on the pulse wave waveform structure, combined with dynamic thresholds, moving windows, and local extreme values, for quickly extracting the key feature points of the pulse wave signal. For normal people, the peak and trough of the pulse wave usually appear between 0.7 seconds and 0.9 seconds per cycle, so the designed moving window width is According to the waveform structure of the pulse wave, both the main wave peak and the tidal wave peak appear as local maxima. At the same time, the pulse wave data is discrete, which means that there may be multiple maximum points identified as peaks in one cycle.

[0129] To solve this problem, this embodiment uses a method combining local maximum search and dynamic threshold to determine the peak feature point of the main wave, as follows:

[0130] Extracting local maxima in a moving window LocMax i ; If the local maximum LocMax i If the following conditions are met, the local maximum value in the moving window is determined LocMax iThe corresponding position is the peak characteristic point of the main wave.

[0131]

[0132] in, LocMax i represents the local maximum value within the window, T b Indicates the peak threshold of the main wave, the initial value is set to 0.9; ΔT Indicates the threshold tolerance; the threshold range can be expanded or narrowed by modifying the parameter α. In order to improve the applicability of the algorithm, the peak threshold of the main wave T b Dynamic adjustment is made according to the peak size of the last five main waves. The specific calculation formula is as follows:

[0133]

[0134] in, T n-i is the peak amplitude of the main wave.

[0135] Within the window range, if the local maximum value does not meet the formula (11), the window is moved to the right with a step size of 1 and the above steps are repeated. If there are multiple maximum values ​​that meet the conditions within the window, the largest local maximum value is taken as the peak of the main wave, and the step size is set to the window length.

[0136] According to the characteristics of the pulse wave, the first minimum value on the left side of the peak of each main wave is the trough feature point of the main wave, and the time interval between the trough and the peak accounts for about one tenth of the pulse cycle. Therefore, the trough detection window is constructed with the peak feature point of the main wave as the end point; the trough detection window width ; PP Int is the interval between the two adjacent wave peaks, that is, the pulse cycle. In the window interval, find the location of the trough feature point of the main wave according to the local minimum value. The key feature point positioning result of the pulse wave signal is as follows: Fig.11 shown.

[0137] The peak point P and trough point U of the main wave can be extracted through the above steps, and 7 time domain features and 4 frequency domain features of the pulse wave signal can be extracted based on the peak point P and trough point U. Each feature and its specific description are shown in Table 2.

[0138] Table 2 Pulse wave signal characteristics

[0139]

[0140] Among them, the ascending branch area UP_Area and descending branch area PU_Area The calculation method is as follows:

[0141]

[0142] The ratio of the pulsating component of the pulse wave pressure to the maximum value of the arterial component K The expression is as follows:

[0143]

[0144] in, P s It indicates systolic pressure, i.e. peak value; P d It indicates diastolic pressure, i.e. trough value; P m represents mean arterial pressure; T Represents a pulse wave cycle; K It is a key feature of the pulse waveform, reflecting the elasticity of the vascular wall and peripheral resistance. K The value is related only to the area of ​​the pulse wave shape.

[0145] The pulse wave input features are composed of the pulse wave signal features extracted from the peak point P and the trough point U of each main wave.

[0146] Step 5: Based on the random forest algorithm, emotion type recognition is performed on the ECG input features and pulse wave input features respectively.

[0147] After the above steps 2, 3, and 4, the ECG data and pulse wave data are processed to obtain ECG input features and pulse wave input features related to emotional states. Further classification of these feature samples according to different emotional states is a key step in achieving accurate emotional judgment. By testing different machine learning algorithms, including KNN, SVM, decision tree, etc., the random forest algorithm (RF) is finally selected to perform real-time emotional analysis on pulse wave input features and ECG input features based on prediction accuracy.

[0148] The random forest algorithm is an ensemble learning algorithm that improves the accuracy and robustness of predictions by building multiple decision trees and combining their outputs (using voting for classification and averaging for regression).

[0149] The random forest algorithm includes Bootstrap sampling from the training set to generate multiple sub-datasets, building a decision tree for each subset, randomly selecting features at each node split, and finally obtaining the model output through majority voting or average value. The random forest algorithm can effectively process multidimensional data, has strong anti-noise ability and anti-overfitting characteristics, and is widely used in classification and regression tasks; information gain is an important indicator for evaluating the effectiveness of feature partitioning of data sets. The core gain calculation formula of the random forest algorithm is as follows:

[0150]

[0151] in, Gain(S,A) Representation characteristics A For the dataset S Information gain of Entropy(S) It is a dataset S The entropy reflects the purity of the data set; |S v | is in the feature A The value of v , the number of samples in the subset; |S| It is a dataset S The total number of samples; Entropy(S v ) Representation characteristics A The value of v The entropy of the data subset at time .

[0152] Through the above formula, the random forest algorithm can effectively evaluate the contribution of each feature to the classification decision, thereby selecting the optimal feature for node division and finally building a robust emotion recognition model. The embedded deployment of this method not only improves the real-time emotion analysis capability of the device, but also provides reliable technical support for emotion regulation in rehabilitation training.

[0153] The hyperparameters involved in the random forest algorithm model all use the random parameter search method to perform random combination search within a given range to obtain the optimal result of the current model. The hyperparameters related to the random forest algorithm are shown in Table 3.

[0154] Table 3 Random forest algorithm related hyperparameters

[0155] Algorithm Type parameter Random Forest Algorithm Number of decision trees: (n_estimators:47) Minimum number of separation samples: (min_samples_split:2) Maximum number of separation features: (max_features:auto) Maximum depth of decision tree: (max_depth:10) Minimum number of leaf node samples: (min_samples_leaf:3)

[0156] Through the random forest algorithm, the three-division (i.e., negative, positive, and neutral) emotion type vectors based on the ECG input features and the three-division emotion type vectors based on the pulse wave input features are obtained as follows:

[0157] Emotional type vector divided into three parts:

[0158]

[0159] in, e ECG,1 represents the probability of negative emotions obtained through ECG data analysis, e ECG,2 represents the probability of neutral emotion obtained through ECG data analysis, e ECG,3 Represents the probability of positive emotions obtained through ECG data analysis.

[0160] Pulse wave three-division emotion type vector:

[0161]

[0162] in, e PPG,1 represents the probability of negative emotions obtained through pulse wave data analysis, e PPG,3 represents the probability of neutral emotion obtained through pulse wave data analysis, e PPG,3 Represents the probability of positive emotions obtained through pulse wave data analysis.

[0163] Step 6: Identify the emotion type of the user's facial image based on the YOLOv5 algorithm.

[0164] The image emotion recognition module 8 based on MAIX-III AXera-Pi embedded edge AI device is used; the image emotion recognition module 8 performs real-time emotion recognition analysis on facial expressions through the YOLOv5 algorithm; the target detection algorithm aims to accurately identify and locate multiple objects in images or videos. Traditional methods usually require multiple scans of images to identify targets, which is inefficient. YOLO transforms the target detection problem into a regression problem of a single neural network, and achieves target detection by dividing the input image into grids and predicting bounding boxes and class probabilities in each grid cell; YOLO's network structure includes multiple convolutional layers and pooling layers, as well as fully connected layers for outputting prediction results. The algorithm uses a deep convolutional neural network and divides the input image into S×S grids, each of which is responsible for predicting multiple bounding boxes and corresponding confidence scores and class probabilities. Specifically, for each bounding box, the YOLO network predicts its center coordinates, width and height, as well as the category probability and confidence of the corresponding target, and predicts a total of 5 values. Among them, the confidence indicates whether the bounding box contains the target object, and the category probability indicates which category the object belongs to. Its loss function comprehensively considers the positioning error and classification error. The YOLO algorithm is also optimized for the detection of multiple targets. To this end, the YOLO algorithm uses fine-grained feature maps to detect small targets, and uses convolutional neural networks to perform multi-scale calculations on feature maps before network output. At the same time, the YOLO algorithm merges feature maps of different scales into a global feature vector for overall detection.

[0165] By building an emotion dataset to train the YOLOv5 algorithm model, a model library that meets expectations is obtained and deployed to the embedded board. The final three-partition expression recognition emotion type vector is as follows:

[0166]

[0167] in, e Face,1 represents the classification probability of negative emotions obtained through expression recognition analysis, e Face,2 represents the classification probability of neutral emotions obtained through expression recognition analysis, e Face,3 Represents the classification probability of positive emotions obtained through expression recognition analysis.

[0168] Step 7: Based on the three-division emotion type vector fusion of the decision tree, the user's current emotion type is obtained.

[0169] Since the user's facial expressions change slightly and are easily affected by subjective concealment and delayed emotional response, facial expressions are difficult to use as the main basis for emotion analysis. Based on physiological psychology theory, emotional changes are closely related to the activity of the autonomic nervous system. Changes in physiological signals can more objectively reflect emotional states and are not affected by subjective will, thereby providing a more accurate identification method. Therefore, the present invention uses physiological signal features (i.e., ECG input features and pulse wave input features) as the main basis for emotion analysis, and performs preliminary emotion recognition through decision trees; at the same time, combined with the gradient boosting trees (GBT) algorithm, the emotion vectors obtained by expression recognition are used as auxiliary corrections to further improve the accuracy of emotion analysis. The specific steps for decision fusion of three emotion type vectors (emotion vectors based on ECG, pulse wave and expression recognition) are as follows:

[0170] 7-1. Preliminary sentiment prediction (based on decision tree).

[0171] Use the three-part ECG and pulse wave emotion vectors in equations (17) and (18) as input features and combine them into a feature vector :

[0172]

[0173] Using decision tree model f bio Train physiological signal features and output preliminary prediction vectors Y initial :

[0174]

[0175] Here Y initial The results of the initial prediction of the three-division sentiment categories (negative, neutral, and positive probabilities).

[0176] 7-2. Auxiliary correction (based on gradient boosting tree).

[0177] Use the three-partition expression recognition emotion type vector obtained in formula (19) E Face and the initial prediction vector Y initial Use it as input feature for auxiliary correction. Construct input feature matrix X correction :

[0178]

[0179] Using Gradient Boosted Tree Model f GBT For the input feature matrix X correction Train to learn the correction pattern between expression recognition and preliminary prediction, so as to obtain the final emotion prediction result Y fusion :

[0180]

[0181] The gradient boosting tree algorithm can handle the complex nonlinear relationship between input features, making the auxiliary correction more accurate. Through the auxiliary correction of facial expressions, we can use the facial recognition information to fine-tune the emotions with unclear boundaries based on the dominant physiological signal features, thereby achieving deep fusion of multi-source information.

[0182] 7-3. Regularization-assisted correction.

[0183] The regularization-assisted correction is to adjust the final emotion prediction by predefined rules to make it more consistent with the actual situation. The following are the predefined rules in this embodiment.

[0184] 7-3-1. Threshold rule: If the probability of a certain emotion in facial expression recognition exceeds the set threshold (set to 0.8 in this embodiment), the preliminary prediction vector is ignored and the emotion category is directly used as the final result.

[0185] 7-3-2. Offset correction rule: When the probability of a certain emotion in expression recognition is significantly higher than the initial prediction, but does not reach the threshold of direct replacement set in rule 7-3-1, the initial predicted emotion vector can be offset corrected. The specific steps are as follows:

[0186] (1) Trigger condition for modification: When the probability of a certain emotion (such as positive, negative, or neutral) in expression recognition is higher than the initial prediction vector Y initial When the probability of a replacement exceeds the set value (35%) but does not reach the threshold for direct replacement, an offset correction is triggered.

[0187] (2) Correction strategy: Increase the probability of the emotion in the preliminary prediction vector, which is set to increase by 10% in this embodiment. In order to keep the sum of the emotion vectors at 100%, it is necessary to reduce the corresponding probabilities of other categories of emotions accordingly.

[0188] (3) Adjustment calculation: Assume that the initial prediction vector is P =[ p 1, p 2, p 3] (representing the probabilities of positive, negative, and neutral, respectively).

[0189] When a certain emotion category of expression recognition p i When the trigger correction condition needs to increase by 10%, the probability of updating the emotion category p i for:

[0190]

[0191] In order to keep the total probability equal to 1, we need to adjust the probabilities of other emotions and distribute the reduced probabilities to the remaining emotion categories in proportion. If the original probabilities of the remaining two categories are p j and p k , then adjust the update probability of the other two emotions and for:

[0192]

[0193] Through this offset correction, the emotion prediction results can be fine-tuned when expression recognition shows a strong trend but not significant enough, making emotion fusion more flexible and accurate.

[0194] 7-3-3. Emotion reinforcement rule: Set a reinforcement factor for the expression recognition results of specific emotions. For example, when a specific strong emotion such as anger is detected (a subclass of negative emotions), the weight of negative emotions can be increased even if the initial prediction is neutral or positive. Similarly, when the expression recognition shows a strong smile (a subclass of positive emotions), the weight of positive emotions can be increased even if the initial prediction is neutral.

[0195] 7-3-4. Combination correction rules for emotion categories: mainly for some special combinations of preliminary predictions and expression recognition results. For example, if the probability of negative and neutral emotions is high in the expression recognition results, and the preliminary prediction is "positive", the final prediction will be adjusted to "neutral" to reflect the mixed state of emotions. This adjustment rule helps to classify emotions more appropriately when multiple emotion signals are relatively close, and avoid being too biased towards a single emotion category.

[0196] 7-3-5. Time smoothing rule: In the emotion type vector fusion, the detection of the same emotion for a long time can be used as a weight adjustment condition. The specific implementation steps are as follows:

[0197] (1) Tracking of emotion duration: The emotion results of the expression recognition algorithm in the past 15 seconds are recorded. If the same emotion (such as positive, negative or neutral) is detected continuously, the emotion is considered to be consistent.

[0198] (2) Weight enhancement strategy: Increase the weight according to the continuous time. Set a time-increasing function (an exponential function in this embodiment) to enhance the weight of the emotion category as the continuous detection time increases. The specific enhancement strategy can be set as follows:

[0199]

[0200] in, W final is the final weight, W base is the base weight, ΔW is the increment of weight increase T consistent is the time (in seconds) that the same emotion is maintained continuously.

[0201] (3) Upper limit: Set a maximum value (20%) for weight increase to prevent the weight from increasing indefinitely.

[0202] This approach can make the emotion results more stable when the same emotion is detected continuously, which helps to improve the accuracy of emotion fusion.

[0203] 7-4. Output prediction results.

[0204] The final comprehensive emotion recognition result can be expressed as:

[0205]

[0206] in, RuleCorrection(x) Represents the regularization auxiliary correction function in 7-3, which is used to adjust the emotion prediction results after gradient boosting tree (GBT) fusion to make it more consistent with the predefined life experience rules, so as to obtain the final comprehensive emotion recognition resultsY final .

[0207] Step 8: Conduct interactive game-style hand training for the user.

[0208] like Fig.12 As shown, the operation process of the training interaction module of the present invention is as follows: first, the user needs to wear the photoelectric pulse wave sensor 5, the electrocardiogram acquisition module 6 and the rehabilitation training ball in sequence, and adjust the position of the digital camera 10 to clearly obtain the position of the rehabilitation training ball, and adjust the position of the image emotion recognition module 8 to ensure that the user's expression state can be clearly acquired; after the device is started, the system first captures the user's specific instructions through voice recognition or touch input, and starts the corresponding rehabilitation training game based on these inputs; at the same time, the system begins to analyze the user's emotions in real time and collect other relevant sensor data; during the rehabilitation training process, the system's built-in intelligent emotion regulation system will dynamically adjust the game's interactive difficulty according to the user's real-time emotional state. This intelligent regulation mechanism ensures that the user can perform rehabilitation training in a relaxed and pleasant environment, effectively reducing the user's negative emotions; finally, the system will record the user's relevant data during the training process, so that the doctor can understand the user's rehabilitation status and adjust the training plan in time. The game-style hand training method and emotional recognition intelligent dynamic adjustment method of the training interaction module are as follows:

[0209] (1) If Fig.13 As shown, it is a schematic diagram of the finger coordination rehabilitation training game of the human-computer training interaction module of the present invention. In this embodiment, the user needs to follow the rhythm of the music and pay attention to the black note blocks moving along different tracks. By exerting force with fingers at the right time to press the spring pressing collection structure 15, the user can eliminate these black note blocks at a specific point of the music rhythm. When the black note block reaches the elimination area (i.e. Fig.13 When the user reaches the required threshold of grip strength, the black note block will be successfully eliminated and the score will be increased. If the black note block is not pressed at the correct rhythm point and the elimination area is missed, the corresponding score will be deducted. The game records the total score after the set time.

[0210] In this embodiment, the intelligent dynamic adjustment method is: the intelligent emotion adjustment system adjusts the elimination area size of the black note block and the required threshold of the gripping force according to the user's current emotional state.

[0211] (2) If Fig.14 As shown in the figure, it is a schematic diagram of the finger force control rehabilitation training game of the human-computer training interaction module of the present invention. In this embodiment, the user continuously presses the spring pressing collection structure 15 with his fingers to reach the required threshold of gripping force, and controls Fig.14The height of the water columns in the five pipes is adjusted so that each water column reaches the set red water level line. By carefully coordinating the grip of different fingers, the user can carry out targeted finger rehabilitation training. The total time required for each water column to successfully reach the red water level line will be converted into a score.

[0212] In this embodiment, the intelligent dynamic adjustment method is: the intelligent emotion adjustment system will intelligently adjust the required threshold of gripping force, the height of the red water level line and the rising speed of the water column according to the user's current emotional state.

[0213] (3) If Fig.15 The figure is a schematic diagram of the all-round wrist flexibility rehabilitation training game of the human-machine training interaction module of the present invention. In this embodiment, the user rotates the rehabilitation training ball by wrist to change the pitch angle, roll angle and yaw angle of the rehabilitation training ball. Fig.15 The ball in the maze passes through the maze until it reaches the end. Among them, the pitch angle controls the up and down movement of the ball, and the roll angle controls the left and right movement of the ball. When facing the diamond-shaped obstacles in the picture, the user needs to adjust the yaw angle of the rehabilitation training ball so that the ball switches its color to the color of the corresponding diamond obstacle to eliminate the obstacle. When the yaw angle is greater than the set rotation amplitude threshold, the ball switches to orange, and when the yaw angle is less than the set rotation amplitude threshold, the ball switches to blue. Finally, the ball needs to be controlled to eliminate all obstacles and reach the green end. The total time the user takes to control the ball through the maze will determine the user's score.

[0214] In this embodiment, the intelligent dynamic adjustment method is: the intelligent emotion adjustment system will intelligently adjust the amplitude thresholds of the pitch angle, roll angle and yaw angle rotation required to control the movement of the ball according to the user's current emotional state.

[0215] (4) If Fig.16 As shown, it is a schematic diagram of the arm flexibility rehabilitation training game of the human-computer training interaction module of the present invention. In this embodiment, the system provides patterns of various difficulty levels, and the difficulty of drawing the pattern is closely related to the size of the key point hitting area and the number of key points. The user controls the movement of the digital camera 10 and the servo gimbal 12 by moving the arm, draws the corresponding graphics in space, hits the key points of the pattern through visual guidance, and the game ends when the last key point is hit. The system will analyze the similarity between the graphics drawn by the user and the original pattern through curve fitting, and calculate the training score in combination with the total drawing time.

[0216] In this embodiment, the intelligent dynamic adjustment method is: the intelligent emotion adjustment system will dynamically adjust the size of the key point hitting area and the number of key points in the pattern according to the user's emotional state.

[0217] (5) If Fig.17The figure shows a schematic diagram of a comprehensive collaborative rehabilitation training game of the human-machine training interaction module of the present invention. Multiple sensors are used to help users perform coordinated movement rehabilitation training for fingers, wrists and arms. In this embodiment, the user needs to control the rehabilitation training ball to move the digital camera 10 and the steering gear pan / tilt 12. Fig.17 The hand positioning pattern in the figure is moved to different fruit areas, and the position coordinates of the hand positioning pattern are displayed at the hand positioning pattern. After that, the user needs to observe the rotation direction of the fruit, and adjust the wrist angle to make the rehabilitation training ball reach the corresponding angle position, so that the hand positioning pattern in the figure rotates to an angle consistent with the fruit pattern. Next, the user presses the spring pressing collection structure 15 with the finger force to reach the required threshold of the gripping force, completing a simulated picking action. Picking different fruits requires different amounts of gripping force. The system calculates the final score based on the number of fruits picked by the user within the specified time.

[0218] In this embodiment, the intelligent dynamic adjustment method is: the emotion adjustment system will dynamically adjust the rotation angle of the fruit pattern and the required threshold value of the gripping force when picking fruits according to the user's emotional state, thereby realizing intelligent adjustment of rehabilitation training.

[0219] During the user's rehabilitation training, the training ball tracking module dynamically tracks the position of the rehabilitation training ball in the image through the KCF algorithm (Kernel Correlation Filter), and the servo gimbal 12 drives the digital camera 10 to move so that the rehabilitation training ball remains within the shooting range of the digital camera 10; the movement of the user's arm is determined based on the position of the rehabilitation training ball.

[0220] In order to verify the effectiveness of the R-peak detection method of ECG data in step 3, this embodiment performs a performance evaluation on the MIT-BIH ECG database. The database contains ECG data of 48 users with arrhythmia, with a recording time of up to half an hour, and is the most widely used ECG data set. The sampling frequency of the ECG signal is 360Hz, the length of each record is 650,000 sampling points, and two cardiologists have annotated each signal in detail. This paper randomly selected the data of 10 users for verification. When the detection method can correctly identify the R peak marked by the expert, it is recorded as a true positive (True Positive, TP). According to the ANSI / AAMI-EC57:1998 standard, if the time difference between the automatic detection point and the marking point is less than 150ms, it is recorded as TP; when missed, it is recorded as a false negative (False Negative, FN), and when wrongly detected, it is recorded as a false positive (False Positive, FP).

[0221] Sensitivity ( Se) is defined as the proportion of correctly identified R peaks, that is, the probability that the algorithm can correctly detect the R peak among all true R peaks. Positive predictive value ( +P ) represents the proportion of all results identified as R peaks by the algorithm that are actually true R peaks. These two indicators can effectively evaluate the accuracy and reliability of the detection method, and are defined as follows:

[0222]

[0223] Table 4 gives the ECG signal detection results of the 10 subjects in detail.

[0224] Table 4 MIT-BIH database detection results

[0225]

[0226] It can be seen from Table 4 that the ECG signal feature extraction method based on bilateral slope can effectively extract the most critical feature point R peak of the ECG signal. The experimental results show that this method can show good results even in the arrhythmia data set, with a sensitivity of 99.22% and a positive predictive value of 99.83%. The accuracy is expressed as follows:

[0227]

[0228] This method is simple and efficient, with an average accuracy of 99.10%. It has good robustness while maintaining a low computational burden, making it suitable for accurate detection of key feature points in real-time systems.

Claims

1. An active grasping hand function training system based on multi-dimensional emotion recognition, characterized in that: The invention comprises a rehabilitation training ball, a core processing module, a training interaction module and a multi-source data acquisition module; the multi-source data acquisition module comprises an image emotion recognition module (8), an electrocardiogram acquisition module (6) and a photoelectric pulse wave sensor (5); the image emotion recognition module (8) is used to collect a user's facial image and identify the emotion type corresponding to the expression; the electrocardiogram acquisition module (6) is used to collect the user's electrocardiogram data; and the photoelectric pulse wave sensor (5) is used to collect the user's pulse wave data; The core processing module performs emotion recognition on pulse wave data and ECG data respectively, and integrates the emotion recognition results corresponding to facial expression, pulse wave and ECG to obtain the final comprehensive emotion recognition result. The rehabilitation training ball is used for the user to hold and detect the user's grip strength; The training interaction module is used to guide the user to perform hand grasping, rotation and movement training; the core processing module is used to perform hand grasping, rotation and movement training according to the comprehensive emotion recognition results. Adjust the difficulty of hand grasping, moving, and rotating training to prevent users from maintaining negative emotions; The core processing module performs emotion recognition on the ECG signal features extracted based on the ECG data and the pulse wave signal features extracted based on the pulse wave data through the random forest algorithm to obtain the ECG emotion type vector , pulse wave emotion type vector The image emotion recognition module (8) performs emotion recognition on facial images through the YOLOv5 algorithm to obtain the expression recognition emotion type vector ; The process of extracting ECG signal features from ECG data by the core processing module is as follows: (1) Based on the bilateral slope, the steepness of each sampling point in the ECG data is detected; (2) Construct a dynamic window based on RR interval to extract R peak. The specific steps are as follows: a. Set the steepness M of each sampling point in the starting segment of the ECG data steep Compare with the preset initial threshold T0 to obtain the suspected R peak; the steepness M of each sampling point after the starting segment of the ECG data is calculated. steep Greater than or equal to the dynamic threshold T a The sampling point is taken as the R peak; the dynamic threshold T a It is set according to the steepness of multiple R peaks before the current sampling point; b. Build dynamic window; Dynamic window width The expression is as follows: ； Among them, RRavr represents the average value of multiple RR intervals before the current sampling point; is the preset interval; the starting point of the dynamic window is the interval before the end point of the previous dynamic window. distance; c. When there are multiple R peaks in a dynamic window, only the local maximum value is retained to determine the position of the R peak in the dynamic window; (3) Extract P, Q, S, and T feature points based on the R peak position, and extract ECG signal features based on the PQRST feature points; The process of fusing the emotion recognition results corresponding to facial expressions, pulse waves, and electrocardiograms is as follows: (1) Concatenate ECG emotion type vectors , pulse wave emotion type vector , and obtain the feature vector ; Through the decision tree model, the feature vector Processing to obtain the preliminary prediction vector ; (2) Recognize the emotion type vector and the initial prediction vector Input the gradient boosting tree algorithm together to obtain the emotion prediction results ; (3) Emotion prediction results Perform regularization-assisted correction to obtain comprehensive emotion recognition results ; The rehabilitation training ball comprises a shell, a posture sensor (1) installed in the shell, and a pressure detection module; the shell is provided with a gripping groove corresponding to a human finger; the posture sensor (1) is used to detect the rotation amplitude of a user's hand; and the pressure detection module is used to detect the pressure exerted on the end of the gripping groove.

2. The active grasping hand function training system based on multi-dimensional emotion recognition according to claim 1, characterized in that: The pressure detection module adopts a spring pressing collection structure (15); the holding groove includes five finger accommodating grooves, and the ends of the five finger accommodating grooves are provided with mounting holes; a spring pressing collection structure (15) is fixed in each mounting hole; the spring pressing collection structure (15) includes a fixing seat, a pressing block, a spring and a pressure sensor (2); the fixing seat is fixed in the mounting hole, the pressing block is arranged on the outside of the fixing seat, and a pressure sensor (2) and a spring are provided between the pressing block and the fixing seat.

3. The active grasping hand function training system based on multi-dimensional emotion recognition according to claim 1, characterized in that: The core processing module measures the comprehensive emotion recognition results When the emotion is negative, the training difficulty is reduced; the method of reducing the training difficulty includes lowering the required threshold of grip force and reducing the threshold of the amplitude of hand movement and rotation; the core processing module is based on the measured comprehensive emotion recognition results The training difficulty is increased when the emotions are positive; ways to increase the training difficulty include increasing the required threshold of grip force and increasing the amplitude threshold of hand movement and rotation.

4. The active grasping hand function training system based on multi-dimensional emotion recognition according to claim 1, characterized in that: The process of calculating the steepness of the sampling point is: (1) Calculate the slopes on both sides of the sampling point , ; (2) For the slope With slope The steepness M is calculated at sampling points with opposite signs. steep as follows: ； in, , are the weight coefficients of the left and right slopes respectively.

5. The active grasping hand function training system based on multi-dimensional emotion recognition according to claim 1, characterized in that: The process of extracting pulse wave signal features from pulse wave data by the core processing module is as follows: Set a moving window in the pulse wave data; extract the local maximum value LocMax in the moving window i ; If the local maximum value LocMax i If the following conditions are met, the local maximum value LocMax in the moving window is determined i The corresponding position is taken as the peak characteristic point of the main wave; ； Among them, T b It represents the peak threshold of the main wave; ΔT represents the threshold allowable error; A trough detection window is constructed with each peak feature point as the termination point; the local minimum point of the amplitude in the trough detection window is taken as the trough feature point of the main wave; the width of the trough detection window is 1 / 10 of the distance between two adjacent peaks; multiple time domain features and frequency domain features are extracted as pulse wave signal features based on the peak feature points and trough feature points of the main wave.

6. The active grasping hand function training system based on multi-dimensional emotion recognition according to claim 1, characterized in that: The training interaction module comprises a touch screen (13), an audio player (14), a distance measurement module (11) and a training ball tracking module; the touch screen (13) is used to display an image that guides a user's hand to grasp, rotate and move; the audio player (14) is used to play background music; the distance measurement module (11) is used to measure the distance between the touch screen (13) and the user's body; the training ball tracking module comprises a steering gear pan-tilt (12) and a digital camera (10) mounted on the steering gear pan-tilt (12); the digital camera (10) is used to photograph a user's hand holding a rehabilitation training ball; the core processing module detects the position of the rehabilitation training ball in the image, and adjusts the direction of the digital camera (10) in real time through the steering gear pan-tilt (12), so that the digital camera (10) continuously photographs the position of the rehabilitation training ball; and the core processing module determines the movement amplitude of the user's hand according to the change in the position of the rehabilitation training ball.

Images