Emotion stress recognition method and system based on action shape fusion graph neural network

Abstract

The application discloses an emotion stress recognition method and system based on an action form fusion graph neural network, and belongs to the fields of image processing and machine learning. A double graph data structure is constructed, including a graph data taking human body key points as nodes and a graph data taking image frames as nodes, which are processed through T-GCN and PCSN respectively. The T-GCN model fuses a GCN layer and a Mamba encoder, strengthens the cohesion and coupling ability of emotion category representation through a graph partition strategy, and effectively captures the spatiotemporal feature relationship of human body actions. The PCSN adopts a three-branch parallel structure, including one-dimensional global average pooling, 3*3 convolution and 5*5 convolution, realizes efficient extraction of non-homogeneous features of the human body, the face and the hand, and establishes long-short range dependence and local cross-channel interaction relationship of global features. The system combines transfer learning and small sample learning, initializes the network by using a public data set, and then performs secondary training on a campus emotion stress small sample data set, so that the recognition precision is significantly improved.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of image processing, and particularly relates to an emotion stress recognition method and system based on a motion form fusion graph neural network. BACKGROUND

[0002] Emotion stress is a psychological stress state formed by an individual under the stimulation of external things; short-term emotion stress helps the individual to adapt to environmental changes by adjusting their own physiology or psychology; however, if the individual cannot reasonably relieve long-term emotion stress, it will evolve into physical and mental health problems and further have extensive and long-term effects. In the current situation of intensified information globalization, increasingly fierce social competition and great changes in the educational environment, physical and mental health has become an important cornerstone of personal growth, interpersonal relationships and career development. It is of utmost importance to pay attention to the development of students' physical and mental health; at present, a number of comprehensive measures have been taken to strengthen and improve students' mental health, so that the system of health education, monitoring and early warning, counseling services and intervention and disposal of students' mental health work is more perfect, and the pattern of students' mental health work coordinated by schools, families, society and relevant departments is more perfect. Among them, monitoring and early warning can diagnose students' emotion stress in a timely manner, prevent the formation of individual psychological problems, and provide ample preparation time for intervention and disposal, which is a key link in the system of students' mental health work.

[0003] According to the different monitoring target elements, emotion stress monitoring can be divided into physiological monitoring and non-physiological monitoring. Physiological monitoring requires wearing monitoring equipment and has professional knowledge reserve to determine emotion stress, and is not conducive to real-time emotion monitoring of school groups. The psychological health assessment scale in non-physiological monitoring may have fraudulent information. Emotion recognition relying on natural language processing technology may have some difficulties in capturing group audio, text and other information, which may lead to a lack of timeliness in emotion analysis. Relying on image processing technology, its process is easy to monitor groups, focus tracking and real-time capture of image characteristics, so that it has received extensive attention in the field of emotion recognition.

[0004] Currently, image-based methods for judging human emotions still have significant shortcomings. Firstly, they rely on a single data type; most existing methods only consider the representation of emotion by the morphology of a single body part, such as facial expressions, while neglecting the emotional information conveyed by other parts of the body, such as posture and body movements. Secondly, they fail to fully consider the spatiotemporal structure and complementary relationships of the movements of different body parts. The human body is an organic whole; the movements and postures of various parts are interconnected across different temporal and spatial dimensions, collectively expressing emotions. However, current methods fail to comprehensively capture and analyze these complex relationships, resulting in a lack of accurate criteria for judging the complex psychological processes of emotions in campus settings. This lack of comprehensive and holistic recognition capabilities leads to misunderstandings or neglect of the emotional states of people on campus. In a campus environment, the inability to promptly identify abnormal emotional states may miss the optimal time for intervention, increasing the risk to campus safety management. Summary of the Invention

[0005] The purpose of this invention is to overcome the problem of inaccurate judgment of people's emotions in campus scenarios, and to propose an emotional stress recognition method and system based on action-morphology fusion graph neural network.

[0006] To achieve the above objectives, the present invention adopts the following technical solution:

[0007] In a first aspect, the present invention provides a method for identifying emotional stress based on action-morphology fusion graph neural networks, comprising the following steps:

[0008] S1. Preprocess the video images of campus personnel, perform liveness detection using deep learning algorithms, and generate images of campus personnel.

[0009] S2. Perform human body, hand and face entity segmentation on images of people on campus, and use the feature extraction model to locate key points in the image based on the dimension of non-physiological feature data;

[0010] S3. Construct a dual graph data space structure and use the T-GCN model combined with a graph partitioning strategy to extract segmentation features for sentiment enhancement.

[0011] The dual-graph data space structure includes:

[0012] The graph data spatial structure uses key points of the human body as nodes, and the edge weights are dynamically allocated based on the spatial configuration partitioning strategy and the distance relationship between the joints and the root node.

[0013] A graph data space structure with image frames as nodes, where edge weights are assigned based on the similarity between image feature vectors;

[0014] S4, by the transfer learning method, the T-GCN model is first trained by using a public emotion dataset, and then is secondly trained and tested by using a small sample dataset of campus emotion pressure;

[0015] S5, personal information of the personnel is one-hot coded, a graph data node is constructed by using the image frame, non-homogeneous features of the human body, the hand and the face are extracted based on the PCSN, and the key point data and the personal information coding are fused to complete comprehensive identification of the emotion pressure, so that emotion pressure discrimination basis is obtained;

[0016] S6, the emotion stress segmentation features of the human body, the hand and the face are pooled and fused with the emotion pressure discrimination basis, and the emotion pressure category recognition result and the classification probability are output by the KAN network classifier.

[0017] Further, the T-GCN model comprises a GCN layer and a Mamba encoder, the GCN layer is used to capture the spatiotemporal features of the human body action, and the Mamba encoder is processed in parallel by the SSMs.

[0018] Further, the PCSN adopts a multi-branch parallel structure to collect image feature information in different scales of the field of view, including a one-dimensional global average pooling coding branch, a 3*3 convolution branch and a 5*5 convolution branch.

[0019] Further, the one-dimensional global average pooling coding branch comprises parallel routing for one-dimensional global average pooling in the horizontal and vertical directions of the feature vector, to obtain a horizontal one-dimensional global average pooling feature vector and a vertical global average pooling feature vector; after transposing the horizontal one-dimensional global average pooling feature vector, the vertical global average pooling feature vector is combined to obtain a global average pooling feature vector; after nonlinear transformation of the global average pooling feature vector, an activated feature vector is obtained, and maximum pooling and average pooling operations are performed on the activated feature vector;

[0020] The 3*3 convolution branch comprises using a convolution kernel number of 3 to process the input feature vector, and then performing BN normalization to obtain a normalized feature vector, and performing maximum pooling and average pooling operations on the normalized feature vector;

[0021] The 5*5 convolution branch comprises using a convolution kernel number of 5 to process the input feature vector, and then performing BN normalization to obtain a normalized feature vector, and performing maximum pooling and average pooling operations on the normalized feature vector;

[0022] The PCSN obtains parallel local cross spatial features by performing matrix point multiplication operation on the result vectors of the maximum pooling and the average pooling of the three-branch structure, and the parallel local cross spatial features are output by the KAN network layer;

[0023] The KAN network classifier and the KAN network of the KAN network layer of the PCSN both adopt a network structure of 3 grid intervals and 3 orders.

[0024] Further, the non-physiological feature data dimension is:

[0025] [N, C, T, V, M]

[0026] Wherein, N is the number of input videos of the same batch, C is the joint feature, T is the number of image frames in a single video, V is the number of key points, and M is the maximum number of person identifications in a single image frame;

[0027] The key points of the campus personnel image include human body key points, face key points, and hand key points;

[0028] The face key points include internal key points and external contour key points, and the internal key points include the positions and opening and closing degree states of eyebrows, eyes, mouth, and nose;

[0029] The hand key points include hand key points and background data;

[0030] The human body key points include human body key points, and the human body key points include human body degree of freedom joints, and the human body degree of freedom joints include neck, shoulder, elbow, wrist, waist, knee, and ankle;

[0031] The campus personnel information includes gender, age, major, and score;

[0032] The emotional stress categories include happiness, sadness, fear, surprise, anger, and jealousy.

[0033] Further, the preprocessing adopts Instruct-IPT and FSRCNN, and the FSRCNN adopts a transpose convolution layer, and the convolution kernel size of the transpose convolution layer includes 9x9, 81x81, and 27x27;

[0034] The living body detection adopts a cascade detector, and the width of the target recognition region is 1000 pixels, and the height is 1000 pixels;

[0035] The entity segmentation adopts a cascade model of a cascade detector and an image segmentation algorithm; the cascade model of the cascade detector includes a whole body feature cascade classifier, a hand feature cascade classifier, and a front face feature classifier;

[0036] In the segmentation mask of the image segmentation algorithm, the background region is marked as 0, and the foreground region is marked as 255; the cascade detector samples a Haar cascade detector in an open source library OpenCV; and the image segmentation algorithm adopts a GrabCut algorithm;

[0037] The minimum recognition area of the whole body human feature cascade classifier has a width of 1000 pixels and a height of 1000 pixels, the minimum recognition area of the hand feature cascade classifier has a width of 500 pixels and a height of 500 pixels, and the minimum recognition area of the front face feature classifier has a width of 500 pixels and a height of 500 pixels.

[0038] In a second aspect, the present application provides an emotion stress recognition system based on action morphology fusion graph neural network, comprising:

[0039] A living body detection module is configured to pre-process video images of campus personnel, perform living body detection through a deep learning algorithm, and generate campus personnel images.

[0040] A key point positioning module is configured to perform human body, hand and face entity segmentation on the campus personnel images, and position the image key points based on the dimensions of the non-physiological feature data using a feature extraction model.

[0041] A dual graph data construction module is configured to construct a dual graph data space structure, and extract emotion-enhanced segmentation features using a T-GCN model combined with a graph partitioning strategy. The dual graph data space structure includes: a graph data space structure with human body key points as nodes, edge weights based on a spatial configuration partitioning strategy, and dynamically allocated according to the distance relationship between joints and root nodes; a graph data space structure with image frames as nodes, and edge weights allocated based on the similarity between image feature vectors.

[0042] A T-GCN model training module is configured to use a transfer learning method, where the T-GCN model is first trained using a public emotion dataset, and then trained and tested again on a small sample dataset of campus emotion stress.

[0043] A non-homogeneous feature extraction module is configured to one-hot encode the personal information of the personnel, construct graph data nodes using image frames, extract non-homogeneous features of the human body, hands and face based on PCSN, and fuse the key point data and personal information encoding to complete comprehensive recognition of emotion stress and obtain emotion stress discrimination basis.

[0044] A classified emotion recognition result module is configured to pool and fuse the emotion-enhanced segmentation features of the human body, hands and face with the emotion stress discrimination basis, and output emotion stress category recognition results and classification probabilities through a KAN network classifier.

[0045] Further, the system further comprises:

[0046] A perception control module is configured to capture campus personnel image information and time information accompanying the campus personnel image information.

[0047] A user module is configured to participate in system management, use and data sharing, and interaction between users.

[0048] a service providing module for interacting through a service interface, the user module including a personal information management service, an image data management service, a mental health information management service, and an emotional stress early warning and push service;

[0049] a resource exchange module for data and service interaction with an external environment for resource exchange;

[0050] an operation and management control module for monitoring system operation and service and device and network resource operation.

[0051] In a third aspect, the present application provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the emotion stress recognition method based on the action form fusion graph neural network.

[0052] In a fourth aspect, the present application provides a computer readable storage medium, which stores a computer program, wherein the computer program is executed by a processor to implement the emotion stress recognition method based on the action form fusion graph neural network.

[0053] Compared with the prior art, the present application has the following beneficial technical effects:

[0054] The emotion stress recognition method based on the action form fusion graph neural network, the emotion stress recognition method aggregates human body forms such as facial actions, human postures, gesture actions, and constructs a graph data taking human body key points and image frames as graph nodes, to serve as a judgment method, the graph data constructed by the human body key points adopts a graph partition strategy and T-GCN processing, which strengthens the cohesion and coupling ability of emotion category representation, the graph data constructed by the image frames satisfies the non-homogeneous feature extraction demand of the human body, the hand and the face through PCSN (Parallel Cross Spatial Multi Scale Feature Extraction Network, parallel cross spatial multi-scale feature extraction network), and long-range dependence and local cross-channel interaction relationship of global feature information are established; the method not only constructs the internal correlation of the multi-level spatial domain cross fusion of the human body form action, but also improves the influence of the time domain change of the multi-level spatial form action feature on the emotion stress, and further enriches the emotion stress discrimination basis, and improves the accuracy and reliability of the result. The multi-dimensional joint reasoning of the human posture, the local limb action and the micro-expression is realized for the first time, the model training data amount demand is reduced through the small sample migration learning mechanism, and the campus scene landing feasibility is stronger. The system combines migration learning and small sample learning, initializes the network by using a public data set, and then performs secondary training on a small sample data set of campus emotion stress, so that the recognition precision is improved significantly.

[0055] Further, the method breaks through the limitation of traditional emotion monitoring based on single physiological characteristics, comprehensively integrates body, face and gesture information, and effectively models the spatio-temporal structure of action form. By combining small sample learning with transfer learning, the application performance of the system in the campus scene is significantly improved, and the system has strong real-time monitoring capability and high recognition accuracy, and has a wide practical application prospect in the fields of education, health and safety management. The present application provides effective technical support for emotion stress monitoring and intervention, and has unique advantages in real-time monitoring and emotion early warning of groups.

[0056] The emotion stress recognition system based on the action form fusion graph neural network improves the data and business value coordinated sharing ability; the image perception collection equipment such as campus and classroom cameras is used to realize group perception, key tracking and real-time capture of campus personnel form action image information and fusion of personal information, the MGCN is combined with the PCSN to realize the ability of associating and memorizing and perceiving and analyzing personnel emotion stress with action form as reference, improve the credibility of emotion stress measurement and analysis results; and the personal exercise, education, diet, sleep and environment selection activities can be recommended according to the emotion stress results to adjust and maintain positive energy emotion, especially for the personnel with abnormal emotion stress, early intervention guidance and strengthened traceability monitoring can be provided, through real-time monitoring of the emotion stress of the campus personnel, the abnormal personnel can be helped to recover quickly and the local influence can be reduced. BRIEF DESCRIPTION OF DRAWINGS

[0057] The drawings described herein are for the purpose of explanation and are not intended to limit the scope of the present disclosure in any way. Additionally, the shapes and proportions of the components in the drawings are merely illustrative and are used to facilitate the understanding of the present disclosure, and are not specifically limited to the shapes and proportions of the components. In the drawings:

[0058] Figure 1 The flow chart of the emotion stress recognition method based on the action form fusion graph neural network of the present application.

[0059] Figure 2 The structure diagram of the emotion stress recognition system based on the action form fusion graph neural network of the present application.

[0060] Figure 3 The electronic device diagram of the emotion stress recognition method based on the action form fusion graph neural network of the present application.

[0061] Figure 4 The emotion stress recognition model network framework diagram based on the action form feature fusion graph convolution network in the embodiment.

[0062] Figure 5 The schematic diagram of the parallel cross spatial multi-scale feature extraction network.

[0063] Figure 6 The schematic diagram of the spatio-temporal graph convolutional neural network model.

[0064] Figure 7 The campus emotion Internet of Things monitoring system architecture of the emotion recognition model network based on the action form feature fusion graph convolution network in the embodiment. DETAILED DESCRIPTION

[0065] In order for those skilled in the art to better understand the technical scheme of the present application, the technical scheme in the embodiments of the present application will be described clearly and completely below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor should belong to the scope of protection of the present application.

[0066] Embodiment one

[0067] Referring to Figure 1 A kind of emotion stress recognition method based on action form fusion graph neural network, comprising the following steps:

[0068] The video image of campus personnel is preprocessed, live body detection is carried out by deep learning algorithm, and campus personnel image is generated;The human body, hand and face entity segmentation is carried out to the campus personnel image, and the image key point is positioned based on the dimension of non-physiological feature data using feature extraction model;Dual graph data space structure is constructed, and T-GCN model is used to extract emotion reinforced segmentation features combined with graph partition strategy;The dual graph data space structure includes: the graph data space structure with human body key points as nodes, the edge weight is based on space configuration partition strategy, and is dynamically distributed according to the distance relationship between joints and root nodes;The graph data space structure with image frame as node, the edge weight is based on the similarity between image feature vectors;Through the transfer learning method, T-GCN model is first trained using public emotion data set, and then secondary training and testing are carried out on the small sample data set of campus emotion stress;The personal information of personnel is one-hot encoded, the graph data node is constructed using image frame, the non-homogeneous features of human body, hand and face are extracted based on PCSN, and the key point data and personal information code are fused to complete the comprehensive identification of emotion stress, to obtain the emotion stress discrimination basis;The human body, hand and face emotion reinforced segmentation features and emotion stress discrimination basis are pooled and fused, and the emotion stress category recognition result and classification probability are output through KAN network classifier.

[0069] In this embodiment, the graph data including key point graph data and image frame graph data is used as the data basis for emotion recognition, wherein the image frame data is used as global information features including personal overall features, interaction features with personnel, etc., and the key point graph data is used as local features including facial features, human body shape, and hand action fine features. A multi-layer perception graph data processing network structure model is constructed, which improves the accuracy of emotion representation and the discrimination reference basis.

[0070] Embodiment two

[0071] Referring to Figure 2 A kind of emotion stress recognition system based on action form fusion graph neural network, comprising:

[0072] A living body detection module is configured to preprocess a video image of a campus staff, perform living body detection through a deep learning algorithm, and generate a campus staff image;

[0073] A key point positioning module is configured to perform human body, hand, and face entity segmentation on the campus staff image, and position key points of the image based on dimensions of non-physiological feature data by using a feature extraction model;

[0074] A dual graph data construction module is configured to construct a dual graph data space structure, and extract emotion-enhanced segmentation features by using a T-GCN model combined with a graph partitioning strategy; the dual graph data space structure includes: a graph data space structure with human body key points as nodes, edge weights based on a spatial configuration partitioning strategy, and dynamically allocated according to distance relationships between joints and root nodes; a graph data space structure with image frames as nodes, and edge weights allocated based on similarities between image feature vectors;

[0075] A T-GCN model training module is configured to train the T-GCN model by using a public emotion dataset first, and then perform secondary training and testing by using a small sample dataset of campus emotion stress;

[0076] A non-homogeneous feature extraction module is configured to one-hot encode personal information of the staff, construct graph data nodes by using image frames, extract non-homogeneous features of human bodies, hands, and faces based on a PCSN, and fuse key point data and personal information encoding to complete comprehensive identification of emotion stress and obtain emotion stress discrimination basis;

[0077] A classified emotion recognition result module is configured to pool and fuse emotion-enhanced segmentation features of human bodies, hands, and faces and emotion stress discrimination basis, and output emotion stress category recognition results and classification probabilities through a KAN network classifier.

[0078] Referring to Figure 7 , a perception control module is configured to capture campus staff image information by using high-definition cameras in the campus, classrooms, and gates, and the campus staff image information is attached with time information;

[0079] A user module is configured to associate with involved institutions and personnel, participate in system management, use, and data sharing, and interaction between users, and the involved institutions and personnel include teachers, student management departments, psychological health rooms, and individual teachers and students;

[0080] A service providing module is configured to interact through a service interface, and the user module includes personal information management services, image data management services, psychological health information management services, and emotion stress early warning and push services;

[0081] A resource exchange module is configured to exchange data and services with an external environment, and the interface of the resource exchange includes a mental health guidance interface, a psychosocial research interface, an online education interface, and a game entertainment service interface.

[0082] An operation management and control module is configured to monitor system operation services and equipment and network resource operation.

[0083] An emotional stress recognition system is established based on a six-domain model to improve the ability of data and business value coordination and sharing. The six-domain model is an important concept in the field of Internet of Things. By systematically analyzing the application-related elements of Internet of Things industry, six domains are set to build a collaborative ecological system of Internet of Things. The group perception, key tracking and real-time capture of school personnel motion image information are realized by using image perception collection devices such as campus and classroom cameras, and personal information is fused. A spatio-temporal graph convolution network algorithm model is constructed to realize the ability of associating and memorizing to perceive and analyze the emotional stress of personnel by taking the motion form as a reference, and improve the credibility of the emotional stress measurement and analysis results. In addition, individual exercise, education, diet, sleep and environment selection activities can be recommended according to the emotional stress results to adjust and maintain positive emotions, especially for personnel with abnormal emotional stress, early intervention guidance and enhanced traceability monitoring can be provided.

[0084] Embodiment three

[0085] Referring to Figure 3 An electronic device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor implements the emotional stress recognition method based on the action form fusion graph neural network when executing the computer program.

[0086] Embodiment four

[0087] A computer readable storage medium stores a computer program, and the computer program implements the emotional stress recognition method based on the action form fusion graph neural network when executed by a processor.

[0088] Embodiment five

[0089] The embodiment provides an emotional stress recognition method based on an action form fusion graph neural network, referring to Figure 1 , the specific workflow is as follows:

[0090] The first step is image capture and data cleaning. Personnel image data captured by campus and classroom cameras is denoised, deblurred, derained, defogged, and desnowed using Instruct-IPT (Instruct-Image Processing Transformer). Instruct-IPT is an image processing technology that solves multiple image restoration tasks such as denoising, deblurring, deraining, defogging, and desnowing through weight modulation. FSRCNN (Fast Super-Resolution Convolutional Neural Network) with 9x9, 81x81, and 27x27 transpose convolution layers is used for video image upsampling to generate high-resolution images. Due to issues such as no target, target blur, too small or too large occlusion, excessive angle offset, and overly dark night shots during image capture, the information content is low during model training, which cannot update the model parameters and have a significant impact, or the model overfits due to excessive focus on these features. Haar cascade detector in OpenCV (Open Source Computer Vision Library) is used for liveness detection, and the target recognition area size is set to (1000, 1000) to achieve an automated data cleaning process. (1000, 1000) represents a width of 1000 pixels and a height of 1000 pixels.

[0091] Second step, human body part instance segmentation. The face, human body, and hand entity segmentation of the image is carried out by using Haar cascade detector in OpenCV and GrabCut algorithm (an image segmentation algorithm), and the cascade models are selected respectively: haarcascade_fullbody.xml (a file containing a Haar feature cascade classifier for detecting a full body in an image), haarcascade_hand.xml (a file containing a Haar feature cascade classifier for detecting a hand), and haarcascade_frontalface_default.xml (a file containing a Haar feature cascade classifier for detecting a frontal face), and the minimum recognition area pixels in the three detectMultiScale (an object detection function) are (1000, 1000), (500, 500), and (500, 500), and the background area in the segmentation mask is marked as 0 and the foreground area is marked as 255. (1000, 1000) represents a width of 1000 pixels and a height of 1000 pixels, and (500, 500) represents a width of 500 pixels and a height of 500 pixels.

[0092] Third step, key point positioning and identification. In the process of human emotion change, the spatio-temporal change of non-physiological characteristics is an important manifestation of emotion, such as facial expression, body posture, hand movement. One method to identify the corresponding category of non-physiological characteristics is to construct the key point positioning and identification of human body parts. At the same time, the key points in this example are also used as nodes in the graph data in the emotion stress recognition. This example uses Heatmaps (heat map) to construct key points. For facial key point detection, OpenCV+Dlib library (Dlib is a machine learning algorithm and tool library, known for its efficient and accurate face detection and 68-point feature point detector) is used to perform 68-point calibration using shape_predictor_68_face_landmarks.dat (a face feature point detection model). The internal key points are 51, which can represent the position and opening degree of eyebrows, eyes, mouth, and nose, and the external contour key points are 17. For human and hand key point detection, OpenPose (a human pose estimation framework based on deep learning) is used to extract features from the input image and estimate the joint position of the body. body_pose_deploy.prototxt (a file that defines the neural network architecture of the body pose estimation model) and body_pose_model.pth (a file that defines the neural network weights of the body pose estimation model), hand_pose_deploy.prototxt (a file that defines the neural network architecture of the hand pose estimation model) and hand_pose_model.pth (a file that defines the neural network weights of the hand pose estimation model) are used to extract key points. There are 22 hand points, 21 hand key nodes, and 22 background data points. There are 18 human key points, corresponding to the degrees of freedom joints of the human body, such as neck, shoulder, elbow, wrist, waist, knee, ankle, etc.

[0093] The non-physiological characteristic data dimension is [N, C, T, V, M], where N is the number of input videos in a batch; C is the joint feature, i.e. the horizontal and vertical positions of the key points and the ACC (Accuracy, accuracy) confidence; T is the number of image frames in a single video; V is the number of key points; M is the maximum number of characters identified in a single image.

[0094] Fourth step, graph data space structure establishment. This example constructs two kinds of graph data respectively for Figure 4Part 1, 2, one is to take the human key points as nodes, and the edges are the connections between the nodes. The weight distribution strategy of the edge adopts spatial configuration partition, that is, the distance between the root node and the center of gravity of different parts of the human body is taken as the reference distance, and then the relationship between the distance between other adjacent nodes and the root node and the reference distance is judged. When it is greater than the reference distance, the weight is 3, when it is less than the reference distance, the weight is 0, and when it is equal to the reference, the weight is 1. The farther the joint distance from the center of gravity, the greater the action, and the greater the influence on the behavior characteristics should be considered more. The second type is to take the image frame as the graph node, and the edge is the connection between different frames. The weight is the similarity between the image feature vectors.

[0095] Figure 4 The time dimension represents the time dimension, the spatial dimension represents the spatial dimension, the full body is the human posture, the hand is the hand, the face is the face, and the POOL is the pooling.

[0096] The fifth step is to build an emotional stress small sample data set. The actions that may be generated by the human form under different emotions can be referred to the following table. The face, body, hand joint and emotion category are labeled by using the target detection labeling tool LabelImg (open source image labeling tool) and associated with personal information.

[0097]

[0098] The sixth step is to obtain and encode personal information. In the campus mental health statistics, emotional stress is often highly correlated with gender, age, major, and academic or teaching and research performance. According to the mental health statistics, men have greater emotional fluctuations during extreme events and are more inclined to express emotions through physical actions. When angry or excited, men may exhibit more obvious body language, such as clenching their fists and stomping their feet.

[0099] This example obtains personnel information through OpenCV face recognition linkage. The information including gender, age, major, and performance is one-hot encoded into a one-dimensional feature vector, with an age interval of 2. The academic (research) average performance is divided into three categories: failing, passing, and excellent;

[0100] For example: [Zhang San, male, 20, software engineering, average score 82] corresponds to the encoded information [0101000101],

[0101] From left to right, the 0-1 position corresponds to gender, the 2-4 position corresponds to age, the 5-7 position corresponds to major, and the 8-9 position corresponds to academic performance.

[0102] Step 7: Normalization of Human Keypoint Features. In both time and space, the relative positions of keypoints in images vary significantly depending on the emotional state and the actions performed. Furthermore, the positions of keypoints in different batches and image frames follow a random distribution pattern according to statistical principles. Normalizing the keypoint features using Batch Normalization (BN) will not cause drastic fluctuations in prediction accuracy, but it can accelerate model convergence.

[0103] The eighth step involves constructing a PCSN using a KAN network to meet the requirements for extracting non-homogeneous features from the face, body, and hands.

[0104] 8.1, such as Figure 5 As shown, parallel routing is used to perform one-dimensional global average pooling in the horizontal and vertical directions on the input feature vector X[C, H, W], where C is the number of channels, H and W are the spatial dimensions of the input, H is the height, and W is the width. The encoding expressions for the one-dimensional global average pooling in the horizontal and vertical directions are as follows:

[0105]

[0106] M is the magnitude of the output vector of the X feature vector after one-dimensional average pooling, and M(H) is the output vector calculated in the vertical direction. It is a whole, which is the vector data of the i-th row in the horizontal direction, and its range is from 0 to W, that is, scanning the entire width of the image.

[0107]

[0108] M(W) is the output vector calculated in the vertical direction. It takes the vector data of the i-th column in the vertical direction, and its range is from 0 to H, which means scanning the entire height of the image.

[0109] One-dimensional global average pooling in the horizontal and vertical directions is the output feature vector after performing one-dimensional global pooling on the image. It mainly obtains the shallow global features of the image in order to fuse with the deep semantic features in the later stage and more accurately represent emotional stress.

[0110] Figure 5 In this diagram, Input represents the input, X Avg Pool represents one-dimensional global average pooling in the parallel direction, Y Avg Pool represents one-dimensional global average pooling in the vertical direction, Concat represents a connection, Max Pool represents max pooling, Avg Pool represents average pooling, matmul represents the matmul function (a function that performs matrix multiplication), KANConv represents KAN network convolution, BN represents batch normalization, and Output represents the output.

[0111] 8.2, transpose the one-dimensional global average pooling encoding vector in the horizontal direction, and merge it with the vertical direction global average pooling vector to obtain , ]T is vector transpose, where one-dimensional pooling shares convolution kernel parameters, thereby associating local cross-influence and global similarity;

[0112] 8.3, use ELU non-saturated activation function (Exponential Linear Unit is a non-saturated activation function) to process the global average pooling encoding vector and the feature vector X, accurately save the spatial structure information of the input vector in the channel, and prevent the problem of falling into dead neurons, avoid the problem of gradient explosion or disappearance.

[0113] 8.4, the above vector maximum pooling and average pooling respectively retain the explicit features and local features.

[0114] 8.5, use convolution kernel number 3, 5 respectively to process the input feature vector X, then perform BN normalization, and then perform the same maximum pooling and average pooling operation, and ELU as the activation function to perform feature merging processing, and constitute a three-branch structure with the one-dimensional global average pooling encoding to collect image feature information in different scales of visual field.

[0115] 8.6, use matmul function matrix dot product operation to operate the result vectors of the three-branch structure maximum pooling and average pooling, realize parallel local cross spatial feature extraction, and establish long and short range dependence and local cross channel interaction relationship of global feature information.

[0116] 8.7, use 3 grid intervals and 3-order KAN network to improve the network nonlinear fitting as PCSN network.

[0117] In the ninth step, different entity segmentation objects are processed by PCSN respectively. Since the face, human body and hand have different biological structures, different attention feature extraction network structures are needed. Human body feature extraction mainly involves joint position and body posture key point detection, which requires a large area feature extraction network to accurately mark the positions of each key point. Hand feature extraction focuses on finger posture, palm posture, joint completeness, etc. In addition, compared with the size of the body, the hand is small and has great flexibility, so a fine-grained feature extraction network is needed. Especially for face feature extraction, the face contour, eyes, nose, mouth and their relationships can best represent the emotional characteristics of personnel. Compared with the human body and hand, the face has rich texture feature information, so more attention is paid to different level feature vector extraction. In the model training process, the parameters of PCSN are randomly initialized to expand the feature extraction capability of the face, human body and hand. The internal correlation of the multi-level spatial domain cross-fusion of PCSN, the fusion of KAN network (Kolmogorov-Arnold network, a neural network architecture) mechanism, enhances the nonlinear fitting ability of the model and improves the accuracy of the model.

[0118] In the tenth step, the spatio-temporal graph convolutional neural network model. Mamba can not only process time domain sequence data in parallel through SSMs (Selective State Space Models), but also has fewer parameter quantities than the Transformer (a deep learning model based on attention mechanism) and a hardware-aware algorithm optimization process, which is beneficial to further improve the real-time performance of the system monitoring, optimize the network structure and improve the real-time performance of the system response. Based on Mamba (a simplified state space model architecture) and GCN (Graph Convolutional Network), a spatio-temporal graph convolutional neural network is constructed, which takes into account the spatio-temporal structure and complementary relationship of morphological actions, as well as the hardware-aware computing parameter optimization mechanism of Mamba, which can improve the real-time performance of the system response and realize real-time monitoring. By taking advantage of the complementary advantages of MGCN (Mamba Graph Convolutional Network, spatio-temporal graph convolutional neural network model) and GCN model, the understanding of emotional image video is realized. As shown in Figure 6 , the model input X is a T-frame human morphological action video, Figure 6 X0, X T-1 , X T are the 0th, T-1th and Tth frames of the image respectively. GCN is used to capture the spatial feature relationship between the nodes in the two types of graph data constructed, and Mamba can capture the global feature relationship in the time dimension over a long distance.

[0119] Figure 6 The Mamba encoder in the MGCN model is an input to the Mamba encoder. By using the human body shape and action feature vector output by the GCN in the time domain as the input of the Mamba encoder, the model can simultaneously learn spatiotemporal information. Finally, the feature vector Y output by the MGCN model has a comprehensive expressive ability for emotional stress recognition.

[0120] In this example, the MGCN model has 4 GCN layers. Increasing the number of layers allows for the aggregation of multi-hop node features, obtaining higher-level semantic information, and improving the model's recognition accuracy.

[0121] Step 11: Categorize emotional stress. For example... Figure 4 As shown, the results of the two types of graph data processing are finally fused to achieve deep semantic representation of multi-level spatial and temporal morphological action features. After pooling and processing by a KAN network classifier, the emotional stress category and classification probability are output. The KAN network classifier uses a network structure with 3 grid intervals and 3rd order. By introducing the KAN network, a dynamic activation function is used in the classifier, which enhances the model's ability to extract nonlinear features and further improves the model's discrimination accuracy.

[0122] Step 12: Model training.

[0123] 1) The L-Softmax loss (large margin Softmax loss function) is used as the loss function in the emotion recognition training process. Its feature space is more compact, which enhances the saliency between emotion stress categories. The L-Softmax loss increases the difficulty of classification by adding angular margins to the Softmax (normalized exponential function), thereby improving the generalization ability of the model.

[0124] 2) Transfer learning was employed, using publicly available datasets Fer2013 (Facial Expression RecognitionChallenge 2013), E-Gait (Body Posture Emotion Dataset), and SMG (Microgesture Dataset) for training only. Figure 4 The dashed box 1 portion of the MGCN network model is now disconnected. Figure 4 The network is shown in the dashed box 2. The hyperparameter settings include a learning rate of 0.0001, a batch size of 100, an iteration count of 10000, and the optimizer is Adam (Adaptive Moment Estimation). Adam optimizer is an optimization algorithm based on gradient descent, which combines momentum and adaptive learning rate methods and is widely used in the training of deep learning models.

[0125] 3) Load the model parameters obtained from step 2) into... Figure 4The virtual box 1 network structure in the model network structure, and then the small sample data of emotional stress in this example is divided into 7:2:1 as a training set, a validation set and a test set, and the model is saved for system testing and deployment after training is completed.

[0126] The eighth step is system construction. As shown in the figure, the emotional stress monitoring system is constructed by using the six-domain model of the Internet of Things. The mode constructs the system according to resources and service characteristics, effectively avoids the repeated deployment of software and hardware devices, systems and the like, and improves the data and business value coordination and sharing capability. Figure 7

[0127] The system in this example includes a perception control module, a user module, a service providing module, a resource exchange module and an operation and management control module.

[0128] The perception control module is a device entity set, which transmits and interacts with the network environment with real world entity resource information. This example includes capturing campus personnel image information through high-definition cameras on campus, gates and the like, with time information attached. The service providing module is an entity set of system applications and services, which is associated with other domains for control and interacts through service interfaces. This example includes personal information management services, image data management services, mental health information management services, and emotional stress early warning and push services. The user domain is associated with the institutions and personnel involved in the system, participates in system management, use and data sharing, and interaction between users. This example includes teachers, student management offices, mental health rooms and individual teachers and students. The resource exchange domain is to interact with the external environment for data and service, establish business relationships, and promote each other's information and service capability. This example includes resource exchange through business interfaces such as mental health guidance, psychological and social research, online education and game entertainment. The operation and management control module is an entity set to ensure the normal operation of the overall system. This example covers pressure monitoring system operation and maintenance services and devices, network resource operation and maintenance.

[0129] The interaction relationship between each module in this example includes the following: the perception control module collects data through high-definition cameras, mobile phones, computers and other devices with image shooting functions, uploads it to the server of the deployed service providing module for image storage, processing and identification, and can push the results to the campus security office, the mental health room or specific personnel. The mental health room can also monitor the state of key personnel in real time through the emotional stress monitoring service for early warning or intervention treatment. The resource exchange domain provides services to the outside through business interfaces such as mental health guidance, psychological and social research, online education and game entertainment, and on the other hand, obtains external service experience and data, uses the reinforcement learning mechanism to improve the accuracy of models and systems in the service providing domain, and iteratively optimizes service quality. The operation and management control domain interacts with the perception control domain and the service providing domain to monitor software and hardware, network devices in the system to ensure the normal operation of the entire system.​

[0130] Those skilled in the art will appreciate that embodiments of the present application can be readily used as a method, apparatus such as a system, or computer program product. Accordingly, the present application can take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present application can take the form of a computer program product on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage etc.) embodying computer readable program code.

[0131] The present application is described in relation to flow diagrams and / or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the present application. It is to be understood that each block of the flow diagrams and / or block diagrams, and combinations of blocks in the flow diagrams and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general purpose computer, special purpose computer, embedded processor or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flow diagrams and / or block diagrams block or blocks. Figure 1 one or more functions specified in one or more of the flow diagram or block diagram block or blocks. Figure 1 means for performing each of the functions specified in the flow diagram or block diagram block or blocks.

[0132] These computer program instructions can also be stored in a computer- readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instructions which implement the flow diagram or block diagram block or blocks. Figure 1 one or more functions specified in one or more of the flow diagram or block diagram block or blocks. Figure 1 means for performing each of the functions specified in the flow diagram or block diagram block or blocks.

[0133] These computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the flow diagram or block diagram block or blocks. Figure 1 one or more functions specified in one or more of the flow diagram or block diagram block or blocks. Figure 1 means for performing each of the functions specified in the flow diagram or block diagram block or blocks. Finally, it should be noted that the above-mentioned embodiments illustrate rather than limit the application, since the scope of the application is determined by the appended claims.

Claims

1. A method for identifying emotional stress based on action-morphology fusion graph neural networks, characterized in that, Includes the following steps: S1. Preprocess the video images of campus personnel, perform liveness detection using a deep learning algorithm, and generate images of campus personnel; the liveness detection uses a cascaded detector; S2. Perform human body, hand and face entity segmentation on images of people on campus, and use the feature extraction model to locate key points in the image based on the dimension of non-physiological feature data; The entity segmentation employs a cascaded model of a cascaded detector and an image segmentation algorithm; the cascaded model of the cascaded detector includes a cascaded classifier of whole-body human features, a cascaded classifier of hand features, and a frontal face feature classifier; The key points of the images of people on campus include human body key points, facial key points, and hand key points; Key points of the human body include the joints with degrees of freedom; S3. Construct a dual graph data space structure and use the T-GCN model combined with a graph partitioning strategy to extract segmentation features for sentiment enhancement. The T-GCN model includes a GCN layer and a Mamba encoder. The GCN layer is used to capture the spatiotemporal features of human movements, and the Mamba encoder processes the sequence data in parallel using SSMs. The dual-graph data space structure includes: The graph data space structure uses key points of the human body as nodes. The edge weights are based on the spatial configuration partitioning strategy. The distance between the root node and the center of gravity of different parts of the human body is used as the reference distance. Then, the relationship between the distance between other adjacent nodes and the root node and the reference distance is judged, and the weights are dynamically allocated according to the distance relationship between the joint and the root node. The graph data space structure uses image frames as nodes, with edges connecting different frames, and edge weights assigned based on the similarity between image feature vectors. S4. Using the transfer learning method, the T-GCN model is first trained using a public sentiment dataset, and then trained and tested a second time using a small sample dataset of campus emotional stress. S5. Perform one-hot encoding on the personal information of personnel, construct graph data nodes using image frames, extract non-homogeneous features of the human body, hands, and face based on PCSN, and integrate key point data with personal information encoding to complete the comprehensive identification of emotional stress and obtain the basis for judging emotional stress. The PCSN uses a multi-branch parallel structure to collect image feature information within different scale fields of view, including a one-dimensional global average pooling coding branch, a 3×3 convolution branch and a 5×5 convolution branch. The PCSN obtains parallel local cross-space features by using the result vectors of max pooling and average pooling in the three-branch structure of matrix dot product operation, constructs the long and short-range dependencies and local cross-channel interaction relationships of global feature information, and outputs the parallel local cross-space features through the KAN network layer. S6. The emotional enhancement segmentation features of the human body, hands, and face are pooled and fused with the emotional stress discrimination criteria. The emotional stress category recognition result and classification probability are output through the KAN network classifier.

2. The emotional stress recognition method based on action-morphology fusion graph neural network according to claim 1, characterized in that, The one-dimensional global average pooling encoding branch includes performing parallel routing on the feature vectors for parallel and vertical one-dimensional global average pooling, resulting in horizontal one-dimensional global average pooling feature vectors and vertical one-dimensional global average pooling feature vectors; transposing the horizontal one-dimensional global average pooling feature vectors and merging them with the vertical one-dimensional global average pooling feature vectors to obtain a global average pooling feature vector; performing a nonlinear transformation on the global average pooling feature vectors to obtain activated feature vectors; and performing max pooling and average pooling operations on the activated feature vectors. The 3×3 convolution branch includes processing the input feature vector with a convolution kernel of 3, then normalizing it with BN to obtain a normalized feature vector, and performing max pooling and average pooling operations on the normalized feature vector. The 5×5 convolution branch includes processing the input feature vector with a convolution kernel of 5, then normalizing it with BN to obtain a normalized feature vector, and performing max pooling and average pooling operations on the normalized feature vector. Both the KAN network classifier and the KAN network layer of the PCSN adopt a network structure with 3 grid intervals and 3rd order.

3. The emotional stress recognition method based on action-morphology fusion graph neural network according to claim 1, characterized in that, The preprocessing uses Instruct-IPT and FSRCNN. The FSRCNN uses transposed convolutional layers, and the kernel sizes of the transposed convolutional layers include 9×9, 81×81 and 27×27. The target recognition region has a width of 1000 pixels and a height of 1000 pixels; in the segmentation mask of the image segmentation algorithm, the background region is marked as 0 and the foreground region is marked as 255; the cascade detector samples the Haar cascade detector in the open-source library OpenCV; and the image segmentation algorithm adopts the GrabCut algorithm. The minimum recognition area of ​​the whole-body feature cascade classifier is 1000 pixels wide and 1000 pixels high, the minimum recognition area of ​​the hand feature cascade classifier is 500 pixels wide and 500 pixels high, and the minimum recognition area of ​​the frontal face feature classifier is 500 pixels wide and 500 pixels high.

4. An emotional stress recognition system based on action-morphology fusion graph neural network, using the emotional stress recognition method based on action-morphology fusion graph neural network as described in any one of claims 1-3, characterized in that, include: The liveness detection module is used to preprocess video images of people on campus, and to perform liveness detection using deep learning algorithms to generate images of people on campus. The key point localization module is used to segment human body, hand and face entities in images of people on campus, and to locate key points in the image based on the dimension of non-physiological feature data using a feature extraction model. A dual graph data construction module is used to construct a dual graph data spatial structure and to extract segmentation features for emotion enhancement using a T-GCN model combined with a graph partitioning strategy. The dual graph data spatial structure includes: a graph data spatial structure with human keypoints as nodes, where edge weights are dynamically allocated based on the spatial configuration partitioning strategy and the distance relationship between joints and root nodes; and a graph data spatial structure with image frames as nodes, where edge weights are allocated based on the similarity between image feature vectors. The T-GCN model training module is used to train the T-GCN model first on a public sentiment dataset and then conduct secondary training and testing on a small sample dataset of campus emotional stress using the transfer learning method. The non-homogeneous feature extraction module is used to perform one-hot encoding of personal information, construct graph data nodes using image frames, extract non-homogeneous features of the human body, hands, and face based on PCSN, and integrate key point data with personal information encoding to complete the comprehensive identification of emotional stress and obtain the basis for emotional stress judgment. The emotion recognition module is used to pool and fuse the enhanced segmentation features of human body, hand, and facial emotions with the criteria for judging emotional stress, and output the emotional stress category recognition results and classification probabilities through a KAN network classifier.

5. The emotional stress recognition system based on action-morphology fusion graph neural network according to claim 4, characterized in that, Also includes: The perception and control module is used to capture image information of personnel on campus, as well as the time information accompanying the image information; The user module participates in system management, usage, data sharing, and interactions between users; The service provision module is used for interaction through service interfaces, and the user module includes personal information management services, image data management services, mental health information management services, and emotional stress early warning and push services; The resource exchange module is used to exchange resources with the external environment by interacting with data and services. The operation and maintenance management module is used to monitor system operation and maintenance services and the operation and maintenance of equipment and network resources.

6. An electronic device, characterized in that, The device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, implements the emotional stress recognition method based on action morphology fusion graph neural network as described in any one of claims 1-3.

7. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the emotional stress recognition method based on action-morphology fusion graph neural network as described in any one of claims 1-3.

Images