Method for recognizing emotion in outdoor environment based on multi-feature and attention mechanism

Abstract

The application discloses a multi-feature outdoor environment emotion recognition method based on an attention mechanism, and comprises the following steps: acquiring physiological data of a measured person and images of an outdoor environment; adopting a Relief nonlinear feature extraction algorithm to extract physiological data features of the measured person, and adopting a ConvLSTM recognition model to obtain an emotion grade label corresponding to any frame image group; performing face recognition and positioning by using an MTCNN network; extracting texture features of the images of any frame image group by using a Resnet module, and converting the images of any frame image group into one-dimensional vectors to obtain a first output vector; performing rough feature extraction on the images subjected to face recognition and positioning by using a Lenet5 convolution network, and extracting generation features of the images of any frame image group by using a VAE self-encoder to obtain a first feature vector corresponding to the generation features; inputting the first output vector and the first feature vector into a fully-connected classification layer, adding an entropy function of a classification layer and a real label, and obtaining an emotion grade prediction result.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, in particular to a multi-feature outdoor environment emotion recognition method based on an attention mechanism. BACKGROUND

[0002] The development of science and technology has promoted the wide application and research of emotion recognition technology, which can be applied to video monitoring, interrogation, satisfaction survey and other fields. At present, inferring the changes in the inner psychological activities through emotion analysis technology has become a very popular research field, and the mature DEAP dataset has made outstanding contributions to this field. In addition, emotion analysis technology can also be used in education, natural environment and other occasions to reflect the acceptance and cognitive degree of the subject learning and the satisfaction degree of the external environment. Through emotion analysis, relevant personnel can make timely adjustments and controls.

[0003] Emotion is a comprehensive reflection of a person's physiological and psychological state, and is a comprehensive reflection of a person under the influence of changes or stimuli in the external environment. Emotion reflects a person's attitude towards objective things, is influenced by a person's subjective consciousness, reflects the connection between a person's needs and the objective environment, and reflects a person's psychological activity. The understanding and evaluation of the external environment and things will lead to differences in emotion, which will not only affect the individual's physical and behavioral conditions, but also cause corresponding changes in the physiological conditions of the body itself, thereby causing changes in physiological signals. Research has found that when a person is angry, the breathing rate will significantly increase, causing a short change in the breathing signal; when a person is excited, the heart rate will increase, causing an increase in the frequency of the electrocardiogram signal. It can be seen that emotion is a combination of psychology and physiology, and the way of physiological activity is not the same under different emotional states. Therefore, using physiological data for emotion analysis is scientific and reliable.

[0004] However, in actual application scenarios, physiological data is often obtained with the help of sensors and other devices, and it is difficult to capture directly at any time and any place, but it is helpful for further learning and cognition to reflect the true labels of the data set. For example, the Chinese invention patent with the patent publication number CN111931616A and the name "Emotion recognition method and system based on mobile intelligent terminal sensor device" collects original sensor data under different emotion categories through each sensor on the mobile intelligent terminal as training samples, and labels the true emotion categories corresponding to each training sample; features are extracted from the preprocessed sensor data to obtain the feature vectors corresponding to each training sample under different emotion categories, and a training set is constructed; an emotion recognition classifier is obtained based on the training set; and the original sensor data collected by each sensor on the mobile intelligent terminal in real time is predicted using the emotion recognition classifier to obtain an emotion prediction result.

[0005] At present, the emotion recognition technology in the prior art has been widely applied to daily life, but the difficulty of emotion recognition is often greatly affected by the environment, so that the use of corresponding sample data for training the emotion recognition model in different outdoor environments can often obtain higher precision recognition effect, and multiple training can greatly reduce the efficiency of the algorithm. And the traditional deep learning needs a large amount of training data, and the large amount of acquisition of user data in the outdoor environment will bring privacy, ethics and other problems. In addition, the emotion samples in the outdoor environment come from the emotional feedback in the natural scene, not through experimental stress or suggestion, so the sample label distribution is uneven.

[0006] Therefore, it is urgent to provide a multi-feature outdoor environment emotion recognition method based on attention mechanism, which is simple in logic and reliable in recognition. SUMMARY

[0007] In view of the above problems, the purpose of the present application is to provide a multi-feature outdoor environment emotion recognition method based on attention mechanism, and the technical solution adopted by the present application is as follows:

[0008] The multi-feature outdoor environment emotion recognition method based on attention mechanism comprises the following steps:

[0009] Obtain the physiological data of the measured person and the image in the outdoor environment;

[0010] The physiological data features of the measured person are extracted by using the Relief nonlinear feature extraction algorithm, and the emotion level label corresponding to any frame image group is obtained by using the Conv LSTM recognition model;

[0011] The images of any frame image group are preprocessed, and the face recognition and positioning are performed by using the MTCNN network;

[0012] The texture features of the images of any frame image group are extracted by using the Resnet module, and the images of any frame image group are converted into one-dimensional vectors to obtain the first output vector;

[0013] The images of face recognition and positioning are coarsely extracted by using the Lenet5 convolution network, and the generative features of the images of any frame image group are extracted by using the VAE autoencoder to obtain the first feature vector corresponding to the generative features;

[0014] The first output vector and the first feature vector are input into the fully connected classification layer, and the entropy function of the classification layer and the real label is added to obtain the emotion level prediction result.

[0015] Compared with the prior art, the present application has the following beneficial effects:

[0016] (1) The present application ingeniously adopts the Relief nonlinear feature extraction algorithm to extract the physiological data features of the measured personnel, and adopts the Conv LSTM recognition model to obtain the emotion level label corresponding to any frame image group, which has the advantages that real and reliable data set labels are obtained, and video image data which is easy to obtain is used as emotion recognition judgment basis which can be widely applied, and the technical problem of "outdoor natural environment without stress condition stimulation emotion generation, physiological data cannot reliably reflect emotion level" in the prior art is overcome.

[0017] (2) The present application ingeniously adopts the Resnet module to extract the texture features of the images of any frame image group according to the features of human face such as weak symmetry, heredity and distributed standard, and the generation features of the whole image can better describe the image characteristics and increase the accuracy of emotion prediction.

[0018] (3) The present application ingeniously adds the double optimization of MSE loss and emotion level prediction loss between images, which can reduce the probability of local convergence and ensure the stability and continuity of feature learning.

[0019] (4) The present application ingeniously uses the texture features and generation features of images, which reflect the special effects of images from different angles, the former can reflect the intuitive characteristics of images as the main features, and the latter can analyze the network features through the reconstruction performance. The former tests the multi-feature dimension learning performance, so the channel attention mechanism is introduced to strengthen the learning emphasis of the main features, and the latter focuses on the generative nature of images, so the spatial attention mechanism can further highlight the key and non-key features of the generated image.

[0020] (5) The present application ingeniously introduces the attention mechanism combined with multi-feature learning, which correlates the cross entropy loss function of emotion level prediction through the MSE loss between generated images and input images, so that the whole learning optimization process is related and mutually influenced, which can make the final learning goal get better image generation features and texture features at the same time.

[0021] In summary, the present application has the advantages of simple logic and reliable recognition, and has high practical value and popularization value in the field of image processing technology. BRIEF DESCRIPTION OF DRAWINGS

[0022] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following will briefly introduce the drawings needed to be used in the embodiments, and it should be understood that the following drawings only show some embodiments of the present application, and should not be regarded as limiting the scope of protection, and for those skilled in the art, other related drawings can also be obtained without creative labor on the basis of these drawings.

[0023] Figure 1Logic flow diagram for the present invention.

[0024] Figure 2 Schematic diagram of the residual convolutional network of the present invention.

[0025] Figure 3 Schematic diagram of the channel attention module of the present invention.

[0026] Figure 4 Schematic diagram of the spatial attention module of the present invention.

[0027] Figure 5 Schematic diagram of the VAE autoencoder of the present invention.

[0028] Figure 6 Schematic diagram of the physiological signal heterogeneous dataset mapping model of the present invention.

[0029] Figure 7 Logic flow diagram for the present invention.

[0030] Figure 8 Network architecture diagram for the present invention. DETAILED DESCRIPTION

[0031] In order to make the purposes, technical solutions and advantages of the present application clearer, the present application will be further described below in conjunction with the accompanying drawings and embodiments. The embodiments of the present application include but are not limited to the following embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0032] In the present embodiment, the term "and / or" is merely used to describe the association relationship of the associated objects, and can represent the existence of three relationships, for example, A and / or B can represent the existence of A alone, the existence of A and B together, and the existence of B alone.

[0033] The terms "first" and "second" and the like in the description and claims of the present embodiment are used to distinguish different objects, and are not used to describe the specific order of the objects. For example, the first target object and the second target object are used to distinguish different target objects, and are not used to describe the specific order of the target objects.

[0034] In the present embodiment, the words "exemplary" or "for example" are used to represent an example, illustration or description. Any embodiment or design scheme described as "exemplary" or "for example" in the present embodiment should not be interpreted as more preferred or more advantageous than other embodiments or design schemes. Rather, the words "exemplary" or "for example" are intended to present the relevant concept in a specific manner.

[0035] In the description of the embodiments of the present application, unless otherwise specified, the meaning of "a plurality of" is two or more. For example, a plurality of processing units refers to two or more processing units; a plurality of systems refers to two or more systems.

[0036] As shown in Figures 1 to 8 , the present embodiment provides a multi-feature outdoor environment emotion recognition method based on attention mechanism. First, the network architecture involved in the present embodiment is described:

[0037] First, the residual convolutional network:

[0038] The emotion recognition model applied to the outdoor environment is mainly affected by the natural environment, and the emotional fluctuations of the testers are not obvious, so the model uses a deep residual structure as the main training and learning model. The residual architecture can effectively solve the problems of gradient disappearance and gradient explosion in deep neural networks. The Resnet module based on convolutional splitting used in the present embodiment is as shown in Figure 2 , where k is the size of the convolution kernel, and in the present patent, k=5 and k=3 are mainly used to achieve fine extraction of image features.

[0039] Second, the principle of attention mechanism based on convolution module:

[0040] In the present embodiment, the attention mechanism module applied to image generation features and texture features combines spatial and channel attention mechanism modules respectively. Among them,

[0041] As shown in Figure 3 , the channel feature enhancement network of the present embodiment processes the input feature image through global maximum value pooling and global average value pooling based on width and height respectively, and then through a multilayer perceptron (MLP). The features output by the MLP are combined and then subjected to a sigmoid activation operation to generate the final channel attention feature layer. The channel attention feature layer and the input feature are multiplied by an element multiplication operation to generate an input feature with feature channel attention weight.

[0042] As shown in Figure 4 , the spatial feature enhancement network of the present embodiment first performs a channel-based global maximum value pooling and global average value pooling on the input feature image, and then connects the two results based on the channel. Then, after a convolution operation, it is reduced to a single channel. Then, a sigmoid activation function is used to generate a spatial attention weight. Finally, the feature and the input feature of this module are multiplied to obtain an input feature with feature spatial attention weight.

[0043] Third, VAE autoencoder:

[0044] like Figure 5 As shown, the model compresses vectors from the input feature space into the hidden feature space through the encoder to obtain hidden variables, while the decoder restores the representation from the hidden feature space to the original input space. During training, to ensure effective encoding, the parameters are updated by combining the decoder's output with the sample input and performing backpropagation. The final objective function of the VAE is L(θ,φ; x). (i) )for:

[0045]

[0046] Among them, D Z Represents the dimension of the latent variable Z; and Let P represent the mean and variance of the Gaussian distribution obtained from the intermediate layers of the VAE network, respectively; L represents the number of times the input sample is obtained from the known probability distribution; θ x represents the likelihood function that maximizes the parameterization; (i) Indicates the input image; z (i,l) This represents the l-th sampling of the i-th component of z.

[0047] Fourth, the MTCNN network:

[0048] In this embodiment, the MTCNN network first transforms the image at different scales to construct an image pyramid, adapting to the detection of faces of different sizes. The detection process of MTCNN can be generally divided into a three-layer network structure: P-Net, R-Net, and O-Net.

[0049] The basic structure of P-Net is a fully convolutional network. The image pyramid constructed in the previous step is processed by a fully convolutional network for initial feature extraction and bounding box labeling. Regression box localization is then used to adjust the window, and NMS (Non-Maximum Suppression) is applied to filter most of the windows. P-Net is a face region proposal network. After the feature input is processed through three convolutional layers, this network can determine whether the current region contains a face and ultimately outputs many face regions that may contain faces. These regions are then input into R-Net for further processing.

[0050] In addition, the R-Net is based on the P-Net and adds a fully connected layer to perform more fine feature extraction. All face prediction windows output by the P-Net are sent to the R-Net for further filtering. The R-Net performs fine selection on the input and filters most of the wrong inputs, and performs face region bounding box regression and positioning again, to output more reliable face regions, so that the O-Net can perform final face region detection.

[0051] In addition, the O-Net is based on the R-Net and adds a convolutional layer again. The network performs face discrimination, face region bounding box regression and face feature positioning at the same time, and finally outputs the positioning coordinates of the face region and the corresponding five feature points. The MTCNN is mainly used for its accurate detection and positioning ability in the present patent.

[0052] Fifth, the Relief nonlinear feature extraction network:

[0053] In the present embodiment, first, the physiological data is preprocessed, feature extracted, and feature selected, and then trained and classified in a neural network model to obtain an implicit emotional response.

[0054] The feature selection function is to select certain physiological features to provide preparation for later physiological data analysis. The present embodiment uses the Relief feature selection algorithm to extract features to find features that affect emotional classification, and analyzes the performance under different emotional states according to the distribution of these features in the nervous system. Relief is an instance-based nonlinear feature extraction method. The algorithm randomly selects a sample S from the training set D, then finds the nearest neighbor H from the same class of S, called Near Hit, and finds the nearest neighbor M from the different class of S, called Near Miss. The nearest neighbor M is obtained by using the Euclidean distance formula.

[0055] According to the above formula, if the distance between sample S and Near Hit in a certain feature is smaller than the distance between S and Near Miss, then this feature is beneficial to distinguishing similar or different nearest neighbors, and its weight will be increased; on the contrary, if the distance between sample S and Near Hit in a certain feature is larger than the distance between S and Near Miss, then this feature has a negative effect on distinguishing similar and different nearest neighbors, and its weight will be decreased. Repeat the above steps, and finally obtain the average weight of each feature.

[0056] The multi-feature outdoor environment emotion recognition method of the present embodiment will be described in detail below, which includes the following steps:

[0057] First, the image of the measured person in the outdoor environment is collected, in the embodiment, the physiological data features of the measured person are extracted by using the Relief nonlinear feature extraction algorithm, a sample is randomly selected from the video training set, and the nearest neighbor samples of the same kind and different kind are selected, the Euclidean distance between the samples is calculated to obtain the average weight of each feature to select the features with strong classification ability.

[0058] Second, the construction of the physiological signal heterogeneous data set mapping model, which includes a manifold reduction layer, a common embedding space, a tangent projection layer and a transfer learning layer connected in turn.

[0059] The expression of the manifold reduction layer is:

[0060] R i (S j )=W i ·S j ·W i T

[0061] Wherein, R i (S j ) represents the mapping output of the manifold reduction layer, W i represents the weight matrix of the neural network; S j represents the covariance matrix; W i T represents the transpose of the weight matrix of the neural network;

[0062] In the common embedding space, the probability distribution of the image of any two feature data sets is calculated The expression is:

[0063]

[0064] Wherein, represents the maximum average difference; sup represents the minimum upper limit; represents the expectation of the probability distribution Q i of the common embedding space; R i (S) represents a projection matrix of a low-dimensional mapping of i to the low-dimensional SPD manifold; represents the expectation of the probability distribution Q j of the common embedding space; R j (S) represents a projection matrix of a low-dimensional mapping of j to the low-dimensional SPD manifold; represents the probability distribution;

[0065] In the tangent projection layer, the matrices of the manifold reduction layer and the common embedding space are projected to the tangent space of the tangent projection layer, and the expression is:

[0066] -1 -1

[0067] wherein V1, V2 represent two tangent vectors of the tangent projection layer; P -1 represents the inverse matrix of the matrix P on the common embedding space;

[0068] In the transfer learning layer, the model weight is trained by transfer learning, the feature map aggregation of the joint average of any measured person is aggregated, and the physiological data features are aggregated by the classifier.

[0069] The Softmax layer is added in the transfer learning layer, and its expression is:

[0070]

[0071] wherein y Si represents the predicted output of the covariance sequence S i of the input variable; B(X) represents the output of the tangent layer; FC represents the full connection layer;

[0072] The domain loss L1 and the classification loss L2 of the emotion level are added in the physiological signal heterogeneous data set mapping model, and its expression is:

[0073]

[0074]

[0075] wherein m represents the total number of manifolds; K represents the total number of labels; represents the exponential distribution parameter under the entropy loss function, represents the maximum average difference with labels; represents the probability distribution on the ith manifold with labels; represents the probability distribution on the jth manifold with labels; L C represents the cross entropy loss function, represents the predicted label of the ith sample; Y represents the predicted result of the ith sample.

[0076] ​​​Thirdly, a Conv LSTM recognition model is constructed. In the embodiment, the physiological data features collected and the standard data set features aggregated therewith are taken as inputs, and the positive, neutral and negative three emotional levels are taken as outputs. The emotional level is calculated based on the state of the current frame and the hidden state of the Conv LSTM at the previous time. In order to identify the emotional state, the ConvLSTM of the linear layer is used to calculate the emotional score according to the physiological features of the current time frame t, the hidden state and the memory of the previous stage. Since emotions often have persistence, the average emotional level score of the segment video corresponding to T time is rounded to its emotional label.

[0077] wherein the expression of the input gate of the basic unit of the Conv LSTM recognition model is:

[0078]

[0079] wherein i t represents the output of the input gate of the basic unit; σ represents the activation function of the input gate, which is composed of a sigmoid neural network layer and a pair of multiplication operations; W xi represents the weight of the input unit; X t represents the input of the current network of the input gate of the basic unit; W hi represents the weight of the hidden unit of the input gate; H t-1 represents the output of the previous moment of the input gate of the basic unit; W ci represents the weight of the information learned at the previous moment; C t-1 represents the cell state of the previous moment of the input gate of the basic unit; b i represents the bias of the input gate;

[0080] In addition, the expression of the dropout information f t of the Conv LSTM recognition model is:

[0081]

[0082] wherein Wx f represents the weight of the input unit of the forgetting gate; Wh f represents the weight of the hidden unit of the forgetting gate; W cf represents the weight of the information learned at the previous moment of the forgetting gate; b f represents the bias of the forgetting gate;

[0083] Further, the expression of the update information C t of the Conv LSTM recognition model is:

[0084]

[0085] wherein tanh represents a hyperbolic tangent function returning any real number, W xc represents a weight of an input unit of an output gate; W hc represents a weight of a hidden unit of a forget gate of an output gate; b c represents a bias of an output gate;

[0086] In addition, the output information O t of the Conv LSTM recognition model has an expression as follows:

[0087] H t = o t ⊙ tanh(C t )

[0088] wherein C t represents learned information at a current time; and represents a convolution operation; and ⊙ represents a Hadamard product.

[0089] In the embodiment, the Conv LSTM recognition model uses a Softmax function as a classifier to output an emotion level label, and the Softmax function outputs a record S t (y n ) at any time frame.

[0090] The difference m t (y) between the emotion state with the highest emotion score and other emotion states is calculated, and has an expression as follows:

[0091]

[0092] wherein S t (y) represents a classification emotion score of an emotion classification y at time t, and S t (y') represents a classification emotion score different from the classification emotion score of the emotion classification y at time t; y' represents the classification emotion score of the emotion classification y at time t; and y n represents n emotion classifications.

[0093] A fluctuation loss L t is introduced to punish the fluctuation of the difference, and has an expression as follows:

[0094]

[0095] wherein -log S t (y) represents a traditional classification loss; λ1 represents a relative constant; m t' (y) represents a discrimination boundary of the emotion classification y at time t; and t' represents a time point corresponding to a maximum value of the discrimination boundary m t '(y) of the emotion classification y at time t in a time period from t0 to t-1.

[0096] The bias of the emotion score is corrected by using a difference loss function, which is expressed as:

[0097] L t = -log S t (y) + ||S t (y1) - S

[0098] wherein S t (y1) represents the classification emotion score of the emotion classification y1 at time t; and S t (y2) represents the classification emotion score of the emotion classification y2 at time t.

[0099] The fourth step is frame extraction of dynamic expressions. The overall idea of the embodiment is to use physiological data to make real and reliable emotion rating labels for the training data set, and then use an expression recognition model to learn and train emotions through expressions. The steps of the fifth step to the tenth step below mainly correspond to the expression recognition model.

[0100] The fifth step is frame image preprocessing. The input frame image group is subjected to grayscale processing, and images without human faces are filtered out.

[0101] The sixth step is to use the MTCNN network for face recognition and positioning. The frame image is input into the MTCNN network, and the weight value of the pre-trained model is used to calibrate and crop the accurate position of the face in each frame image, remove irrelevant information in the image, and improve the accuracy of the expression recognition model. The MTCNN network uses the weight value of the pre-trained model to calibrate and crop the accurate position of the face in each frame image, which is expressed as:

[0102]

[0103] wherein round represents the rounding function; F represents the emotion recognition model; x(i) represents the i-th image input in the time; and N represents the total number of emotion rating categories.

[0104] The seventh step is to use the Resnet module to extract the texture features of the image:

[0105] In the embodiment, the MTCNN network is composed of a feature extraction network and an attention mechanism feature enhancement network; the attention mechanism feature enhancement network is composed of a channel feature enhancement network and a spatial feature enhancement network; the feature extraction network is composed of an upper half and a lower half of a parallel two-way; the upper half is composed of three Resnet network models based on channel attention mechanism enhancement, i.e., a first Resnet module, a second Resnet module and a third Resnet module; and the lower half is composed of an image feature rough extraction part based on a Lenet5 convolution structure and a VAE autoencoder.

[0106] Firstly, the gray image is input into the first Resnet module for training, and a 16*5*5 detachable convolution kernel is used for learning.

[0107] Eighthly, the output image of the first Resnet module is input into the channel feature enhancement network, and the obtained channel attention weight is multiplied with the input image.

[0108] Ninthly, the output of the first Resnet module is input into a 2*2 maximum value pooling layer to reduce memory usage and further extract main features.

[0109] Tenthly, the output of the ninth step is sequentially input into the next three Resnet modules with the same structure.

[0110] Eleventhly, the image after the tenth step is converted into a one-dimensional vector, and a first output vector is obtained through a full connection layer.

[0111] Twelfthly, the VAE autoencoder is used to extract the generated features of the image.

[0112] The image obtained in the sixth step is input into the convolution module of Lenet5, including a 6*5*5 convolution layer, a 2*2 maximum value pooling layer and a 16*5*5 convolution layer. The convolution structure in Lenet5 is used to preliminarily screen and refine the features of the image.

[0113] Thirteenthly, the image group obtained in the twelfth step is input into the spatial feature enhancement network, and the obtained spatial attention weight is multiplied with the input image.

[0114] Step 14: After converting the image output in step 13 into a one-dimensional vector, input the one-dimensional vector into the VAE autoencoder as input, and use the autoencoder to train a first feature vector that can reflect the image generation characteristics. The VAE autoencoder is composed of an encoder (fully connected layer (786, 256), fully connected layer (256, 64), fully connected layer (64, 20)) and a decoder (fully connected layer (10, 64), fully connected layer (64, 256), fully connected layer (256, 784)). The training process compares the generated image restored by the output vector of the decoder with the input image.

[0115] Step 15: Connect the two one-dimensional vectors obtained in steps 11 and 14, and then input them into a fully connected classification layer. Calculate the entropy function of the classification layer and the true label, that is, the emotion level recognition loss function, and finally obtain the emotion level prediction result. The expression of the emotion level recognition loss function is:

[0116]

[0117] where M is the one-dimensional vector after flattening the image; x i represents the pixel value of the i-th element coordinate in the x direction; y i represents the pixel value of the i-th element coordinate in the y direction.

[0118] Specific case:

[0119] The scheme adopted in this embodiment requires a large amount of video data with good clarity, so when using the training network, the use scenario needs to be determined in advance. Given the application field and value, the requirement for the scene is a camera with good clarity and stability for outdoor natural environment use. For the preparation process of the training, verification, and test data set, facial video data and physiological data of each test person need to be obtained.

[0120] Step 1: Wear a finger sleeve type skin electricity acquisition device on the tester to measure the skin resistance between two fingers, and collect the resistance change as the skin electricity signal. The HKR skin electricity acquisition device is used to collect the skin electricity signal, and the range is set to 100K-3M, and the measurement accuracy is 2.5K. In addition, the HKG infrared pulse acquisition device is used to collect the pulse signal, and the sampling accuracy is 10 bits.

[0121] Step 2: Obtain the video data of the testee in the outdoor natural scene through a stable high-definition camera.

[0122] Step 3: Preprocess the obtained video, divide each second of video into 14 frame images, and filter out images without human faces using the dlib library function.

[0123] Step4: Randomly split the pre-processed video segments (frame image groups) into datasets, and divide the training set and the test set in a ratio of 4:1.

[0124] Step5: Divide the training set again in a ratio of 4:1 to obtain the training set and the validation set.

[0125] Step6: Use physiological data to train each frame image group to obtain its emotional rating label. First, use the Relief nonlinear feature extraction method to extract the physiological data features of the subjects, and calculate the average weight of each feature by calculating the Euclidean distance between samples to select features with strong classification ability.

[0126] Step7: Use the DEAP dataset and the physiological data part of the collected dataset to combine to form a distributed heterogeneous dataset.

[0127] Step8: Based on the Conv LSTM recognition model

[0128] In this embodiment, the collected physiological data features and the aggregated standard dataset features are used as input, and the emotional rating is calculated based on the state of each frame and the hidden state of the previous time. The number of layers of LSTM is set to 5.

[0129] Step9: The complete dataset should have image data, image emotional rating label, and segment video label. Set the batch size to 32, that is, 32 image groups are input into the network each time, and resize each frame in the image group to 112*112. Thus, the input is set to [32,3,112,112], where 3 represents 3 color channels of each frame image.

[0130] Step10: Grayscale the input image group to obtain [32,1,112,112]

[0131] Step11: Extract texture features and generated features from the image through a dual-channel processing method.

[0132] Step12: First, input the one-dimensional image obtained in Step10 into the first Resnet convolution module. This layer uses a 1 kernel with a template of 16*5*5, the image expansion padding is 0, and the step length stride is 1, to obtain an output image group of [32,16,108,108].

[0133] Step 13: The output image obtained in Step 12 is processed by AdaptiveMaxpool2d(1) and AdaptiveAvgpool2d(1) to obtain spatial layer information, and the two vectors are respectively passed through fully connected layers ((32*16, 32), (32, 32*16)), connected through a Sigmoid activation function, and converted to [32, 16, 1, 1], which is multiplied by the image of [32, 16, 108, 108].

[0134] Step 14: The output image obtained in Step 13 is processed by a 2*2 maximum value pooling layer to obtain an output image [32, 16, 54, 54]

[0135] Step 15: The steps of Step 12-14 are repeated using the next three Resnet modules to obtain output images [32, 16, 25, 25], [32, 16, 10, 10], and [32, 16, 3, 3], respectively.

[0136] Step 16: The results obtained in Step 15 are converted to one-dimensional vectors, i.e., [32, 16*3*3].

[0137] Step 17: The results obtained in Step 16 are input into a fully connected layer (144, 20) to obtain a single-dimensional vector output of [32, 20].

[0138] Step 18: The one-dimensional image obtained in Step 10 is input into the Lenet5 network to obtain an output image [32, 16, 50, 50].

[0139] Step 19: The image obtained in Step 18 is respectively processed using maximum value pooling and average value pooling for image feature channel dimensions to obtain outputs [32, 1, 50, 50] and [32, 1, 50, 50], and the outputs are connected to obtain [32, 2, 50, 50], which is processed using a convolution kernel template (2, 1, 1, 1) with padding 0 and stride 1 to obtain spatial attention weight value [32, 1, 50, 50], which is multiplied by the output image of Step 18.

[0140] Step 20: The results obtained in Step 19 are input into the VAE autoencoder network to obtain a feature vector [32, 20] output by the encoder, half of which is the mean and half of which is the variance, and a random noise with a dimension of 10 is introduced to obtain the output of the decoder.

[0141] Step21: Calculate the loss in batches, and use the loss back propagation to optimize the MSE loss between the output image of Step19 and the output image of Step20.

[0142] Step22: Fuse the feature vector obtained in Step20 with the feature vector obtained in Step17 to obtain [32, 2, 20, 20], and convert it into a vector [32, 800].

[0143] Step23: Input the vector obtained in Step22 into the fully connected layer ((800, 256), (256, 64), (64, 3)) to obtain the output [32, 3], calculate the cross-entropy loss, and output the predicted value of the emotion level. Compare with the label, and use the Adam optimizer to optimize.

[0144] Step24: When the training of the training set is completed, input the validation set obtained by the initial division into the neural network for verification, and calculate the MSE loss of the VAE autoencoder and the emotion prediction loss.

[0145] Step25: Save and compare the loss function values after each training set traversal training by the iterative idea, stop training and output the network parameters when the loss value increases.

[0146] Step26: Test the input test set using the optimal network obtained in Step25.

[0147] The above examples are only preferred embodiments of the present application, and are not intended to limit the scope of protection of the present application. Any changes made on the basis of the design principles of the present application and non-creative labor shall be within the scope of protection of the present application.

Claims

1. A multi-feature outdoor environment emotion recognition method based on attention mechanism, characterized in that, Includes the following steps: Acquire physiological data and images of the subjects in outdoor environments; The Relief nonlinear feature extraction algorithm was used to extract the physiological data features of the test subjects, and the Conv LSTM recognition model was used to obtain the emotion level label corresponding to any frame image group. The images in any frame group are preprocessed and then face recognition and localization are performed using the MTCNN network; The ResNet module is used to extract texture features from any group of images in a frame, and the images in any group of images in a frame are converted into one-dimensional vectors to obtain the first output vector. Specifically, the network architecture consists of two parts: a feature extraction network and an attention mechanism feature enhancement network. The attention mechanism feature enhancement network consists of a channel feature enhancement network and a spatial feature enhancement network. The feature extraction network consists of a parallel dual-path upper half and a lower half. The upper half is composed of four ResNet network models based on channel attention mechanism enhancement, namely the first ResNet module, the second ResNet module, the third ResNet module, and the fourth ResNet module. The lower half consists of a coarse image feature extraction part based on Lenet5 convolutional network and a VAE autoencoder. First, the grayscale image is input into the first ResNet module for training, and it is learned through a 16*5*5 separable convolutional kernel. The output image of the first ResNet module is input into the channel feature enhancement network, and the channel attention weights obtained from the output are multiplied with the input image; the output of the first ResNet module is input into a 2*2 max pooling layer; the image output from the max pooling layer is passed through three identical ResNet modules, a second ResNet module, a third ResNet module, and a fourth ResNet module connected in sequence. The image output by the fourth ResNet module is converted into a one-dimensional vector and then passed through a fully connected layer to obtain the first output vector; The Lenet5 convolutional network is used to coarsely extract features from the face recognition and localization images, and the VAE autoencoder is used to extract the generation features of any frame group of images to obtain the first feature vector corresponding to the generation features. Specifically, the face recognition and localization images are input into the convolutional module of the Lenet5 convolutional network, including a 6*5*5 convolutional layer, a 2*2 max pooling layer, and a 16*5*5 convolutional layer; the output of the convolutional module is connected to the spatial feature enhancement network; the spatial attention weights obtained from the output of the spatial attention network are multiplied with the input face recognition and localization images to obtain the second image; the second image is converted into a one-dimensional vector and used as the input of the VAE autoencoder, which is used to extract the generation features of any frame group of images to obtain the first feature vector corresponding to the generation features. The first output vector and the first feature vector are input into a fully connected classification layer, and the entropy function of the classification layer and the true label is added to obtain the emotion level prediction result.

2. The multi-feature outdoor environment emotion recognition method based on attention mechanism according to claim 1, characterized in that, The Relief nonlinear feature extraction algorithm is used to extract physiological data features of the subjects, including the following steps: Images of the subjects in outdoor environments are extracted, and the Euclidean distance algorithm is used to find the nearest neighbor samples of the same and different types of subjects. The average weight of any feature is calculated based on the nearest neighbor samples of the same and different classes, and the feature dataset with strong classification recognition is selected. A physiological signal heterogeneous dataset mapping model is constructed, and the feature dataset is input into the physiological signal heterogeneous dataset mapping model to obtain physiological data features.

3. The multi-feature outdoor environment emotion recognition method based on attention mechanism according to claim 2, characterized in that, The physiological signal heterogeneous dataset mapping model includes a manifold reduction layer, a common embedding space, a tangent projection layer, and a transfer learning layer connected in sequence. The expression for the manifold reduction layer is: ;in, This represents the mapped output of the manifold reduction layer. The weight matrix representing the neural network; Represent the covariance matrix; This represents the transpose of the weight matrix of a neural network. Within the common embedding space, the probability distribution of images from any two feature datasets is calculated, and its expression is: ;in, Indicates the maximum average difference; Indicates the least upper bound; Probability distribution representing the common embedding space Expectations; Representing a low-dimensional mapping will The projection matrix mapped onto the low-dimensional SPD manifold; Probability distribution representing the common embedding space Expectations; Representing a low-dimensional mapping will The projection matrix mapped onto the low-dimensional SPD manifold; Represents the probability distribution; Within the tangent projection layer, the matrices of the manifold reduction layer and the common embedding space are projected onto the tangent space of the tangent projection layer, and its expression is: ;in, , Represents the two tangent vectors of the tangent projection layer; Let represent the inverse matrix of matrix P in the common embedding space; Within the transfer learning layer, the model weights are trained using transfer learning, and the feature maps of any subject are aggregated by joint averaging. The physiological data features are then obtained by aggregating the aggregated data using a classifier.

4. The multi-feature outdoor environment emotion recognition method based on attention mechanism according to claim 3, characterized in that, The transfer learning layer includes a Softmax layer, the expression of which is: ;in, Represents the covariance matrix for the input variables The predicted output; This indicates the output of the tangent layer; Indicates a fully connected layer; Adding domain loss to the mapping model of heterogeneous physiological signal datasets Categorical loss of emotion level Its expression is: ; ;in, This represents the total number of manifolds; Indicates the total number of tags; This represents the parameters of the exponential distribution under the entropy loss function; Indicates the maximum average difference with labels; Indicates the tagged first Probability distribution on a manifold; Indicates the tagged first Probability distribution on a manifold; Represents the cross-entropy loss function. Indicates the first Predicted labels for each sample; Indicates the first The prediction results for each sample.

5. The multi-feature outdoor environment emotion recognition method based on attention mechanism according to claim 1, characterized in that, The expression for the input gate of the basic unit of the Conv LSTM recognition model is: ;in, This represents the output of the input gate of the basic unit; The activation function representing the input gate consists of a sigmoid neural network layer and a pairwise multiplication operation; Indicates the weight of the input unit; This represents the input of the current network to the input gate of the basic unit; The weights of the hidden units in the input gate are represented. This represents the output of the input gate of the basic unit at the previous time step; This indicates the weight of the information learned in the previous time step; This indicates the cell state of the input gate of the basic cell at the previous moment; Indicates the bias of the input gate; The Conv LSTM recognition model discards information. The expression is: ;in, The weights of the input cells of the forget gate are represented. The weight of the hidden unit in the forget gate; This represents the weight of the forget gate relative to the information learned in the previous time step; Indicates the bias of the forget gate; The update information of the Conv LSTM recognition model The expression is: ;in, This represents the hyperbolic tangent function that returns any real number. This represents the weights of the input cells of the output gate; The weights of the hidden units in the forget gate of the output gate are indicated. Indicates the bias of the output gate; The output information of the Conv LSTM recognition model The expression is: ;in, This represents the information learned at the current moment; Indicates the convolution operation; This represents the Hadamard product.

6. The multi-feature outdoor environment emotion recognition method based on attention mechanism according to claim 5, characterized in that, The Conv LSTM recognition model uses a Softmax function as a classifier to output sentiment level labels, and the output of the Softmax function at any given time frame is recorded. ; Calculate the difference between the emotional state with the highest emotion score and other emotional states. Its expression is: ;in, This represents the sentiment score of sentiment category y at time t. This represents the sentiment score of a sentiment category other than sentiment category y at time t; This represents the sentiment category other than sentiment category y; Represents n types of emotion categories; Introducing volatility loss To penalize fluctuations in the difference, its expression is: ;in, This represents the traditional classification loss; Represents a relative constant; The threshold for distinguishing sentiment category y at time t; express The discrimination threshold of sentiment category y at time t within the time period t-1. The corresponding time point of maximum value; The difference loss function is used to correct the bias in sentiment scores; its expression is as follows: ;in, Indicates sentiment classification The sentiment score at time t; Indicates sentiment classification The sentiment score at time t.

7. The multi-feature outdoor environment emotion recognition method based on attention mechanism according to claim 1, characterized in that, The MTCNN network uses the weights of the pre-trained model to accurately locate and crop the face within each frame of the image. Its expression is as follows: ;in, round represents the rounding function; F represents the emotion recognition model; Indicates the first time within a time period Input images; N represents the total number of emotion level categories.

Images