Eye imaging emotion detection method and system based on three-flow convolutional neural network

Abstract

The application provides an eye imaging emotion detection method and system based on a three-flow convolutional neural network, which comprises the following steps: acquiring an eye image, an optical flow image and a paradigm image; inputting the eye image, the optical flow image and the paradigm image into a three-flow convolutional neural network model; extracting time features of the eye image, space features of the eye image and paradigm semantic features of the paradigm image by using the three-flow convolutional neural network model; fusing the time features, the space features and the paradigm semantic features to obtain fused features; inputting the fused features into a full connection layer to calculate a classification result label of the eye image by using a combination function; and outputting a classification result according to the classification result label. The method can obtain more accurate eye change features from the eye image by using the time features, the space features and the paradigm semantic features of the eye image, and thus can detect the emotional changes of individuals, and can improve the accuracy of the detection result and solve the problem of inaccurate detection result when detecting the emotions of individuals.

Description

Technical Field

[0001] This application relates to the field of eye imaging detection technology, and in particular to an eye imaging emotion detection method and system based on a three-stream convolutional neural network. Background Technology

[0002] Emotion is a general term for a series of subjective cognitive experiences; it is a psychological and physiological state resulting from the synthesis of various feelings, thoughts, and behaviors. Whether positive or negative, emotions are the motivations that trigger individual actions. Paying attention to an individual's emotional characteristics plays a crucial role in emotional guidance and individual safety. Emotions such as joy, anger, sorrow, surprise, and fear can be identified through facial expressions, while subtle emotions such as jealousy, shame, embarrassment, and pride require observation of changes in eye characteristics.

[0003] Eye movement patterns and pupillary changes can serve as biomarkers for emotion and cognitive dysfunction. Detecting eye movement patterns and pupillary changes is a form of emotion detection based on physiological signal data. However, in eye movement behavior recognition, some undefined eye changes are ignored, causing the recognition process to lose important information from the eye images. Furthermore, the performance of some algorithms heavily depends on the quality and selection of eye movement feature design; different emotional changes require different features, thus necessitating significant manpower and time for feature design and optimization. This leads to inaccurate detection results when using fundus imaging to detect an individual's emotions. Summary of the Invention

[0004] This application provides a method and system for emotion detection in eye imaging based on a three-stream convolutional neural network, in order to solve the problem of inaccurate detection results when detecting individual emotions.

[0005] The first aspect of this application provides a method for emotion detection in eye imaging based on a three-stream convolutional neural network, including:

[0006] Acquire eye images, optical flow images, and paradigm images. The eye images include eye movement behavior images, the optical flow images are motion relationship images between two of the eye images, and the paradigm images are images from the paradigm video.

[0007] The eye image, the optical flow image, and the paradigm image are input into a three-stream convolutional neural network model, which is a neural network model trained using sample images; the sample images include eye images with training labels and the paradigm image; the three-stream convolutional neural network model includes a spatial module, a temporal module, and a paradigm module;

[0008] The temporal and spatial features of the eye image and the paradigm semantic features of the paradigm image are extracted using the three-stream convolutional neural network model.

[0009] The temporal features, spatial features, and paradigm semantic features are fused to obtain the fused features;

[0010] The fused features are input into a fully connected layer to calculate the classification result label of the eye image using a combination function. The classification result label is used to represent the emotion change result, and the classification result label includes a first emotion change label and a second emotion change label.

[0011] Output the classification results based on the classification result labels.

[0012] Optionally, acquiring the eye image, optical flow image, and paradigm image includes:

[0013] Obtain eye-related videos and paradigm videos;

[0014] The eye video is segmented into frames to obtain several eye images;

[0015] Optical flow is calculated for each pair of adjacent eye images after frame segmentation to obtain an optical flow image;

[0016] The paradigm video is processed by frame segmentation to obtain several paradigm images; the paradigm images include: neutral stimulus images, positive stimulus images, and negative stimulus images.

[0017] Optionally, the step of inputting the eye image, the optical flow image, and the paradigm image into the three-stream convolutional neural network model further includes:

[0018] The eye image, the optical flow image, and the paradigm image are preprocessed to obtain a preprocessed image, wherein the preprocessing is an enhancement process.

[0019] Optionally, the method further includes:

[0020] Construct an eye image dataset and a paradigm image dataset;

[0021] Input the eye image dataset and the paradigm image dataset into the pre-trained model;

[0022] Iterative training is performed on samples in the eye image dataset and the paradigm image dataset to extract the temporal and spatial features of the eye image dataset and generate the potential code of the paradigm image dataset;

[0023] Output the predicted label of the eye image based on the temporal features, the spatial features, and the latent code;

[0024] The loss of the model parameters is calculated using a loss function based on the predicted labels;

[0025] The model parameters are updated using the loss to obtain a three-stream convolutional neural network model.

[0026] Optionally, the construction of the eye image dataset and the paradigm image dataset includes:

[0027] A predetermined number of the eye images and the optical flow images are selected and processed equally to obtain several sets of eye image datasets;

[0028] A set number of the paradigm images are selected and processed equally to obtain several sets of paradigm image datasets.

[0029] Optionally, the loss function is the cross-entropy loss function, and the loss of the model parameters is calculated using the loss function based on the predicted labels according to the following formula:

[0030]

[0031] Where L is the loss, L i For the loss of the i-th model parameter, y i The image is labeled with a positive value of 1 (normal emotional change) and a negative value of 0 (abnormal emotional change). i Let N be the probability that the i-th eye image is predicted as positive, and N be the number of eye images.

[0032] Optionally, the spatial module includes an input channel, which is an RGB channel;

[0033] The input channels of the time module include a slow frame rate optical flow channel and a fast frame rate optical flow channel;

[0034] The paradigm module includes three paradigm convolutional layers, two paradigm fully connected layers, and a BatchNorm layer. The BatchNorm layer is used to normalize the outputs of the paradigm convolutional layers and the paradigm fully connected layers.

[0035] Optionally, the optical flow density of the fast frame rate optical flow channel is 8 times that of the slow frame rate optical flow channel.

[0036] Optionally, the BatchNorm layer normalizes the outputs of the paradigmatic convolutional layer and the paradigmatic fully connected layer according to the following formula:

[0037]

[0038] Among them, y paradigm′Let x be the normalized result of the normalized image, E|x| be the mean of the normalized image, Var[x] be the variance of the normalized image, ε be the variable, and γ and β be the linear transformations.

[0039] A second aspect of this application provides an eye imaging emotion detection system based on a three-stream convolutional neural network, applied to the eye imaging emotion detection method based on a three-stream convolutional neural network provided in the first aspect above. The system includes:

[0040] Acquisition Unit: Used to acquire eye images, optical flow images, and paradigm images. The eye images include eye movement behavior images, the optical flow images are motion relationship images between two of the eye images, and the paradigm images are images from the paradigm video.

[0041] Input unit: for inputting the eye image, the optical flow image, and the paradigm image into a three-stream convolutional neural network model, wherein the three-stream convolutional neural network model is a neural network model trained using sample images; the sample images include eye images with training labels and the paradigm image; the three-stream convolutional neural network model includes a spatial module, a temporal module, and a paradigm module; and inputting fused features into a fully connected layer;

[0042] Extraction unit: used to extract the temporal features, spatial features, and paradigm semantic features of the paradigm image using the three-stream convolutional neural network model;

[0043] Fusion unit: used to fuse the temporal features, spatial features, and paradigm semantic features to obtain fused features;

[0044] Calculation unit: used to calculate the classification result label of the eye image using a combination function, the classification result label is used to represent the emotion change result, the classification result label includes a first emotion change label and a second emotion change label;

[0045] Output unit: Used to output the classification result based on the classification result label.

[0046] As can be seen from the above technical solution, this application provides a method and system for emotion detection in eye imaging based on a three-stream convolutional neural network. The method includes: acquiring an eye image, an optical flow image, and a paradigm image; inputting the eye image, the optical flow image, and the paradigm image into a three-stream convolutional neural network model; extracting temporal features, spatial features of the eye image, and paradigm semantic features of the paradigm image using the three-stream convolutional neural network model; fusing the temporal features, spatial features, and paradigm semantic features to obtain fused features; inputting the fused features into a fully connected layer to calculate the classification result label of the eye image using a combination function; and outputting the classification result label based on the classification result label. This method, by acquiring the temporal features, spatial features, and paradigm semantic features of the eye image, can obtain more accurate eye change features from the eye image, thereby detecting individual emotional changes, improving the accuracy of the detection results, and solving the problem of inaccurate detection results when detecting individual emotions. Attached Figure Description

[0047] To more clearly illustrate the technical solution of this application, the drawings used in the embodiments will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0048] Figure 1 A flowchart of the eye imaging emotion detection method based on a three-stream convolutional neural network provided in this application;

[0049] Figure 2 A schematic diagram illustrating the composition of the paradigm video provided in this application;

[0050] Figure 3 A schematic diagram of the algorithm code for training the model provided in this application;

[0051] Figure 4 The architecture diagram of the three-stream convolutional neural network provided in this application;

[0052] Figure 5 A schematic diagram illustrating the grid search algorithm provided in this application for finding the optimal frame rate;

[0053] Figure 6 The paradigm module framework diagram provided for this application. Detailed Implementation

[0054] The embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described below do not represent all embodiments consistent with this application. They are merely examples of systems and methods consistent with some aspects of this application as detailed in the claims.

[0055] Since eye movement patterns and pupillary changes can serve as biomarkers of emotion, individual emotional changes can be detected by analyzing these patterns. However, in eye-movement behavior recognition, some undefined eye changes are ignored, causing the recognition process to lose important information from the eye images. Furthermore, the performance of some algorithms heavily relies on the quality and selection of eye movement feature design; different emotional changes require different features, necessitating significant manpower and time for feature design and optimization. This leads to inaccurate detection results when using fundus imaging to detect an individual's emotions.

[0056] To address the issue of inaccurate emotion detection results when detecting individual emotions, some embodiments of this application provide an eye-based emotion detection method based on a Three-Stream Convolutional Neural Network (TSCNN). See [link to relevant documentation]. Figure 1 , Figure 1 The flowchart illustrates the eye imaging emotion detection method based on a three-stream convolutional neural network provided in this application. The eye imaging emotion detection method based on a three-stream convolutional neural network provided in this application includes:

[0057] S100: Acquire eye images, optical flow images, and paradigm images.

[0058] In some embodiments, the eye image, optical flow image, and paradigm image can be obtained respectively through the following methods:

[0059] S110: Acquire eye video and acquire paradigm video.

[0060] An eye tracker can be used to capture video of an individual's eye movements. This video, in essence, is a video of changes in an individual's eye movements, or eye-movement behavior. For example, the captured video might have a frame rate of approximately 200 frames per second, a frame width of 320, and a frame height of 200. It's important to note that eye-movement behavior refers to the changes in eye movements an individual produces while watching a paradigm video. Different individuals exhibit different eye-movement responses to neutral, negative, and positive stimuli; that is, different emotional stimuli elicit corresponding eye-movement behaviors. The paradigm video is used to stimulate individuals to evoke different emotions, thereby producing different eye-movement behaviors. In some embodiments, the paradigm video can be acquired by connecting the eye tracker to a computer host.

[0061] See Figure 2 , Figure 2 The diagram illustrates the composition of the paradigm video provided in this application. The paradigm video is played according to the following process: Playing 5 seconds of imageless video, i.e., playing 5 seconds of fixed image. Playing a total of 25 seconds of neutral stimulus images, with one neutral stimulus image played every 5 seconds; playing 5 seconds of fixed image. Playing a total of 25 seconds of positive stimulus images, with one positive stimulus image played every 5 seconds; playing 5 seconds of fixed image. Playing a total of 25 seconds of neutral stimulus images, with one neutral stimulus image played every 5 seconds; playing 5 seconds of fixed image. Playing a total of 25 seconds of negative stimulus images, with one negative stimulus image played every 5 seconds.

[0062] S120: Perform frame-by-frame processing on the eye video to obtain several eye images.

[0063] Understandably, the acquired eye images are primarily images of eye movement behavior. By segmenting the eye video into frames, several eye images can be obtained.

[0064] S130: Calculate the optical flow for each pair of adjacent frames of the eye image after framing to obtain an optical flow image.

[0065] An optical flow image is a representation of the motion relationship between two eye images. To improve the generality of the model and the diversity of samples, eye images are extracted frame by frame in this application. In some embodiments, for each pair of adjacent frames, the optical flow can be calculated using the calc Optical Flow Farneback module in OpenCV to obtain an optical flow image.

[0066] S140: Perform frame-by-frame processing on the paradigm video to obtain several paradigm images.

[0067] Paradigm images are images from a paradigm video, including neutral stimulus images, positive stimulus images, and negative stimulus images. Several paradigm images can be obtained by segmenting the paradigm video into frames.

[0068] In some embodiments, after obtaining the eye image and the paradigm image, in order to obtain more eye image data, optical flow image data, and paradigm image data, it is necessary to preprocess the eye image, optical flow image, and paradigm image to obtain a preprocessed image. The preprocessing method used in this embodiment can be enhancement processing. Enhancement processing increases the amount of relevant data in the dataset by making minor changes to the eye image, optical flow image, and paradigm image, such as rotation or shifting. It should be noted that before preprocessing, the eye image, optical flow image, and paradigm image can be uniformly adjusted to the same size, for example, 224×224, to facilitate preprocessing.

[0069] S200: Input the eye image, optical flow image, and paradigm image into the three-stream convolutional neural network model.

[0070] After acquiring the eye image, optical flow image, and paradigm image, these images can be input into a three-stream convolutional neural network model. It is understood that the input eye image, optical flow image, and paradigm image in this step are preprocessed images. In some embodiments, to make the model's results more accurate, the model needs to be trained before use; that is, the three-stream convolutional neural network model in this step is a neural network model trained using sample images, including eye images and paradigm images with training labels. See also... Figure 3 , Figure 3 This application provides a schematic diagram of the algorithm code for training the model. Training the neural network model using sample images to obtain a three-stream convolutional neural network model includes the following steps:

[0071] S210: Construct an eye image dataset and a paradigm image dataset.

[0072] The construction of the eye image dataset and the paradigm image dataset includes the following steps:

[0073] S211: Select a set number of eye images and optical flow images respectively, and process them equally to obtain several sets of eye image datasets.

[0074] For example, the first 8000 eye images and 8000 optical flow images of the frame-segmented eye video can be divided into 5 groups, each containing 1600 eye images and 1600 optical flow images. Then, each group's images can be shuffled and divided into 5 smaller samples, resulting in a dataset of 25 eye images. This dataset can be defined as D1 = (X... O ).

[0075] S212: Select a set number of paradigm images and process them equally to obtain several sets of paradigm image datasets.

[0076] We can select 8000 paradigm images after frame-segmentation, divide them into 5 groups of 1600 paradigm images each, and then shuffle each group into 5 smaller samples to obtain 25 paradigm image datasets. These paradigm image datasets can be defined as D2 = (X... P ).

[0077] S220: Input the eye image dataset and the paradigm image dataset into the pre-trained model.

[0078] S230: Perform iterative training on samples from the eye image dataset and the paradigm image dataset to extract the temporal and spatial features of the eye image dataset, as well as the potential code to generate the paradigm image dataset.

[0079] It should be noted that the training in this embodiment is performed within the PyTorch framework, and the initial learning rate of the pre-trained model is 0.001. Furthermore, to improve the model's generalization ability and prevent overfitting, a dropout strategy can be used to train the pre-trained model.

[0080] In some embodiments, multiple Epoche iterations can be performed on samples from the eye image dataset and the paradigm image dataset, for example, K iterations, to extract the temporal and spatial features of the eye image dataset, as well as the latent code for generating the paradigm image dataset. The temporal features are defined as follows: Spatial features are defined as Potential code is defined as

[0081] S240: Outputs predicted labels for eye images based on temporal features, spatial features, and latent codes.

[0082] The predicted label is the classification prediction result of the emotion expressed by the eye image. However, this result may not be accurate, and further optimization of the predicted label result is needed, that is, to calculate the loss of the model parameters.

[0083] S250: Calculate the loss of model parameters using a loss function based on the predicted labels.

[0084] In some embodiments, the loss function may be the cross-entropy loss function, which is used to calculate the loss of the model parameters based on the predicted labels according to the following formula:

[0085]

[0086] Where L is the loss, L i For the loss of the i-th model parameter, y iThe image is labeled with a positive value of 1 (normal emotional change) and a negative value of 0 (abnormal emotional change). i Let N be the probability that the i-th eye image is predicted as positive, and N be the number of eye images.

[0087] S260: Update the model parameters using the loss function to obtain a three-stream convolutional neural network model.

[0088] See Figure 4 , Figure 4 This is a diagram of the architecture of the three-stream convolutional neural network provided in this application. The three-stream convolutional neural network model includes a spatial module, a temporal module, and a paradigm module. The spatial module is used to extract spatial features from the eye image. Since the eye image is an image of a single eye, the background information is relatively simple, so spatial information can be learned using a single frame. In some embodiments, ResNet50 can be selected as the network for the spatial module. The spatial module includes one input channel, which is an RGB channel. That is, the input of the spatial module is an RGB format eye image, and the output is the spatial features.

[0089] The temporal module is used to extract temporal features from eye images. To enable the temporal module to include optical flow images at different time intervals, it learns the individual's eye changes over different time intervals in channels at different frame rates. The input channels of the temporal module include a slow frame rate optical flow channel and a fast frame rate optical flow channel. Assume the fast frame rate optical flow channel is sampled after time T, and the slow frame rate optical flow channel is sampled after time αT, where α > 1, meaning the optical flow density of the fast frame rate optical flow channel is α times that of the slow frame rate optical flow channel. To find the optimal fast and slow frame rates for classification, a grid search algorithm can be used to find the most suitable frame rate. See [link to relevant documentation]. Figure 5 , Figure 5 The diagram illustrates the grid search algorithm provided in this application for finding the optimal frame rate. When the frame rates of both optical flows are multiples of 8, higher accuracy can be obtained, that is, the optical flow density of the fast frame rate optical flow channel is 8 times that of the slow frame rate optical flow channel, i.e., the value of α is 8.

[0090] In some embodiments, ResNet50 can be selected as the network for the temporal module. It should be noted that, unlike the spatial module, the temporal module takes as input optical flow images stacked according to frame rates and outputs temporal features. The feature maps for fast and slow frame rates output by the temporal module are defined as follows: and By concatenating these elements, the time characteristics output by the time module can be represented as follows:

[0091]

[0092] Where, x temporal For time characteristics, Concat is a combined function.

[0093] See Figure 6 , Figure 6 The diagram below illustrates the framework of the paradigm module provided in this application. The paradigm module utilizes an encoder to extract latent codes from paradigm images viewed by an individual with different stimuli, and a decoder to output paradigm semantic features. By adding the semantic features of the paradigm images as prior knowledge, the paradigm module can provide more informational features, thereby improving the accuracy of emotion detection. The paradigm module includes three paradigm convolutional layers and two paradigm fully connected layers, and can utilize mean squared error loss (MSEloss) to optimize the latent variables. In this network structure, the ReLU activation function can be used. It is understood that the input to the paradigm module is a paradigm image in RGB format, and the output is the paradigm semantic features.

[0094] To accelerate model convergence, the paradigm module also includes a BatchNorm layer, which normalizes the outputs of the paradigm convolutional and fully connected layers. The BatchNorm layer normalizes the outputs of the paradigm convolutional and fully connected layers according to the following formula:

[0095]

[0096] Among them, y paradigm′ Let E|x| be the normalized result of the normalized image, x be the normalized image, E|x| be the mean of the normalized image, Var[x] be the variance of the normalized image, ε be the variable, and γ and β be the linear transformations of the input.

[0097] S300: Utilizes a three-stream convolutional neural network model to extract temporal features, spatial features, and paradigmatic semantic features of eye images.

[0098] For each set, the input values ​​for the temporal module, spatial module, and normalization module are defined as p, q, and l, respectively, resulting in three feature maps. The extracted temporal features, spatial features, and paradigmatic semantic features of the eye image are as follows:

[0099]

[0100] S400: Integrates temporal features, spatial features, and paradigmatic semantic features to obtain integrated features.

[0101] Flatten the output temporal features, spatial features, and paradigm semantic features:

[0102]

[0103] in, The time characteristics after flattening The spatial characteristics after flattening The flattened paradigm semantic features.

[0104] Will The concatenated features are obtained, and the fused feature vector can be represented as: Among them, y n′ This is a feature of fusion.

[0105] S500: Input the fused features into the fully connected layer to use the combination function to calculate the classification result label of the eye image.

[0106] In some embodiments, the combination function may be a Concat function. The calculated classification result labels are used to represent the sentiment change results, and the classification result labels may be represented as follows: in For the classification result labels, FC is a fully connected layer. The classification result labels include a first emotion change label and a second emotion change label, where the first emotion change can be a normal emotion change, for example, an individual who has an emotional response to the stimulus of the paradigm image. The second emotion change can be an abnormal emotion change, for example, an individual who has no emotional response to the stimulus of the paradigm image.

[0107] S600: Outputs the classification results based on the classification result labels.

[0108] The classification results output provide information about an individual's emotional changes, indicating whether these changes are normal or abnormal. Individuals with normal emotional changes are considered more prone to emotional fluctuations, while those with abnormal emotional changes are less likely to experience them. These classification results can be used to detect an individual's emotional changes.

[0109] To demonstrate the effectiveness of the detection method in the embodiments of this application, in some embodiments, a dataset containing eye videos of 81 individuals is selected, each eye video being approximately 5 minutes long. Individuals with normal emotional changes in step S600 are defined as the easy set, and individuals with very long emotional changes are defined as the non-easy set. The 81 datasets are divided into 64 training sets (33 non-easy sets, 31 easy sets), 8 validation sets (3 non-easy sets, 5 easy sets), and 9 test sets (5 non-easy sets, 4 easy sets). Some classic methods in the field of video recognition are selected and classified according to the patterns of the input data required by these methods. C3D, R3D, R2Plus1D, and SlowFast models only require eye images as input, two-stream convolutional neural networks require eye images and optical flow images as input, while three-stream convolutional neural networks require a paradigm image in addition to eye images and optical flow images. Table 1 shows the experimental results for different networks. The performance of different substructures is described by mean ± standard deviation. A t-test was used for significance analysis. The three-stream convolutional neural network (containing all types of input) served as the control group. In Table 1, ** indicates extremely significant difference (<0.01), and * indicates significant difference (<0.05).

[0110] Table 1

[0111]

[0112] As can be seen from the results in Table 1, the three-stream convolutional neural network has the highest classification accuracy of 79.3%, precision of 84.0%, F1 score of 75.2%, and specificity of 88.0%. All of these results are higher than the baseline method, indicating that the model performs well and the detection results are relatively accurate.

[0113] This application also provides an eye imaging emotion detection system based on a three-stream convolutional neural network, which is applied to the eye imaging emotion detection method based on a three-stream convolutional neural network provided in the above embodiments. The system includes: an acquisition unit, an input unit, an extraction unit, a fusion unit, a calculation unit, and an output unit.

[0114] Acquisition Unit: Used to acquire eye images and paradigm images. Eye images include eye movement behavior images, and paradigm images are images from paradigm videos.

[0115] Input Unit: Used to input the eye image and the paradigm image into the three-stream convolutional neural network model. The three-stream convolutional neural network model is a neural network model trained using sample images; the sample images include eye images and paradigm images with training labels; the three-stream convolutional neural network model includes a spatial module, a temporal module, and a paradigm module.

[0116] Extraction Unit: Used to extract temporal features, spatial features, and paradigm semantic features of eye images using a three-stream convolutional neural network model.

[0117] Fusion unit: Used to fuse temporal features, spatial features, and paradigmatic semantic features to obtain fused features. After obtaining the fused features, they need to be input into the fully connected layer for computation. It can be understood that the above-mentioned input unit is also used to input the fused features into the fully connected layer.

[0118] Calculation unit: used to calculate the classification result label of the eye image using a combination function. The classification result label is used to represent the emotion change result. The classification result label includes a first emotion change label and a second emotion change label.

[0119] Output unit: Used to output the classification result based on the classification result label.

[0120] As can be seen from the above technical solution, this application provides a method and system for emotion detection in eye imaging based on a three-stream convolutional neural network. The method includes: acquiring an eye image, an optical flow image, and a paradigm image; inputting the eye image, optical flow image, and paradigm image into a three-stream convolutional neural network model; extracting temporal features, spatial features of the eye image, and paradigm semantic features of the paradigm image using the three-stream convolutional neural network model; fusing the temporal features, spatial features, and paradigm semantic features to obtain fused features; inputting the fused features into a fully connected layer to calculate the classification result label of the eye image using a combination function; and outputting the classification result based on the classification result label. This method, by acquiring the temporal features, spatial features, and paradigm semantic features of the eye image, can obtain more accurate eye change features from the eye image, thereby detecting individual emotion changes, improving the accuracy of the detection results, and solving the problem of inaccurate detection results when detecting individual emotions.

[0121] Similar parts between the embodiments provided in this application can be referred to mutually. The specific implementation methods provided above are only a few examples under the overall concept of this application and do not constitute a limitation on the scope of protection of this application. For those skilled in the art, any other implementation methods extended from the solution of this application without creative effort shall fall within the scope of protection of this application.

Claims

1. A method for emotion detection in eye imaging based on a three-stream convolutional neural network, characterized in that, include: Acquire eye images, optical flow images, and paradigm images. The eye images include eye movement behavior images, the optical flow images are motion relationship images between two of the eye images, and the paradigm images are images from the paradigm video. The eye image, the optical flow image, and the paradigm image are input into a three-stream convolutional neural network model, which is a neural network model trained using sample images; the sample images include eye images with training labels and the paradigm image; the three-stream convolutional neural network model includes a spatial module, a temporal module, and a paradigm module; The temporal and spatial features of the eye image and the paradigm semantic features of the paradigm image are extracted using the three-stream convolutional neural network model. The temporal features, spatial features, and paradigm semantic features are fused to obtain the fused features; The fused features are input into a fully connected layer to calculate the classification result label of the eye image using a combination function. The classification result label is used to represent the emotion change result, and the classification result label includes a first emotion change label and a second emotion change label. Output the classification results based on the classification result labels.

2. The eye imaging emotion detection method based on a three-stream convolutional neural network according to claim 1, characterized in that, The acquisition of eye images, optical flow images, and paradigm images includes: Obtain eye-related videos and paradigm videos; The eye video is segmented into frames to obtain several eye images; Optical flow is calculated for each pair of adjacent eye images after frame segmentation to obtain an optical flow image; The paradigm video is processed by frame segmentation to obtain several paradigm images; the paradigm images include: neutral stimulus images, positive stimulus images, and negative stimulus images.

3. The eye imaging emotion detection method based on a three-stream convolutional neural network according to claim 1, characterized in that, The step of inputting the eye image, the optical flow image, and the paradigm image into the three-flow convolutional neural network model further includes: The eye image, the optical flow image, and the paradigm image are preprocessed to obtain a preprocessed image, wherein the preprocessing is an enhancement process.

4. The eye imaging emotion detection method based on a three-stream convolutional neural network according to claim 1, characterized in that, The method further includes: Construct an eye image dataset and a paradigm image dataset; Input the eye image dataset and the paradigm image dataset into the pre-trained model; Iterative training is performed on samples in the eye image dataset and the paradigm image dataset to extract the temporal and spatial features of the eye image dataset and generate the potential code of the paradigm image dataset; Output the predicted label of the eye image based on the temporal features, the spatial features, and the latent code; The loss of the model parameters is calculated using a loss function based on the predicted labels; The model parameters are updated using the loss to obtain a three-stream convolutional neural network model.

5. The eye imaging emotion detection method based on a three-stream convolutional neural network according to claim 4, characterized in that, The construction of the eye image dataset and the paradigm image dataset includes: A predetermined number of the eye images and the optical flow images are selected and processed equally to obtain several sets of eye image datasets; A set number of the paradigm images are selected and processed equally to obtain several sets of paradigm image datasets.

6. The eye imaging emotion detection method based on a three-stream convolutional neural network according to claim 4, characterized in that, The loss function is the cross-entropy loss function, and the loss of the model parameters is calculated using the loss function based on the predicted labels according to the following formula: Where L is the loss, L i For the loss of the i-th model parameter, y i The images of the eyes are labeled, with 1 for positive (normal emotional change) and 0 for negative (abnormal emotional change). i Let N be the probability that the i-th eye image is predicted as positive, and N be the number of eye images.

7. The eye imaging emotion detection method based on a three-stream convolutional neural network according to claim 1, characterized in that, The spatial module includes an input channel, which is an RGB channel. The input channels of the time module include a slow frame rate optical flow channel and a fast frame rate optical flow channel; The paradigm module includes three paradigm convolutional layers, two paradigm fully connected layers, and a BatchNorm layer. The BatchNorm layer is used to normalize the outputs of the paradigm convolutional layers and the paradigm fully connected layers.

8. The eye imaging emotion detection method based on a three-stream convolutional neural network according to claim 7, characterized in that, The optical flow density of the fast frame rate optical flow channel is 8 times that of the slow frame rate optical flow channel.

9. The eye imaging emotion detection method based on a three-stream convolutional neural network according to claim 7, characterized in that, The BatchNorm layer normalizes the outputs of the normalized convolutional layer and the normalized fully connected layer according to the following formula: Among them, y paradigm′ Let x be the normalized result of the normalized image, E|x| be the mean of the normalized image, Var[x] be the variance of the normalized image, ε be the variable, and γ and β be the linear transformations.

10. An eye-imaging emotion detection system based on a three-stream convolutional neural network, characterized in that, The system applied to the eye imaging emotion detection method based on a three-stream convolutional neural network as described in any one of claims 1-9, the system comprising: Acquisition Unit: Used to acquire eye images, optical flow images, and paradigm images. The eye images include eye movement behavior images, the optical flow images are motion relationship images between two of the eye images, and the paradigm images are images from the paradigm video. Input unit: for inputting the eye image, the optical flow image, and the paradigm image into a three-stream convolutional neural network model, wherein the three-stream convolutional neural network model is a neural network model trained using sample images; the sample images include eye images with training labels and the paradigm image; the three-stream convolutional neural network model includes a spatial module, a temporal module, and a paradigm module; and inputting fused features into a fully connected layer; Extraction unit: used to extract the temporal features, spatial features, and paradigm semantic features of the paradigm image using the three-stream convolutional neural network model; Fusion unit: used to fuse the temporal features, spatial features, and paradigm semantic features to obtain fused features; Calculation unit: used to calculate the classification result label of the eye image using a combination function, the classification result label is used to represent the emotion change result, the classification result label includes a first emotion change label and a second emotion change label; Output unit: Used to output the classification result based on the classification result label.

Images