Progressive self-distillation emotional state estimation method based on viewpoint and environment interaction

Through the progressive self-distillation emotional state estimation method of viewpoint interaction with the environment, combined with deep neural network and self-distillation mechanism, the problem of complexity and limited accuracy of data acquisition in traditional methods is solved, and a more efficient and accurate emotional state is achieved. Estimated, suitable for multi-scenario applications.

CN119046762AActive Publication Date: 2024-11-29CHINA UNIV OF PETROLEUM (EAST CHINA)
View PDF 3 Cites 0 Cited by

Patent Information

Application Number
CN202411185190.9
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-27
Publication Date
2024-11-29
Estimated Expiration
2044-08-27

AI Technical Summary

Technical Problem

Traditional emotional state estimation methods rely on physiological signals and facial expressions, and there are problems such as complex data acquisition, high cost and limited accuracy. The method based on eye movement data ignores environmental factors, making it difficult to accurately reflect the individual's emotional state.

Method used

Using a progressive self-distillation emotional state estimation method that interacts with the environment, viewpoint data is collected through high-precision eye tracking equipment, combined with image location and content, and using deep neural network and self-distillation mechanism to gradually optimize the emotional state estimation model to enhance the Accuracy and robustness.

Benefits of technology

It significantly improves the accuracy and robustness of emotional state estimation, is suitable for diverse and subjective emotional state tasks, adapts to different types of image data and display devices, and provides reliable emotional state analysis results.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119046762A_ABST
    Figure CN119046762A_ABST
Patent Text Reader

Abstract

The invention discloses a progressive self-distillation emotional state estimation method for viewpoint and environment interaction, and aims to realize accurate judgment of an emotional state by collecting eyeball viewpoint data of a viewer under different images. The method comprises the following steps of: forming an initial data set by using high-precision eye movement tracking equipment, segmenting an image into a plurality of viewpoint staying areas, and segmenting and sorting according to a staying sequence; then, the preliminary deep learning model is used to carry out difficulty assessment on the image sample so as to support progressive learning. According to the method, a deep neural network is designed, and a prediction result is continuously optimized in combination with a self-distillation mechanism. The model gradually learns from easy to difficult, current-stage learning is guided through high-confidence prediction, and finally the effect is evaluated on an independent verification set. The method effectively improves the accuracy and robustness of emotional state estimation, is suitable for multiple fields of psychological research, intelligent human-computer interaction, user experience optimization and the like, has high precision and multi-scene application capability, and provides a reliable emotional state analysis result.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the field of emotional state estimation, and in particular relates to a progressive self-distillation emotional state estimation method for viewpoint-environment interaction. Background Art

[0002] Emotional state estimation is a key issue in many fields, including psychology research, intelligent human-computer interaction, emotional computing, advertising effectiveness analysis, and user experience optimization. In psychology research, emotional state estimation can help researchers better understand individuals' emotional reactions and psychological states; in intelligent human-computer interaction, accurate emotional state estimation can enable machines to better understand and respond to human emotions, thereby improving the naturalness and effectiveness of human-computer interaction; in emotional computing, emotional state estimation is the basis for machine emotion recognition and emotion generation; in advertising effectiveness analysis, emotional state estimation can help advertisers understand the audience's emotional response to advertising, thereby optimizing advertising content and delivery strategies; in user experience optimization, emotional state estimation can help product designers understand users' emotional feedback, thereby improving product design and enhancing user experience.

[0003] Traditional methods for estimating emotional states mostly rely on explicit features such as physiological signals and facial expressions. Physiological signals such as heart rate, galvanic skin response, and brain waves can reflect an individual's physiological state, thereby indirectly inferring his or her emotional state. However, the acquisition of these physiological signals often requires professional equipment and a strict experimental environment, which is easily disturbed by the external environment, and the data acquisition process is complex and costly. In addition, facial expressions, as another type of explicit feature, although they can more intuitively reflect an individual's emotional state, also have certain limitations. For example, an individual's facial expressions may be affected by factors such as social culture and personal habits, resulting in limited accuracy in emotion recognition. In addition, some emotional states (such as introverted emotions) are not obvious in facial expressions, which further increases the difficulty of recognition.

[0004] In recent years, the estimation of emotional state based on eye tracking data has gradually attracted attention. Eye movement data can reflect the individual's attention distribution and viewpoint movement pattern by recording the viewpoint position, dwell time and other information when the individual views different images. Eye movement data has high temporal and spatial resolution and can provide fine-grained emotional state information. For example, when an individual is interested in an image or has emotional fluctuations, his or her viewpoint may stay in a specific area of ​​the image for a long time, or frequently move between multiple areas of interest. However, the emotional state estimation method based on eye movement data also has some shortcomings. Using only eye movement data for emotional state estimation may ignore the impact of the environment on individual emotions, such as the specific content of the image, the position of the image on the screen and other factors. In addition, eye movement data may not accurately reflect the individual's true emotional state in some cases, such as when the viewer deliberately controls his or her eye movement behavior. Therefore, a more efficient and accurate emotional state estimation method is needed to meet the challenges and needs of emotional state estimation. Summary of the invention

[0005] In order to solve the technical problems existing in the background technology, the present invention aims to provide a progressive self-distillation emotional state estimation method based on the interaction between viewpoint and environment, by collecting and analyzing the eye viewpoint data of people when viewing different types of images on the screen, and combining the image position to accurately determine the emotional state. Compared with the existing methods, the method of the present invention can capture the emotional state of an individual more comprehensively. For example, by analyzing the viewpoint distribution and dwell time of an individual when viewing a specific image, his or her emotional response to the image can be inferred. At the same time, combined with the content and position of the image, the accuracy and robustness of the emotional state estimation can be further improved. In addition, the method based on the interaction between viewpoint and environment can also be applied to more scenarios, such as different types of image data, different display devices and usage environments, so that it has higher adaptability and broad application prospects.

[0006] In order to solve the technical problem, the technical solution of the present invention is:

[0007] A progressive self-distillation emotional state estimation method for viewpoint-environment interaction, the method comprising:

[0008] S1: Use high-precision eye tracking equipment to collect the viewer's eye viewpoint data when viewing different types of images on the screen to obtain training sets and test sets;

[0009] S2: Using the interactive relationship between viewpoints and images, the images in the training set are segmented into multiple viewpoint retention areas, and segmented and sorted according to the viewpoint retention order to obtain a viewpoint area sequence;

[0010] S3: Use the preset viewpoint emotion evaluation model to evaluate the difficulty of the viewpoint data samples in the training set, sort and grade the samples according to the evaluation results, and provide a basis for the progressive learning strategy;

[0011] S4: Build a deep neural network model that combines a self-distillation mechanism to continuously optimize its own prediction results during training, thereby improving the accuracy and reliability of emotional state estimation;

[0012] S5: Gradually learn samples from easy to difficult through a progressive learning strategy. In each progressive learning stage, the model uses a self-distillation mechanism to guide the learning of the current stage through the high-confidence prediction of the previous stage;

[0013] S6: Repeat the above steps S3 and S4 to gradually optimize the performance and generalization ability of the model;

[0014] S7: Use a test set to evaluate the performance and accuracy of the model, where the test set includes eye viewpoint data when a person's eyes view different types of images on the screen, images corresponding to the viewpoints, and corresponding real emotion categories.

[0015] The evaluation method of the present invention introduces a progressive self-distillation learning mechanism to analyze the interaction characteristics between viewpoints and environment, attempting to explore deeper emotional laws in continuous self-iteration and optimization, aiming to make up for the shortcomings of traditional algorithms and better adapt to diverse and subjective emotional state estimation tasks.

[0016] The method of the present invention introduces a self-distillation mechanism, which enables the model to improve its performance through self-learning without a large amount of labeled data. The model uses the high-confidence predictions of the previous stage to guide the learning of the current stage, which enhances the model's ability to handle novel samples and makes its evaluation results more accurate and robust. Through a progressive learning strategy, the model can gradually learn samples from easy to difficult, thereby more effectively mastering emotional features and patterns at different levels.

[0017] The detection method of the present invention also designs a deep neural network architecture, focusing on both the global features and local relational features of the image. Such a design can better capture the overall structure and semantic information of the viewpoint resting area, and improve the accuracy and comprehensiveness of emotional state estimation.

[0018] Further, the step S2 comprises:

[0019] S201: According to the viewpoint data, each image is segmented according to the viewpoint stay position to form multiple viewpoint areas, and a viewpoint stay time threshold is defined. The viewpoint position whose stay time exceeds the threshold is considered as one viewpoint area;

[0020] S202: Sort each viewpoint region in time order to form a viewpoint region sequence.

[0021] Furthermore, the viewpoint stay area segmentation includes segmenting the image according to the viewpoint data, and each image is segmented into a plurality of viewpoint stay areas. Assuming that the resolution of each image is W×H, the viewpoint stay area R is defined as follows:

[0022] R={(x i ,y i )|t i <τ}

[0023] Among them, τ is the threshold of the viewpoint residence time, and only the viewpoint position whose residence time exceeds the threshold is considered as a viewpoint area;

[0024] The ordering of the viewpoint stop areas includes ordering each viewpoint area in time order to form a viewpoint area sequence; assuming that the viewpoint data is ordered in time order as {(t1, x1, y1), (t2, x2, y2), ..., (t N ,x N ,y N )}, then the viewpoint region sequence S is:

[0025] S={(x i1 ,y i1 ),(x i2 ,y i2 ),...,(x ik ,y ik )}

[0026] Among them, i1, i2, ..., ik are viewpoint indices sorted in chronological order.

[0027] Further, the step S3 comprises:

[0028] S301: using an existing pre-trained image emotion assessment model to perform preliminary emotion classification on the viewpoint data in the training set, wherein the preliminary deep learning model is an existing pre-trained viewpoint emotion assessment model MSBLS;

[0029] S302: Based on the preliminary classification results, evaluate the difficulty of each sample, sort and grade the samples for subsequent progressive learning stages.

[0030] Further, the step S4 comprises:

[0031] S401: Design a feature extractor, which is a deep neural network architecture, including multiple convolutional layers, activation layers, and pooling layers, so as to effectively extract global features of the viewpoint area. The feature extractor is constructed based on the convolutional neural network ResNet50 or VGG16 with the fully connected layer removed;

[0032] S402: Design a multi-scale graph convolutional neural network to obtain multi-scale relational features of the global features of the viewpoint region at the deepest level; design a long short-term memory network to perform temporal learning on the extracted features to capture the dynamic relationship between viewpoint regions;

[0033] S403: Design a sentiment evaluator, including two cascaded fully connected layers, to output sentiment categories;

[0034] S404: Introducing a self-distillation mechanism into the deep network architecture of the model, allowing subsequent modules of the model to refer to the training results of the previous modules and use these results as additional guidance signals to promote the model to learn more abstract and complex emotional features; through three distillation loss functions, namely feature distillation loss, relationship distillation loss and response loss, the teacher network is online distilled at the feature, relationship and response levels respectively, and the teacher network knowledge is transferred to the student network.

[0035] Furthermore, the self-distillation mechanism is implemented in the model through the following steps:

[0036] S4041: The feature distillation loss is constructed by: performing feature distillation between the last layer image global features of the feature extractors of the teacher network and the student network to enable the student network to imitate the feature activation pattern of the corresponding layer in the teacher network; this process is completed by minimizing the feature activation difference between the student network and the teacher network, using L2 norm or cosine similarity as the loss function, thereby realizing knowledge transfer at the feature level;

[0037] S4042: The relation distillation loss is constructed in the following manner: for the multi-scale graph structural features of the teacher network and the student network, the relationship between the internal features is calculated, and a specific metric method, such as cosine similarity or Euclidean distance, is used to quantitatively evaluate and compare these relationships; the relation distillation loss function aims to minimize the difference between the multi-scale graph relationship learned by the student network and the multi-scale graph relationship in the teacher network, so that the student network can learn richer structural knowledge;

[0038] S4043: The response loss is constructed by directly comparing the outputs of the teacher network and the student network, distilling the emotion categories output by the last fully connected layer, and evaluating the performance difference between the two networks on the final prediction task by the response loss, which is implemented using soft label cross entropy loss, so that the student network can learn the prediction behavior of the teacher network at the output level.

[0039] Further, the step S5 comprises:

[0040] S501: According to the difficulty level of the samples, the training process is divided into multiple stages, and the samples are gradually learned from easy to difficult through progressive learning stages; specifically, the simpler viewpoint data samples are learned first, and then more complex samples are gradually introduced. In this way, the model can gradually grasp the emotional features and patterns of different levels;

[0041] S502: In each progressive learning stage, the model uses the self-distillation mechanism to guide the learning of the current stage; specifically, the model uses the high-confidence predictions of the previous stage as labels to guide the learning of the current stage, ensuring that the model maintains a certain stability and accuracy when learning new samples, so as to iteratively improve the learning ability of the model.

[0042] Further, the step S7 comprises:

[0043] S701: After each training phase, a validation set with a different distribution from the training samples is used to test the performance of the model. Evaluation indicators include accuracy and F1 score.

[0044] S702: According to the results of the performance evaluation, adjust the hyperparameters of the model, including the learning rate and the regularization coefficient, and optimize the model structure to improve the emotion evaluation performance of the model.

[0045] A computer device comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, any one of the above-mentioned progressive self-distillation emotional state estimation methods for viewpoint-environment interaction is implemented.

[0046] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any of the above-mentioned progressive self-distillation emotional state estimation methods for viewpoint-environment interaction.

[0047] Compared with the prior art, the advantages of the present invention are:

[0048] Improve the accuracy of emotional state estimation: The invention can more comprehensively capture the emotional state of an individual by combining eye tracking data with environmental factors such as image content and location. By collecting the viewer's viewpoint data through high-precision eye tracking equipment and combining it with image segmentation and sorting, the emotional state estimation not only relies on single eye movement data, but also comprehensively considers the interactive relationship between viewpoint and image, thereby significantly improving the accuracy of emotional state estimation;

[0049] Enhance the robustness of the model: Traditional methods are easily affected by environmental interference, but this invention designs a progressive learning strategy and self-distillation mechanism to enable the model to gradually learn and optimize its own prediction results. In each progressive learning stage, the high-confidence prediction results of the previous stage are used to guide the learning of the current stage, ensuring that the model can still maintain stable prediction performance when facing complex samples, thereby enhancing the robustness of the model in different environments and samples;

[0050] Achieve multi-scenario application capabilities: The method of the present invention is applicable to various types of image data and display devices and has high adaptability. Through the segmentation and sorting of viewpoint areas, and the combination of multi-scale graph convolutional neural networks and long short-term memory networks, the model can effectively extract and learn the global features and dynamic relationships of viewpoint areas, and adapt to the needs of emotional state estimation in different scenarios. Therefore, the present invention has broad application prospects in many fields such as psychology research, intelligent human-computer interaction, emotional computing, advertising effect analysis, and user experience optimization;

[0051] Providing reliable emotional state analysis results: The present invention adopts a progressive learning strategy to gradually learn samples from easy to difficult, and gradually optimizes the performance and generalization ability of the model through the self-distillation mechanism of each learning stage. During the training process, by repeatedly cycling the above learning and optimization steps, the model can fully grasp the emotional characteristics and patterns at different levels. Finally, an evaluation is performed on an independent validation set or test set to verify the performance and accuracy of the model, ensuring the reliability and high precision of the emotional state analysis results.

[0052] In summary, the present invention significantly improves the accuracy, robustness, adaptability and reliability of emotional state estimation by comprehensively utilizing eye movement data and environmental interaction information, designing a progressive learning strategy and self-distillation mechanism, and has broad application prospects. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 This is a framework diagram of the progressive self-distillation emotional state estimation method based on the interaction between viewpoint and environment;

[0054] Figure 2 Schematic diagram of a deep neural network model combined with a self-distillation mechanism;

[0055] Figure 3 Schematic diagram of the multi-scale graph convolution structure. DETAILED DESCRIPTION

[0056] The specific implementation mode of the present invention is described below in conjunction with embodiments:

[0057] It should be noted that the structures, proportions, sizes, etc. shown in this specification are only used to match the contents disclosed in the specification so that people familiar with this technology can understand and read them, and are not used to limit the conditions under which the present invention can be implemented. Any structural modification, change in proportional relationship or adjustment of size should still fall within the scope of the technical content disclosed in the present invention without affecting the effects and purposes that can be achieved by the present invention.

[0058] At the same time, the terms such as "upper", "lower", "left", "right", "middle" and "one" cited in this specification are only for the convenience of description and are not used to limit the scope of implementation of the present invention. Changes or adjustments to their relative relationships should be regarded as the scope of implementation of the present invention without substantially changing the technical content.

[0059] Embodiment 1:

[0060] Figure 1 The flowchart of some embodiments of the progressive self-distillation emotional state estimation method based on viewpoint-environment interaction according to the present disclosure includes the following steps:

[0061] S1: The eye viewpoint data of viewers when viewing different types of images on the screen are collected through high-precision eye tracking equipment to form an initial data set.

[0062] Preferably, the process of collecting viewpoint data using a high-precision eye tracking device includes: device selection, experimental environment preparation, image presentation, and viewpoint data collection.

[0063] Among them, the device selection includes selecting high-precision eye tracking devices, such as Tobii Pro series eye trackers, which can record the viewer's eye movement data at a high sampling frequency (e.g., 500 Hz or higher) and a high spatial resolution (e.g., 0.5 degrees or less); the experimental environment preparation includes installing the eye tracking device in a stable laboratory environment, ensuring that the device is calibrated correctly, and avoiding changes in ambient light and other interference factors. The subject (viewer) needs to sit at a fixed distance from the screen, usually 60-70 cm, to ensure data consistency; the image presentation includes presenting different types of images on the screen, which may include natural scenery, human faces, complex scenes, etc. The display time of each image should be long enough so that the viewer has enough time to browse the image content, usually set to 5-10 seconds; the viewpoint data collection includes the eye tracking device recording the viewpoint position (screen coordinates of the eye gaze) and the viewpoint dwell time when the viewer browses each image. The collected data includes timestamp, X and Y coordinates of the viewpoint, and dwell time.

[0064] Furthermore, in a specific embodiment, the present invention relates to a dataset named EmoGaze360-1K, which is intended to provide high-quality panoramic image resources for emotion recognition research. The dataset contains the viewpoint information of human eyes when viewing panoramic images under six emotions (calm, happy, disgusted, sad, angry, and afraid). The construction of the dataset follows the following technical scheme: (1) Data collection: 1,000 panoramic images were collected from public platforms such as 360cities and Flickr, which truly reflect the diverse indoor and outdoor environments. (2) Selection criteria: According to the standards of the PANDORA project, the present invention considers three key factors when selecting images: the authenticity of the image, the diversity of people and objects contained in each image, and the mixed nature of the indoor and outdoor environments. (3) Image quality: All images have a high resolution of 1920×960, ensuring the clarity of the image and the richness of the visual information. (4) Dataset features: Compared with the prior art, EmoGaze360-1K not only includes eye tracking data, but also integrates contextual panoramic images to form a dual-focus design, which helps to deeply analyze the complex relationship between visual attention and emotional state. (5) Emotion-inducing resources: In order to effectively collect eye tracking data, the present invention specially designs 500 emotion-inducing images and 50 emotion-inducing videos, providing 2,500 images and 250 videos for each emotional state. These resources can effectively stimulate the user's corresponding emotional response before the eye tracking data is collected. Through the above technical solution, the EmoGaze360-1K dataset not only improves the data quality and diversity of emotion recognition research, but also provides a new perspective and research basis for the development of emotion recognition technology by integrating visual stimulation and eye tracking data. At the same time, the present invention uses k-fold cross-validation to train the model. The dataset is randomly divided into k non-overlapping subsets of the same size, of which k-1 subsets are used as training sets to train the model, and the remaining subsets are used as test sets to test the model. The present invention sets k=5 here.

[0065] S2: Using the interactive relationship between viewpoints and images, the image is segmented into multiple viewpoint retention areas, and the image is segmented and sorted according to the viewpoint retention order.

[0066] In a specific implementation, the viewpoint region segmentation includes segmenting the image according to the viewpoint data, and each image is segmented into a plurality of viewpoint retention regions. Assuming that the resolution of each image is W×H, the viewpoint retention region R is defined as follows:

[0067] R={(x i ,y i )|t i <τ}

[0068] Among them, τ is the threshold of the viewpoint residence time, and only the viewpoint position whose residence time exceeds the threshold is considered as a viewpoint area.

[0069] Furthermore, the ordering of the viewpoint regions includes ordering each viewpoint region in time order to form a viewpoint region sequence. Assume that the viewpoint data is ordered in time order as {(t1,x1,y1),(t2,x2,y2),…,(t N ,x N ,y N )}, then the viewpoint region sequence S is:

[0070] S={(x i1 ,y i1 ),(x i2 ,y i2 ),...,(x ik ,y ik )}

[0071] Among them, i1, i2, ..., ik are viewpoint indices sorted in chronological order.

[0072] S3: Use a preliminary deep learning model to evaluate the difficulty of the viewpoint data in the training set, sort and grade the samples according to the evaluation results, and provide a basis for the progressive learning strategy.

[0073] In a specific implementation, the preliminary deep learning model is an existing pre-trained viewpoint emotion assessment model MSBLS.

[0074] S31: Use the existing trained viewpoint emotion assessment model to perform preliminary emotion classification on the images in the training set.

[0075] In a more specific implementation, the viewpoint emotion category is obtained by inputting the collected viewpoint data into an existing pre-trained viewpoint emotion assessment model MSBLS to obtain a preliminary viewpoint emotion category.

[0076] S32: Based on the preliminary classification results, evaluate the difficulty of each sample, sort and grade the samples for the subsequent progressive learning stage.

[0077] In a more specific embodiment, sample difficulty can be defined as: the confidence output by the model during preliminary emotion classification. Samples with lower confidence mean that the model is not sure enough about its classification result, and therefore can be considered to be of higher difficulty. Therefore, in the present invention, samples with a confidence of the preliminary viewpoint emotion category greater than 0.7 (including 0.7) are simple / easy samples, samples with a confidence of the preliminary viewpoint emotion category less than 0.3 (including 0.3) are complex / difficult samples, and samples with a confidence of the preliminary viewpoint emotion category between 0.3 and 0.7 are medium samples. Based on this, a simple / easy sample training set, a medium sample training set, and a complex / difficult sample training set are constructed.

[0078] S4: Design a deep neural network model that combines a self-distillation mechanism to continuously optimize its own prediction results during training, thereby improving the accuracy and reliability of emotional state estimation.

[0079] In a specific embodiment, referring to the attached Figure 2 The deep neural network model combined with the self-distillation mechanism includes a feature extractor, a multi-scale graph convolutional neural network, a long short-term memory network and a sentiment evaluator.

[0080] Furthermore, the deep neural network model combined with the self-distillation mechanism includes:

[0081] S41: Design a feature extractor, which is a deep neural network architecture consisting of multiple convolutional layers, activation layers, and pooling layers to effectively extract global features of the viewpoint rest area;

[0082] In a specific embodiment, the feature extractor is a deep neural network architecture, comprising multiple convolutional layers, activation layers and pooling layers, so as to effectively extract the global features of the viewpoint rest area.

[0083] Furthermore, the feature extractor is constructed based on a convolutional neural network ResNet50 or VGG16 with the fully connected layer removed.

[0084] S42: Design a multi-scale graph convolutional neural network to obtain the multi-scale relational features of the global features of the image at the deepest level; at the same time, design a long short-term memory network to perform temporal learning on the extracted features to capture the dynamic relationship between viewpoint areas.

[0085] In a specific embodiment, the multi-scale graph convolutional neural network is constructed in the following manner:

[0086] Use the global image feature V1 obtained by S31 to construct a graph G = (V1, E1). Where V = {v1, ..., vn} represents a set of nodes, and E1 is a set of edges. Each node vi{i∈[1,n]} in the graph corresponds to a preset scale and a connected feature map, and is composed of N nodes with (1×D) features. Each node has its own unique features, which are represented by an N×D matrix X, where N is the total number of nodes and D is the dimension of the input feature. The relationship between nodes can be represented by an adjacency matrix A, which is a sparse N×N matrix.

[0087] Further, see the attached Figure 3 , a U-shaped network architecture is used to process message passing between nodes. The left half performs graph convolution and downsampling operations, while the right half performs upsampling and graph convolution operations. By performing graph convolution-downsampling-upsampling-graph convolution operations at multiple scales, not only the deep semantic information of high-level features is retained, but also the fine local information provided by the underlying features is retained.

[0088] In a specific embodiment, the graph convolution is implemented in the following way:

[0089] H l+1 =f(H l ,A)=σ(AH l W l )

[0090] The input of each graph convolution layer consists of the adjacency matrix A and the node features H l W is composed of, l represents the number of the layer, f(·) represents the nonlinear function, and represents the rule of cross-layer propagation. l represents the weight matrix of the l-th layer of the neural network, and σ(·) is a nonlinear activation function, such as ReLU.

[0091] Furthermore, in a specific embodiment, the long short-term memory network is constructed in the following manner:

[0092] The feature sequence extracted by the multi-scale graph convolutional network is used as input to the long short-term memory network (LSTM network) for time series learning. The input of LSTM is X t =H t (l) , where H t (l) is the lth layer feature extracted by GCN. The update formula of LSTM is:

[0093] f t =σ(W f ·[h t-1 ,X t ]+bf )

[0094] i t =σ(W i ·[h t-1 ,X t ]+b i )

[0095] c t =f t ⊙c t-1 +i t ⊙tanh(+b c )

[0096] o t =σ(W o ·[h t-1 ,X t ]+b o )

[0097] h t =o t ⊙tanh(c t )

[0098] Among them, f t ,i t ,c t ,o t They represent the forget gate, input gate, cell state and output gate respectively, and σ represents the activation function.

[0099] S43: Design a sentiment evaluator consisting of two cascaded fully connected layers to output sentiment categories.

[0100] In a specific embodiment, the emotion evaluator can be constructed by two connected fully connected layers, and the ReLU activation function is used between the two fully connected layers to introduce a nonlinear relationship. The output of the first fully connected layer is used as the input of the second fully connected layer, and the final emotion category output is obtained after passing through the second fully connected layer. Through this cascaded two fully connected layer structure, the emotion features in the input data can be better captured and extracted, and the corresponding emotion category can be output.

[0101] S44: Introducing a self-distillation mechanism into the deep network architecture of the model allows subsequent modules of the model to refer to the training results of the previous module and use these results as additional guidance signals to promote the model to learn more abstract and complex emotional features. Specifically, through three distillation loss functions, namely feature distillation loss, relation distillation loss and response loss, the teacher network is online distilled at the feature, relation and response levels, and the knowledge of the teacher network is transferred to the student network.

[0102] In a specific implementation, the deep network model that introduces the self-distillation mechanism includes a teacher network and a student network, and the two network models have the same structure and shared weight parameters.

[0103] In one specific embodiment, the self-distillation mechanism is implemented in the model by the following steps:

[0104] S441: The feature distillation loss is constructed by performing feature distillation between the last layer image global features of the feature extractors of the teacher network and the student network to enable the student network to mimic the feature activation pattern of the corresponding layer in the teacher network. This process is done by minimizing the feature activation difference between the student network and the teacher network, usually using the L2 norm or cosine similarity as the loss function, so as to achieve knowledge transfer at the feature level.

[0105] S442: The relation distillation loss is constructed in the following way: for the multi-scale graph structural features of the teacher network and the student network, the relationship between the internal features is calculated, and these relationships are quantitatively evaluated and compared using a specific metric method, such as cosine similarity or Euclidean distance. The relation distillation loss function aims to minimize the difference between the multi-scale graph relationship learned by the student network and the multi-scale graph relationship in the teacher network, so that the student network can learn richer structural knowledge.

[0106] S443: The response loss is constructed by directly comparing the outputs of the teacher network and the student network, that is, distilling the emotion categories output by the last fully connected layer. The response loss evaluates the difference in performance of the two networks on the final prediction task, and is usually implemented using a soft label cross entropy loss, so that the student network can learn the prediction behavior of the teacher network at the output level.

[0107] In the above implementation, the deep neural network model combined with the self-distillation mechanism of the present invention introduces feature distillation loss, relational distillation loss and response loss, so that the student network can not only learn the output behavior of the teacher network, but also deeply understand the complex relationships between the internal features of the teacher network. This mechanism significantly enhances the model's ability to understand and evaluate complex emotional attributes.

[0108] S5: Gradually learn samples from easy to difficult through a progressive learning strategy. In each progressive learning stage, the model uses a self-distillation mechanism to guide the learning of the current stage through the high-confidence predictions of the previous stage.

[0109] Furthermore, the progressive learning strategy may include:

[0110] S51: The training process is divided into multiple stages according to the difficulty level of the samples. Through progressive learning stages, the samples are gradually learned from easy to difficult. Specifically, the simpler viewpoint data samples are learned first, and then more complex samples are gradually introduced. In this way, the model can gradually grasp the emotional features and patterns at different levels.

[0111] In a specific implementation, in the initial stage, the model first learns simpler viewpoint data samples, which may contain obvious emotional features and patterns. By gradually introducing more complex samples, the model can gradually improve its ability to understand and learn more subtle and complex emotional features. This phased learning strategy helps the model better cope with challenges when faced with complex samples and improves the generalization and robustness of the model.

[0112] S52: In each progressive learning stage, the model uses the self-distillation mechanism to guide the learning of the current stage. Specifically, the model uses the high-confidence predictions of the previous stage as labels to guide the learning of the current stage. This ensures that the model maintains a certain stability and accuracy when learning new samples, so as to iteratively improve the learning ability of the model.

[0113] In a specific implementation, before each progressive learning stage begins, the current model is used to predict the newly introduced samples (or samples in the current stage), and the prediction results with high confidence are selected as soft labels. Such high-confidence predictions reflect the model's strong recognition ability for certain features or patterns, and can be used to guide subsequent learning.

[0114] Furthermore, the three distillation functions in S34 are used to quantify the difference between the current output of the model and the high confidence prediction (soft label). By minimizing this difference, the model can inherit and strengthen the learning results of the previous stage in the new training stage, while absorbing the knowledge of new samples.

[0115] Furthermore, according to the learning effect in the self-distillation process, the training parameters (such as learning rate, weight of loss function, etc.) are dynamically adjusted to ensure that the model can learn stably and effectively at each stage. On this basis, more optimization strategies can be introduced, such as dynamically adjusting the introduction speed and quantity of samples to adapt to the learning progress and capabilities of the model.

[0116] The above embodiments of the present invention divide the training process into multiple stages and combine the self-distillation mechanism to guide learning, so that the model can gradually learn and master samples of different difficulty levels, thereby improving the performance of the model in the emotion assessment task. This strategy can enable the model to more comprehensively understand and capture the emotional characteristics of viewpoint and image interaction, while improving the generalization ability and robustness of the model.

[0117] S6: Repeat the above steps S3 and S4 several times to gradually optimize the performance and generalization ability of the model.

[0118] In some embodiments, the more repetitions there are and the fewer difficult samples there are, the smaller the performance improvement is. When the performance no longer improves, the training is terminated.

[0119] S7: Use an independent validation set or test set to evaluate the performance and accuracy of the model, where the validation set or test set contains the eye viewpoint data and the images corresponding to the viewpoints when the human eye views different types of images on the screen, as well as the corresponding real emotion categories.

[0120] In some specific embodiments, after each training phase, a validation set with a different distribution from the training samples is used to test the performance of the model, and evaluation indicators include but are not limited to accuracy (Acc.) and F1 score;

[0121] Furthermore, according to the results of the performance evaluation, the hyperparameters of the model, such as the learning rate, regularization coefficient, etc., are adjusted to optimize the model structure in order to improve the evaluation performance of the model.

[0122] The viewpoint area division and sorting, feature extractor, multi-scale graph convolutional neural network, long short-term memory network, emotion evaluator, three distillation functions and progressive learning strategy in the above process together constitute a progressive self-distillation emotional state estimation method for the interaction between viewpoints and environment.

[0123] In a specific embodiment, according to the above specific implementation methods, a comparison chart of the evaluation results of a progressive self-distillation emotional state estimation method based on viewpoint-environment interaction of the present invention and some existing SOTA models can be obtained as shown in Appendix 1, wherein the fusion strategies of the compared existing model methods cannot effectively distinguish and process samples of different difficulties. It can be seen from the performance comparison table that the method can achieve good evaluation results under both simple and difficult samples of the collected data set, proving the effectiveness of the progressive self-distillation learning strategy.

[0124] Table 1 - Comparison of the obtained emotional state estimation results with the existing SOTA model emotion classification results

[0125]

[0126] It can be understood that the present invention proposes a progressive self-distillation emotional state estimation method based on the interaction between viewpoints and the environment, which aims to accurately determine the emotional state by collecting and analyzing the eye viewpoint data of people when watching different types of images on the screen, and combining the image position. First, the viewpoint data of the viewer is collected by a high-precision eye tracking device to form an initial data set. Using the interactive relationship between the viewpoint and the image, the image is segmented into multiple viewpoint retention areas, and the image is segmented and sorted according to the viewpoint retention order. Then, the image samples in the training set are evaluated for difficulty using a preliminary deep learning model, and the samples are sorted and graded according to the evaluation results, providing a basis for the progressive learning strategy. The present invention designs and implements a deep neural network model, which continuously optimizes its own prediction results during the training process in combination with the self-distillation mechanism. Through the progressive learning strategy, the model gradually learns samples from easy to difficult, and uses the self-distillation mechanism in each progressive learning stage to guide the learning of the current stage through the high confidence prediction of the previous stage. Repeat the above process to gradually optimize the performance and generalization ability of the model. Finally, this method uses an independent validation set or test set to evaluate the model, which contains eye viewpoint data and corresponding real emotion categories. This method effectively improves the accuracy and robustness of emotional state estimation, and its application scenarios include but are not limited to psychological research, intelligent human-computer interaction, emotional computing, advertising effect analysis, and user experience optimization. The advantages of this method are its high precision, high adaptability, and multi-scenario application capabilities. It can adapt to different types of image data and provide reliable emotional state analysis results.

[0127] Embodiment 2:

[0128] This embodiment provides a terminal device, which includes a processor and a memory, wherein the memory is used to store a computer program, the computer program includes program instructions, and the processor is used to execute the program instructions stored in the computer storage medium. The processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. It is the computing core and control core of the terminal, which is suitable for implementing one or more instructions, and is specifically suitable for loading and executing one or more instructions to implement the corresponding method flow or corresponding function; the processor described in the embodiment of the present invention can be used for the operation of a progressive self-distillation emotional state estimation method for viewpoint and environment interaction, including the following steps:

[0129] S1: Use high-precision eye tracking equipment to collect the viewer's eye viewpoint data when viewing different types of images on the screen to obtain training sets and test sets;

[0130] S2: Using the interactive relationship between viewpoints and images, the images in the training set are segmented into multiple viewpoint retention areas, and segmented and sorted according to the viewpoint retention order to obtain a viewpoint area sequence;

[0131] S3: Based on the obtained viewpoint region sequence, the difficulty of the viewpoint data samples in the training set is evaluated using the preset viewpoint emotion evaluation model, and the samples are sorted and graded according to the evaluation results;

[0132] S4: Based on the sorting and grading results, a deep neural network model is constructed. This model combines the self-distillation mechanism to continuously optimize its own prediction results during the training process, thereby improving the accuracy and reliability of emotional state estimation;

[0133] S5: Gradually learn samples from easy to difficult through a progressive learning strategy. In each progressive learning stage, the model uses a self-distillation mechanism to guide the learning of the current stage through the high-confidence prediction of the previous stage;

[0134] S6: Repeat steps S3 and S4 above to gradually build the performance and generalization ability of the deep neural model;

[0135] S7: Use a test set to evaluate the performance and accuracy of the model, where the test set includes eye viewpoint data when a person's eyes view different types of images on the screen, images corresponding to the viewpoints, and corresponding real emotion categories.

[0136] Embodiment 3:

[0137] This embodiment provides a storage medium, specifically a computer-readable storage medium (Memory), which is a memory device in a terminal device for storing programs and data. It is understandable that the computer-readable storage medium here can include both the built-in storage medium in the terminal device and the extended storage medium supported by the terminal device. The computer-readable storage medium provides a storage space, which stores the operating system of the terminal. In addition, one or more instructions suitable for being loaded and executed by the processor are also stored in the storage space, and these instructions can be one or more computer programs (including program codes). It should be noted that the computer-readable storage medium here can be a high-speed RAM memory or a non-volatile memory, such as at least one disk memory.

[0138] One or more instructions stored in a computer-readable storage medium may be loaded and executed by a processor to implement the corresponding steps of a progressive self-distillation emotional state estimation method for viewpoint-environment interaction in the above embodiment; one or more instructions in the computer-readable storage medium are loaded and executed by the processor as follows:

[0139] S1: Use high-precision eye tracking equipment to collect the viewer's eye viewpoint data when viewing different types of images on the screen to obtain training sets and test sets;

[0140] S2: Using the interactive relationship between viewpoints and images, the images in the training set are segmented into multiple viewpoint retention areas, and segmented and sorted according to the viewpoint retention order to obtain a viewpoint area sequence;

[0141] S3: Based on the obtained viewpoint region sequence, the difficulty of the viewpoint data samples in the training set is evaluated using the preset viewpoint emotion evaluation model, and the samples are sorted and graded according to the evaluation results;

[0142] S4: Based on the sorting and grading results, a deep neural network model is constructed. This model combines the self-distillation mechanism to continuously optimize its own prediction results during the training process, thereby improving the accuracy and reliability of emotional state estimation;

[0143] S5: Gradually learn samples from easy to difficult through a progressive learning strategy. In each progressive learning stage, the model uses a self-distillation mechanism to guide the learning of the current stage through the high-confidence prediction of the previous stage;

[0144] S6: Repeat steps S3 and S4 above to gradually build the performance and generalization ability of the deep neural model;

[0145] S7: Use a test set to evaluate the performance and accuracy of the model, where the test set includes eye viewpoint data when a person's eyes view different types of images on the screen, images corresponding to the viewpoints, and corresponding real emotion categories.

[0146] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0147] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0148] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0149] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0150] The preferred embodiments of the present invention are described in detail above, but the present invention is not limited to the above embodiments, and various changes can be made within the knowledge scope of ordinary technicians in this field without departing from the purpose of the present invention.

[0151] Many other changes and modifications may be made without departing from the concept and scope of the present invention.It should be understood that the present invention is not limited to the specific embodiments, and the scope of the present invention is defined by the appended claims.

Claims

1. A progressive self-distillation emotional state estimation method based on viewpoint-environment interaction, It is characterized in that The method comprises: S1: Use high-precision eye tracking equipment to collect the viewer's eye viewpoint data when viewing different types of images on the screen to obtain training sets and test sets; S2: Using the interactive relationship between viewpoints and images, the images in the training set are segmented into multiple viewpoint retention areas, and segmented and sorted according to the viewpoint retention order to obtain a viewpoint area sequence; S3: Based on the obtained viewpoint region sequence, the difficulty of the viewpoint data samples in the training set is evaluated using the preset viewpoint emotion evaluation model, and the samples are sorted and graded according to the evaluation results; S4: Based on the sorting and grading results, a deep neural network model is constructed. This model combines the self-distillation mechanism to continuously optimize its own prediction results during the training process, thereby improving the accuracy and reliability of emotional state estimation; S5: Gradually learn samples from easy to difficult through a progressive learning strategy. In each progressive learning stage, the model uses a self-distillation mechanism to guide the learning of the current stage through the high-confidence prediction of the previous stage; S6: Repeat steps S3 and S4 above to gradually build the performance and generalization ability of the deep neural model; S7: Use a test set to evaluate the performance and accuracy of the model, where the test set includes eye viewpoint data when a person's eyes view different types of images on the screen, images corresponding to the viewpoints, and corresponding real emotion categories.

2. According to claim 1, a progressive self-distillation emotional state estimation method for viewpoint-environment interaction, It is characterized in that The step S2 comprises: S201: According to the viewpoint data, each image is segmented according to the viewpoint stay position to form multiple viewpoint areas, and a viewpoint stay time threshold is defined. The viewpoint position whose stay time exceeds the threshold is considered as one viewpoint area; S202: Sort each viewpoint region in time order to form a viewpoint region sequence.

3. The progressive self-distillation emotional state estimation method based on viewpoint-environment interaction according to claim 2, It is characterized in that The viewpoint stay area segmentation includes segmenting the image according to the viewpoint data, and each image is segmented into a plurality of viewpoint stay areas. Assuming that the resolution of each image is W×H, the viewpoint stay area R is defined as follows: R={(x i ,y i )|t i <τ} Among them, τ is the threshold of the viewpoint residence time, and only the viewpoint position whose residence time exceeds the threshold is considered as a viewpoint area; The ordering of the viewpoint stop areas includes ordering each viewpoint area in time order to form a viewpoint area sequence; assuming that the viewpoint data is ordered in time order as {(t 1 ,x 1 ,y 1 ),(t 2 ,x 2 ,y 2 ),...,(t N ,x N ,y N )}, then the viewpoint region sequence S is: S={(x i1 ,and i1 ),(x i2 ,and i2 ),...,(x ik ,and ik )} Among them, i1, i2, ..., ik are viewpoint indices sorted in chronological order.

4. The method for estimating emotional states by progressive self-distillation based on viewpoint-environment interaction according to claim 1, It is characterized in that The step S3 comprises: S301: using an existing pre-trained image emotion assessment model to perform preliminary emotion classification on the viewpoint data in the training set, wherein the preliminary deep learning model is an existing pre-trained viewpoint emotion assessment model MSBLS; S302: Based on the preliminary classification results, evaluate the difficulty of each sample, sort and grade the samples for subsequent progressive learning stages.

5. The progressive self-distillation emotional state estimation method based on viewpoint-environment interaction according to claim 1, It is characterized in that The step S4 comprises: S401: Design a feature extractor, which is a deep neural network architecture, including multiple convolutional layers, activation layers, and pooling layers, so as to effectively extract global features of the viewpoint area. The feature extractor is constructed based on the convolutional neural network ResNet50 or VGG16 with the fully connected layer removed; S402: Design a multi-scale graph convolutional neural network to obtain multi-scale relational features of the global features of the viewpoint region at the deepest level; design a long short-term memory network to perform temporal learning on the extracted features to capture the dynamic relationship between viewpoint regions; S403: Design a sentiment evaluator, including two cascaded fully connected layers, to output sentiment categories; S404: Introducing a self-distillation mechanism into the deep network architecture of the model, allowing subsequent modules of the model to refer to the training results of the previous modules and use these results as additional guidance signals to promote the model to learn more abstract and complex emotional features; through three distillation loss functions, namely feature distillation loss, relationship distillation loss and response loss, the teacher network is online distilled at the feature, relationship and response levels respectively, and the teacher network knowledge is transferred to the student network.

6. The progressive self-distillation emotional state estimation method based on viewpoint-environment interaction according to claim 5, It is characterized in that The self-distillation mechanism is implemented in the model through the following steps: S4041: The feature distillation loss is constructed by: performing feature distillation between the last layer image global features of the feature extractors of the teacher network and the student network to enable the student network to imitate the feature activation pattern of the corresponding layer in the teacher network; this process is completed by minimizing the feature activation difference between the student network and the teacher network, using L2 norm or cosine similarity as the loss function, thereby realizing knowledge transfer at the feature level; S4042: The relation distillation loss is constructed in the following manner: for the multi-scale graph structural features of the teacher network and the student network, the relationship between the internal features is calculated, and a specific metric method, such as cosine similarity or Euclidean distance, is used to quantitatively evaluate and compare these relationships; the relation distillation loss function aims to minimize the difference between the multi-scale graph relationship learned by the student network and the multi-scale graph relationship in the teacher network, so that the student network can learn richer structural knowledge; S4043: The response loss is constructed by directly comparing the outputs of the teacher network and the student network, distilling the emotion categories output by the last fully connected layer, and evaluating the performance difference between the two networks on the final prediction task by the response loss, which is implemented using soft label cross entropy loss, so that the student network can learn the prediction behavior of the teacher network at the output level.

7. The method for estimating emotional states by progressive self-distillation based on viewpoint-environment interaction according to claim 1, It is characterized in that The step S5 comprises: S501: According to the difficulty level of the samples, the training process is divided into multiple stages, and the samples are gradually learned from easy to difficult through progressive learning stages; specifically, the simpler viewpoint data samples are learned first, and then more complex samples are gradually introduced. In this way, the model can gradually grasp the emotional features and patterns of different levels; S502: In each progressive learning stage, the model uses the self-distillation mechanism to guide the learning of the current stage; specifically, the model uses the high-confidence predictions of the previous stage as labels to guide the learning of the current stage, ensuring that the model maintains a certain stability and accuracy when learning new samples, so as to iteratively improve the learning ability of the model.

8. The progressive self-distillation emotional state estimation method based on viewpoint-environment interaction according to claim 1, It is characterized in that The step S7 comprises: S701: After each training phase, a validation set with a different distribution from the training samples is used to test the performance of the model. Evaluation indicators include accuracy and F1 score. S702: According to the results of the performance evaluation, adjust the hyperparameters of the model, including the learning rate and the regularization coefficient, and optimize the model structure to improve the emotion evaluation performance of the model.

9. A computer device, It is characterized in that The invention comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, a progressive self-distillation emotional state estimation method for viewpoint-environment interaction according to any one of claims 1 to 8 is implemented.

10. A computer-readable storage medium, It is characterized in that The computer-readable storage medium stores a computer program, which, when executed by a processor, implements a progressive self-distillation emotional state estimation method for viewpoint-environment interaction as described in any one of claims 1 to 8.

Citation Information

Patent Citations

  • Shielded facial emotion recognition method based on information difference knowledge distillation

    CN116168443A

  • Single-eye emotion recognition method based on spiking neural network and multi-mode collaborative knowledge distillation

    CN118486067A

  • Method of and system for evaluating consumption of visual information displayed to a user by analyzing user's eye tracking and bioresponse data

    US20200364539A1