Method and device for emotion evaluation, medium and program product

Abstract

The invention relates to a method and device for emotion evaluation, a medium and a program product. The method comprises the steps of obtaining multi-modal emotion evaluation data of a to-be-evaluated object in a process of performing a plurality of predetermined emotion evaluation tasks on the to-be-evaluated object; performing feature extraction processing on each item of data included in the emotion evaluation data to obtain corresponding feature information; fusing the feature information corresponding to each data to obtain corresponding fused feature information; and outputting a corresponding emotion evaluation result based on the fused feature information by using a trained emotion prediction model. According to the method, the trained emotion prediction model is used, the corresponding emotion evaluation result is output based on the feature information of the multi-modal emotion evaluation data, model pre-training and automatic iteration updating are carried out by adopting the meta-learning algorithm, individual differences are rapidly adapted, and the performance, efficiency and robustness of model prediction are improved.

Description

Technical Field

[0001] The present application relates to the field of computer technology, mental health or medical health management, and in particular to a method, device, computer-readable medium and computer program product for emotion assessment. Background Art

[0002] Emotional health issues are currently receiving increasing attention and recognition. Specific groups, such as adolescent students, employees of businesses and institutions, and those working in special environments, are particularly prone to emotional issues due to their unique social roles and age groups. Given the large size of these populations, the development of effective assessment tools and management solutions is particularly urgent. However, existing emotion assessment methods and tools still have significant deficiencies in accuracy, objectivity, timeliness, dynamism, and convenience, making them difficult to meet practical needs.

[0003] Traditional emotion assessments primarily rely on subjective questionnaires or scales, resulting in low accuracy and objectivity. Furthermore, these assessments often involve long-term, cross-sectional evaluations, making them less dynamic and timely. Furthermore, existing emotion assessment methods based on cognitive-behavioral responses often directly correlate pure cognitive responses with emotional states, failing to fully consider the complex interactions between cognitive-behavioral responses and individual emotional states in emotion-inducing situations. This results in insufficient sensitivity and specificity in the assessments. Summary of the Invention

[0004] Multiple aspects of the present application provide a method, apparatus, computer-readable medium, and computer program product for performing emotion assessment.

[0005] In one aspect of the present application, a method for performing emotion assessment is provided, wherein the method comprises:

[0006] Acquiring multimodal emotion assessment data of the subject to be assessed during the subject to be assessed performing a plurality of predetermined emotion assessment tasks;

[0007] Performing feature extraction processing on each data item included in the emotion assessment data to obtain corresponding feature information;

[0008] Fusing the feature information corresponding to each data item to obtain corresponding fused feature information;

[0009] A trained emotion prediction model is used to output corresponding emotion evaluation results based on the fused feature information, wherein the emotion prediction model is pre-trained and automatically iteratively updated using a meta-learning algorithm.

[0010] In one aspect of the present application, a device for performing emotion assessment is provided, wherein the device comprises:

[0011] A device for obtaining multimodal emotion assessment data of a subject to be assessed while the subject to be assessed performs a plurality of predetermined emotion assessment tasks;

[0012] A device for performing feature extraction processing on each data item included in the emotion assessment data to obtain corresponding feature information;

[0013] A device for fusing feature information corresponding to each data item to obtain corresponding fused feature information;

[0014] A device for using a trained emotion prediction model to output corresponding emotion evaluation results based on the fused feature information, wherein the emotion prediction model adopts a meta-learning algorithm for pre-training and automatic iterative update.

[0015] Another aspect of the present application provides an electronic device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the method of an embodiment of the present application.

[0016] In another aspect of the present application, a computer-readable storage medium is provided, on which computer program instructions are stored. The computer program instructions can be executed by a processor to implement the method of the embodiment of the present application.

[0017] In another aspect of the present application, a computer program product is provided, including a computer program, which implements the method of the embodiment of the present application when executed by a processor.

[0018] In the solution provided in the embodiment of the present application, by acquiring emotion assessment data of multiple modalities and using a trained emotion prediction model to output corresponding emotion assessment results based on the feature information of the emotion assessment data of multiple modalities, this multimodal data fusion method reduces the limitations of a single data source, thereby significantly improving the accuracy and reliability of the emotion assessment results; in the data collection process, a variety of methods are used to automatically verify the validity of the data to ensure data quality; the model of the embodiment of the present application adopts a meta-learning algorithm for pre-training and automatic iterative update, and only requires a small amount of data and a few steps of gradient update to quickly adapt to individual differences, realize the personalization and dynamism of the assessment, and further improve the performance, efficiency and robustness of the model prediction; a progressive data fusion strategy is adopted, and old data and new data are combined for comprehensive prediction calculation, which can dynamically evaluate the emotional state of the object, realize continuous tracking and accurate capture of the emotional change process, and effectively improve the continuity of the emotion assessment, the continuity of the data, and the stability and accuracy of the results. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the technical solutions in the embodiments of the present application, a brief introduction will be given below to the drawings required for use in the embodiments or descriptions of the prior art. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0020] Other features, objects and advantages of the present application will become more apparent upon reading the detailed description of non-limiting embodiments made with reference to the following drawings:

[0021] Figure 1 A flow chart of a method for emotion assessment provided in an embodiment of the present application is shown;

[0022] Figure 2 A schematic diagram of an exemplary intelligent emotion assessment system according to an embodiment of the present application is shown;

[0023] Figure 3 A schematic diagram of the structure of a device for emotion assessment provided in an embodiment of the present application is shown;

[0024] Figure 4 A structural diagram of a device suitable for implementing the solution in the embodiments of the present application is shown.

[0025] The same or similar reference numerals in the drawings represent the same or similar components. DETAILED DESCRIPTION

[0026] To make the purpose, technical solutions, and advantages of the embodiments of this application more clear, the technical solutions in the embodiments of this application will be clearly and completely described below in conjunction with the drawings in the embodiments of this application. Obviously, the described embodiments are part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0027] In a typical configuration of the present application, the terminal and the equipment of the service network each include one or more processors (CPUs), input / output interfaces, network interfaces and memories.

[0028] Memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. Memory is an example of a computer-readable medium.

[0029] Computer-readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology for information storage. The information can be computer program instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc-read only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic tape disk storage or other magnetic storage devices or any other non-transmission medium that can be used to store information that can be accessed by a computing device.

[0030] Figure 1 A flow chart of a method for emotion assessment provided by an embodiment of the present application is shown, wherein the method comprises at least step S101, step S102, step S103 and step S104.

[0031] In actual scenarios, the execution subject of this method can be an intelligent hardware device, or it can be an application running on an intelligent hardware device. The intelligent hardware device includes a user device or a network device. The user device includes but is not limited to various terminal devices such as computers, mobile phones, tablets, smart watches, and bracelets. The network device includes but is not limited to network hosts, single network servers, multiple network server sets, or cloud computing-based computer sets, which can be used to implement some processing functions when setting an alarm. Here, the cloud is composed of a large number of hosts or network servers based on cloud computing (Cloud Computing), where cloud computing is a type of distributed computing, a virtual computer composed of a group of loosely coupled computer sets.

[0032] Different from traditional evaluation methods, the method according to the embodiment of the present application obtains emotion evaluation data of multiple modalities, and uses a trained emotion prediction model to output corresponding emotion evaluation results based on the features corresponding to the emotion evaluation data. When performing emotion evaluation, the influence of multiple factors on the emotion evaluation results is comprehensively considered from multiple aspects such as the voice, eye movements, facial features and cognitive behavior of the object to be evaluated, so that the emotion evaluation results obtained by the method of the embodiment of the present application are more accurate and reliable.

[0033] Reference Figure 1 In step S101, during the process of the subject to be evaluated performing a plurality of predetermined emotion evaluation tasks, multimodal emotion evaluation data of the subject to be evaluated is obtained.

[0034] The subject to be evaluated refers to a subject that needs to undergo emotion evaluation.

[0035] The emotion assessment data includes various data that can reflect the emotions of the subject to be assessed. The emotion assessment data can come from a variety of sensor devices and data sources.

[0036] According to one embodiment, the emotion assessment data includes but is not limited to at least any one of the following:

[0037] 1) Voice data;

[0038] 2) Facial video data;

[0039] 3) Facial blood flow data: Facial blood flow data includes information about blood flow in facial vessels. Emotional fluctuations cause changes in the contraction and dilation of facial blood vessels, which in turn affect facial blood flow. Facial blood flow can be used as a reference indicator for emotional assessment.

[0040] When collecting facial blood flow data, a baseline facial blood flow profile is established for subjects undergoing the emotion assessment for the first time. During the presentation of images or video stimuli or the performance of cognitive tasks, facial blood flow signals are captured using rPPG (radiometric photoplethysmography) using visible light imaging technology and a device camera (e.g., a mobile phone's front-facing camera).

[0041] 4) Facial temperature data;

[0042] Facial temperature data indicates the temperature distribution on the facial skin surface. Because different emotions can cause temperature increases in different facial regions, facial temperature can be used as a reference indicator for emotional assessment.

[0043] When collecting facial temperature data, for subjects undergoing emotion assessment for the first time, baseline temperature mapping is performed to construct a baseline facial temperature distribution map. When presenting images or video stimuli or performing cognitive tasks, facial temperature distribution data is collected using a device camera (such as a mobile phone camera equipped with an infrared module) based on infrared thermal imaging technology.

[0044] 5) Eye movement data; the eye movement data includes various eye movement features that can reflect emotions, including but not limited to gaze duration, scanning path, etc.

[0045] 6) Cognitive-behavioral data: This data records the participant's behavioral performance while completing the emotion assessment task, including task completion, reaction time, and accuracy. This data is collected and presented through a software system that controls the task presentation. It reflects the participant's cognitive processing and behavioral response patterns during the emotion assessment task.

[0046] The data collection of voice data, facial video data, facial blood flow data, facial temperature data, eye movement data, and cognitive behavior data can be completed on a single hardware device, such as using only a computer to collect all the data. Alternatively, the above data can be collected simultaneously using multiple devices.

[0047] Optionally, the acquired voice data, facial video data, facial blood flow data, facial temperature data, eye movement data, and / or cognitive behavior data are algorithmically corrected based on factors such as ambient temperature, ambient noise, lighting, and motion to eliminate interference from factors such as ambient temperature, ambient noise, lighting, motion, and makeup. Furthermore, device sensor data (such as a mobile phone gyroscope) is used to assist in tracking voice, facial images, blood flow, temperature, eye movement, and / or behavioral characteristics to address data distortion caused by device movement or sudden changes in lighting.

[0048] According to one embodiment, step S101 includes step S1011 and step S1012.

[0049] In step S1011 , a plurality of predetermined emotion assessment tasks are presented, wherein the emotion assessment tasks include general emotion assessment tasks and cognitive emotion assessment tasks.

[0050] The general emotion assessment task assesses the emotional state of an individual by collecting facial images and videos, voice, and cognitive behavioral responses of the subject to be assessed in a natural state.

[0051] The cognitive emotion assessment task includes a series of tasks that induce an individual's emotional response by combining stimulus materials such as emotional words, emotional pictures, and emotional videos with cognitive tasks, while also evaluating the impact of emotional information on the individual's cognitive processing.

[0052] Optionally, the cognitive emotion assessment task includes but is not limited to using emotion words, emotion pictures and emotion videos as stimulus materials for collecting data, and is formed by combining with a variety of cognitive tasks (such as picture browsing tasks, single-tone pronunciation tasks, voice question-answering tasks, word reading tasks, short text reading tasks, dot detection tasks, visual search tasks, letter search tasks, unexpected recognition tasks, Stroop tasks, Oddball tasks, spatial working memory tasks, letter working memory tasks, proactive cue switching tasks, reactive cue switching tasks, reward delay tasks, risk decision-making tasks and more than ten other cognitive behavioral tasks) to form a task with an emotion priming effect. Cognitive emotion assessment tasks include but are not limited to emotional picture browsing task, emotional single-tone voice collection task, emotional question-and-answer voice collection task, emotional word reading voice collection task, emotional short article reading voice collection task, point detection emotion task, visual search emotion task, emotional letter search task, emotional unexpected recognition task, emotional Stroop task, emotional Oddball task, spatial emotion working memory task, letter emotion working memory task, proactive emotion cue conversion task, reactive emotion cue conversion task, emotional reward delay task, emotional risk decision-making task and other emotional cognitive behavioral tasks.

[0053] In step S1012, while the subject to be evaluated is performing the emotion evaluation task, voice data, facial video data, facial blood flow data, facial temperature data, eye movement data and / or cognitive behavior data of the subject to be evaluated are synchronously collected as emotion evaluation data.

[0054] Among them, the voice data, facial video data, facial blood flow data, facial temperature data, eye movement data and cognitive behavior data can be collected through the hardware equipment used by the subject to be evaluated to complete the emotion evaluation task, or through other equipment independent of the hardware equipment.

[0055] It should be noted that the emotion assessment data involved in the embodiments of this application are obtained with the consent or permission of the relevant subjects, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions.

[0056] According to one embodiment, before the subject to be evaluated performs the emotion evaluation task, the method performs eye movement calibration processing based on the image or video data of the subject to be evaluated.

[0057] Specifically, an eye movement calibration task is configured, and the gaze points of the eye movement calibration task are generated by a software system included in the equipment for performing the emotion assessment task; then, during the process of the eye movement calibration of the subject to be assessed, the eye movement trajectory of the subject to be assessed is automatically recorded; then, the recorded eye movement trajectory is matched with the logical points, and the eye movement calibration is achieved through a preset eye movement calibration algorithm; then, the eye movement trajectory state of the subject to be assessed is calculated and identified based on a predetermined eye movement trajectory recognition algorithm, so as to collect, calculate and record the eye movement data of the subject to be assessed.

[0058] The eye movement calibration algorithm includes but is not limited to at least one of the following:

[0059] Geometric calibration algorithm: calibrates eye movement data based on geometric transformations, such as perspective transformation and affine transformation, by mapping the eye point data to the calibration base point to achieve eye movement calibration;

[0060] Statistical calibration algorithms: Statistical methods are used to analyze eye movement data, such as Kalman filtering and particle filtering. These methods can deal with noise and uncertainty in eye movement data and improve calibration accuracy.

[0061] Machine learning calibration algorithms: These algorithms use machine learning models to learn patterns in eye movement data, such as support vector machines (SVMs) and neural networks. These algorithms can automatically identify and correct biases in eye movement data.

[0062] Feature-based calibration algorithms: Utilize specific features in eye movement data for calibration, such as the pupil center-corneal reflection vector method and iris feature point matching. These methods achieve accurate calibration by extracting and matching feature points.

[0063] The eye movement trajectory recognition algorithm includes but is not limited to an optical-based eye movement trajectory recognition algorithm and / or a machine learning-based eye movement trajectory recognition algorithm.

[0064] The optical eye movement recognition algorithm includes but is not limited to at least one of the following:

[0065] 1) Pupil-corneal reflection method: Eye tracking is performed by connecting the center of the cornea and the center of the pupil. A camera captures an image of the eye, and the pupil contour is extracted from the image to calculate the pupil center. This infers the eye's displacement in each frame, and the gaze point is mapped onto the display using an eye model.

[0066] 2) Iris-Sclera Limb Method: Invisible infrared light is illuminated at the eye. Two infrared phototubes are installed near the eye. The infrared light reflected from the left and right sides of the iris-sclera limbus is received by each phototube. When the eye moves to the right, the iris turns to the right, reducing the infrared light received by the right phototube. Meanwhile, the left side of the sclera reflects more infrared light, increasing the infrared light received by the left phototube. This differential signal allows contactless measurement of eye movement.

[0067] 3) Circle detection based on Hough transform: First, the approximate position of the pupil is detected using the integral projection method. Then, the edge detection operator is used to extract the edge information of the image, and the edge image is binarized. Finally, the circle-based Hough transform is used to quickly locate the human eye.

[0068] 4) Video-based eye movement recognition: Use a video camera (infrared camera or ordinary camera) to record the eye movement process, and automatically analyze and process the video images through a computer. The camera can capture human eye images or infrared images reflected by the eyeballs, and then automatically process the image data through a computer to achieve non-invasive human pupil center detection or gaze point estimation.

[0069] 5) Gaze estimation method based on two-dimensional mapping: Based on corneal reflection, one or more cameras and one or more external near-infrared light sources are used to assist in determining eye features that do not change with line of sight and vary from person to person, such as the eyeball radius and corneal curvature radius. Through these features, the gaze direction and then the eye movement trajectory can be calculated.

[0070] The eye movement trajectory recognition algorithm based on machine learning includes but is not limited to at least any one of the following:

[0071] 1) HOG+SVM+LSTM method (a combination of the Histogram of Oriented Gradients (HOG) + Support Vector Machine (SVM) + Long Short-Term Memory (LSTM) algorithms): HOG features are first used to extract features of the eye region. An SVM model is then used to determine whether the eyes are open, closed, or blinking. The positional changes of the eye centers between adjacent frames are then analyzed to generate a differential sequence of eye movement trajectories, which is used as input to the LSTM network to predict eye movement behavior.

[0072] 2) Convolutional Neural Network (CNN)-based method: Use convolutional neural networks to extract and classify eye images, learn the characteristics of eye movement trajectories directly from images, and thus realize the recognition of eye movement behavior.

[0073] 3) LSTM-based algorithm: The eye movement trajectory data is used as the input of the LSTM network, and the LSTM is used to model the temporal information of the eye movement trajectory to predict eye movement behavior.

[0074] 4) Algorithm based on Hidden Markov Model (HMM): Eye movement trajectory data is used as an observation sequence, and eye movement behavior is modeled using HMM. The relationship between the observation sequence and the hidden state is learned to achieve eye movement behavior recognition.

[0075] 5) Algorithm based on path integral and deep learning: Combine the path integral features with the deep learning model, use the path integral to extract the global features of the eye movement trajectory, and use the deep learning model to learn and represent the local features, thereby realizing the detection and analysis of eye movement behavior.

[0076] According to one embodiment, the method further includes step S105 and step S106.

[0077] In step S105 , the collected emotion assessment data is automatically verified for validity.

[0078] The method verifies the effectiveness of the collected emotion assessment data online and automatically through multiple methods.

[0079] Specifically, voice data validity verification involves comparing the collected voice samples with the system's pre-stored target voice signals using advanced voice feature analysis technology combined with deep semantic parsing algorithms. This comparison may include key features such as voice timbre, pitch, speaking rate, and semantic coherence. If the consistency of the various features in the comparison results reaches a preset threshold, the voice data is considered valid; conversely, if the consistency falls below the threshold, the data is considered invalid.

[0080] Verifying the validity of facial video data involves checking the facial integrity and camera angle of the user being evaluated throughout the facial video data collection process. If the facial integrity and angle do not meet preset values, the data is considered invalid. Specifically, the facial integrity of the user being evaluated is monitored based on a specific facial integrity detection algorithm. This facial integrity detection algorithm, based on a deep learning model, can accurately identify key facial feature points to ensure the integrity and clarity of the facial image. Specifically, when testing the camera angle, the three-dimensional spatial information of the face in the video image is analyzed to determine whether the camera angle meets the preset acquisition standards. If both facial integrity and camera angle meet the system's preset threshold requirements, the facial video data is considered valid. If either condition is not met, the facial video data is considered invalid, meaning it has failed data validity verification. This approach can effectively avoid difficulties in facial feature extraction caused by issues such as occlusion and angle deviation, thereby improving data validity and analysis accuracy.

[0081] Verifying the validity of eye movement data involves tracking the subject's eye movement trajectory during the eye movement calibration process. If the deviation between the subject's eye movement and the target position exceeds a preset threshold, the data is considered invalid. Specifically, high-precision eye tracking equipment is used to accurately record the subject's eye movement trajectory. The actual recorded eye movement trajectory is compared and analyzed in real time with the system's preset target position to calculate the degree of deviation between the two. If the deviation exceeds the system's preset threshold, the eye movement data is deemed invalid; otherwise, it is considered valid.

[0082] The validity verification of cognitive behavioral data includes: when performing cognitive behavioral tasks, if the reaction time of the task requiring the subject to be evaluated to respond exceeds the preset value or the error rate is higher than the preset value, it is invalid data. Specifically, during the process of the subject to be evaluated performing cognitive behavioral tasks, the system monitors the subject's response to the task in real time. Through a comprehensive analysis of the reaction time and error rate of the subject to be evaluated, the user's concentration and cognitive state during the task execution are judged. If the user's average reaction time exceeds the reasonable range set by the system based on a large amount of experimental data, or the error rate is higher than the preset threshold, the cognitive behavioral data is judged to be invalid data; otherwise, it is judged to be valid data.

[0083] Verification of the validity of facial blood flow and temperature data involves real-time signal-to-noise ratio analysis during data collection. If the signal-to-noise ratio exceeds a preset threshold, the data is deemed invalid. Specifically, a signal quality assessment algorithm is used to analyze the signal-to-noise ratio of facial blood flow and temperature data in real time. If the signal-to-noise ratio exceeds a preset threshold, the data is considered valid, indicating a high proportion of valid information and minimal noise interference. Conversely, if the signal-to-noise ratio is below the preset threshold, the data is deemed invalid.

[0084] In step S106 , if the validity verification fails, the corresponding emotion assessment data is collected again.

[0085] By conducting validity verification, we ensure that the collected data is of high quality and high reliability during the emotion assessment process, thereby ensuring the accuracy of subsequent emotion assessments.

[0086] According to one embodiment, the acquired emotion assessment data is preprocessed.

[0087] The preprocessing includes, but is not limited to, data cleaning, data standardization and normalization, data rectification, data quality assessment, and data preservation. Data cleaning includes, but is not limited to, data review, downsampling, noise reduction (artifact processing, filtering, outlier processing, and useless data processing), missing value processing, data alignment, and time synchronization (based on the PTP protocol to ensure the alignment of multimodal data timestamps).

[0088] Continue to refer to Figure 1 To illustrate, in step S102, feature extraction processing is performed on each item of data included in the emotion assessment data to obtain corresponding feature information.

[0089] Specifically, in step S102 , the type of emotion assessment data to be processed is determined, and a feature extraction operation matching the type is adopted.

[0090] Specifically, based on different data feature types, one or more of statistics, frequency domain features, time domain features, nonlinear dynamic features, or waveform features are calculated and extracted from the emotion assessment data. Statistics include, but are not limited to, mean and variance, and frequency domain features include fast Fourier transform (FFT) energy. Time domain features include, but are not limited to, peak values ​​and peak intervals within a sliding window. Nonlinear dynamic features include, but are not limited to, multiscale entropy and fractal dimension. Waveform features include, but are not limited to, rise time, number of peaks, and area integral.

[0091] The following describes the feature extraction operations for different types of emotion assessment data.

[0092] 1) Voice data;

[0093] The features extracted from the speech data include but are not limited to Mel-frequency cepstral coefficients (MFCC), fundamental frequency, energy envelope (RMS), speaking rate, intonation, pause frequency, emotional prosody pattern, semantic emotion and emotion type, and speech text content. MFCC features are calculated by extracting multidimensional coefficients through a Mel filter bank, fundamental frequency is calculated through an autocorrelation function (F0=argmax(R(τ))), and emotional prosody pattern is extracted using the eGeMAPS feature set extracted using the opensmile toolkit.

[0094] 2) Facial video data;

[0095] Features extracted based on the facial image data include but are not limited to facial key points, action units (AUs), local dynamic textures, optical flow features, and facial micro-expressions.

[0096] Among them, facial key points are extracted by using Dlib or MediaPipe; the action unit is obtained by extracting AU intensity based on OpenFace2.0; the local dynamic texture is extracted using spatiotemporal local binary pattern (LBP-TOP); the optical flow feature is obtained by calculating the facial motion vector using the Farneback algorithm; the facial micro-expression is obtained by identifying short-term micro-expressions using the capsule network (CapsuleNet).

[0097] 3) facial blood flow data;

[0098] Features extracted based on facial blood flow data include but are not limited to at least any one of the following: key feature point positioning: such as vascular area positioning in the cheeks, forehead, etc., blood flow signal characteristics: pure blood flow signal characteristics after denoising, blood flow pulsation characteristics: time-frequency graph characteristics of blood flow pulsation extracted by wavelet transform, blood volume pulse characteristics: amplitude and phase characteristics of blood volume pulses, asymmetry and asynchrony indicators: such as pulse wave transmission time (PTT), vascular tension index, blood flow oscillation spectrum, etc.

[0099] When extracting features based on facial blood flow data, deep learning models, such as convolutional neural networks (CNNs), are first used to precisely locate facial vascular regions and segment key feature points, such as the cheeks and forehead, to effectively reduce the interference of motion artifacts. Next, independent component analysis (ICA) or spectral kurtosis methods are used to separate blood flow signals from noise, removing the effects of noise caused by blinking or facial muscle movement. Subsequently, wavelet transforms, such as continuous wavelet transforms (CWTs), are used to extract time-frequency maps of blood flow pulsations, deeply analyzing the amplitude and phase characteristics of blood volume pulses, and calculating asymmetry and asynchrony indicators such as pulse wave transit time (PTT), vascular tension index, and blood flow oscillation spectrum.

[0100] 4) Facial temperature data;

[0101] Features extracted based on facial temperature data include but are not limited to at least any one of the following: distribution features: temperature distribution in different areas of the face, temperature change complexity: the complexity of temperature changes over time, specific temperature duration: the length of time a specific temperature persists in a facial area, distribution position features: the position features of a specific temperature distribution, isotherm migration trajectory: the facial isotherm migration trajectory constructed based on temperature microstate features.

[0102] When extracting features based on facial temperature data, the thermal image is first converted to grayscale data for subsequent processing. Next, micro-steady-state processing is performed on the facial temperature distribution data based on time series data to extract facial temperature microstate features, including distribution characteristics, temperature variation complexity, duration of specific temperatures, and distribution location. Furthermore, the migration trajectory of facial isotherms is constructed based on the facial temperature distribution and variation.

[0103] 5) Eye movement data;

[0104] Features extracted from the eye movement data include, but are not limited to, gaze point coordinates, duration, saccade velocity, pupil diameter, area of ​​interest (AOI) dwell time percentage, entropy index, and microsaccades. The area of ​​interest dwell time percentage is obtained by dividing the AOI using a Voronoi diagram; the entropy index is calculated by Shannon entropy based on the saccade path disorder; and microsaccades are obtained by calculating the frequency of microsaccades.

[0105] 6) cognitive behavioral data;

[0106] Features extracted from cognitive behavioral data include but are not limited to task accuracy, reaction time, click trajectory complexity, and decision hesitation. Click trajectory complexity is calculated by calculating path tortuosity using Hausdorff distance.

[0107] Optionally, for cognitive behavioral data, signal preprocessing is first performed to remove outliers, and then the average reaction time, reaction time coefficient of variation, and accuracy are calculated as features corresponding to the cognitive behavioral data.

[0108] To enhance data privacy protection, the embodiments of the present application utilize biometric desensitization technology to convert raw blood flow signals into abstract vascular motion feature vectors. This conversion method can retain emotion-related features in the data while removing raw biometric details that may contain personal identification information, thereby effectively protecting user privacy. Specifically, for facial temperature data, the embodiments of the present application only collect and record relative temperature changes, rather than absolute temperature values. This processing method can reduce the personal identification information that may be contained in the data while still retaining features valuable for sentiment analysis, such as temperature change trends and patterns.

[0109] Moreover, in terms of data processing architecture, the embodiment of the present application is designed based on the edge computing framework. Based on this design, the original data will be stored on the user's local device and will not be transmitted to an external server or other device. All feature extraction and preliminary analysis work in the embodiment of the present application will be completed on the local device side, thereby minimizing the risk of data leakage. In certain scenarios that require multi-device collaboration, each terminal device will only upload encrypted parameters, which are difficult to decrypt and interpret during transmission, further ensuring the security and privacy of user data.

[0110] Continue to refer to the following Figure 1 To illustrate, in step S103, the feature information corresponding to each data item is fused to obtain corresponding fused feature information.

[0111] The fusion processing method includes but is not limited to at least one of the following:

[0112] 1) Early fusion; the early fusion includes but is not limited to feature concatenation, feature conversion, and shared feature extractors.

[0113] The feature concatenation method directly concatenates features of the same modality into a new feature vector. The feature conversion method converts features of different modalities into the same space through mathematical transformations (such as Fourier transform, wavelet transform, etc.) and then fuses them. The shared feature extractor uses a shared feature extractor (such as a shared convolutional neural network layer) to extract features from data of different modalities, so that the extracted features are consistent to a certain extent, facilitating subsequent fusion.

[0114] 2) Intermediate Fusion: Intermediate fusion includes but is not limited to attention fusion and circulant matrix fusion. Attention fusion uses the attention mechanism to weight features from different modalities, highlighting important features. For example, a gated weighted attention layer can be designed to fuse visual and textual information. Circulant matrix fusion reshapes feature vectors into circulant matrices and defines interactive operations between vectors and matrices to fully explore the interactions between features from different modalities.

[0115] 3) Late Fusion: Late fusion includes but is not limited to voting and model fusion. Voting performs weighted voting based on the prediction results of each modality. Model fusion inputs features from different modalities into different models and then fuses the outputs of these models.

[0116] 4) Learning-based fusion; such learning-based fusion includes, but is not limited to, multi-layer perceptron (MLP) fusion and deep learning model fusion. Multi-layer perceptron fusion inputs features from different modalities into a multi-layer perceptron, automatically learning how to fuse features through its nonlinear transformations. Deep learning model fusion uses deep learning models (such as convolutional neural networks and Transformers) to fuse multimodal features. For example, a combination of Transformer and DenseNet can be used to extract and fuse image features.

[0117] 5) Special fusion methods; including but not limited to consistent feature learning, cross-modal alignment perception, and feature diffusion modeling. Consistent feature learning compensates for unimodal features using a learnable modality dictionary, maintaining consistency between multimodal features. Cross-modal alignment perception further optimizes the fusion process by constructing a correlation matrix to describe the relationship between source image pixels. Feature diffusion modeling uses a feature diffusion model to enhance and classify features, eliminating unnecessary noise and patterns.

[0118] In step S104, the trained emotion prediction model is used to output corresponding emotion evaluation results based on the fused feature information.

[0119] The emotion prediction model adopts a meta-learning algorithm for pre-training and automatic iterative update to adapt to different objects to be evaluated and improve the performance, efficiency and robustness of the model prediction.

[0120] The emotion prediction model is a machine learning-based intelligent classification and evaluation model for intelligently classifying and evaluating emotional states. Optionally, the emotion prediction model is used to identify the type of emotion and assess the intensity of the emotion, such as mild, moderate, severe, or expressed as a percentage.

[0121] The emotion prediction model can be constructed using a variety of machine learning techniques. Optionally, the emotion prediction model is constructed using a deep learning model and a multi-task learning model.

[0122] Among them, the deep learning model includes but is not limited to at least any one of the following: long short-term memory network (LSTM): used to process sequence data and capture emotional changes in time series; convolutional neural network (CNN): used to extract local features in images or text, suitable for emotion recognition; graph convolutional network (GCN): used to process graph structure data and analyze the propagation of emotions in the network; CNN-LSTM hybrid model: combining the feature extraction capability of CNN and the sequence processing capability of LSTM to improve the robustness of the model; multimodal Transformer model: used to process multi-source data (such as speech, text, images) and capture the complex characteristics of emotions through the attention mechanism.

[0123] Among them, the multi-task learning model includes but is not limited to at least any one of the following: Cross-personal activity LSTM multi-task autoencoder (CALMNet): simultaneously predicting basic emotion probabilities and compound emotion scores to improve the generalization ability of the model; sub-indicator multi-task model: performing multi-task learning on different emotion indicators to improve the prediction accuracy of the model; personalized multi-task network model: optimizing emotion prediction based on individual differences to improve the personalization ability of the model.

[0124] Through the above technologies, the model can achieve comprehensive analysis and accurate evaluation of emotional states, providing reliable data support for emotional assessment.

[0125] The emotion assessment results include various information that can indicate the type and intensity of the emotional state of the subject to be assessed.

[0126] According to one embodiment, for each subject to be evaluated, the method stores the emotion evaluation data of each emotion evaluation performed on the subject to be evaluated.

[0127] In step S104, the emotion prediction model is used to perform weighted calculation based on the emotion assessment data of the subject to be assessed and the emotion assessment data corresponding to one or more previous emotion assessments to obtain a corresponding emotion assessment result, wherein the weight of each emotion assessment data is dynamically determined based on a predetermined algorithm.

[0128] Specifically, when the subject to be evaluated is evaluated for the first time, the emotional state is predicted based on the collected data to obtain an emotional state classification result.

[0129] When the subject is evaluated a second time, the prediction model incorporates not only the data from the current evaluation but also the data from the first evaluation. Using a weighted approach, the data from both evaluations are combined to produce a classification of the current and ongoing emotional state.

[0130] For the third and subsequent assessments of the subject, the scope of data integration is further expanded to include data from all previous assessments in addition to the current assessment data. Using the same weighting method and algorithm combination, not only is the current emotional state classification result obtained, but also the continuous emotional state classification results over a longer period of time.

[0131] Among them, the predetermined algorithm includes but is not limited to a hierarchical adjustment algorithm of time series features, an algorithm based on an attention mechanism, and a Shapley value calculation model. The hierarchical adjustment algorithm of time series features fully considers the timeliness and relevance of the data. As the number of evaluations increases and time passes, the weight of each data is automatically adjusted, so that the recent data has a greater impact on the prediction results, thereby better reflecting the latest changes in the emotional state. The attention mechanism-based algorithm can automatically learn and focus on the key features and important information in different evaluation data, dynamically assign weights according to the contribution of different features to the prediction of emotional states, effectively improve the model's attention to important information, and enhance the accuracy of the prediction. The Shapley value calculation model, from the perspective of game theory, comprehensively evaluates the marginal contribution of each evaluation data to the final prediction result, providing a more scientific and reasonable quantitative basis for the determination of weights.

[0132] This embodiment adopts a progressive data fusion strategy, combining old data and new data for comprehensive prediction and calculation, which can dynamically evaluate the emotional state of the subject, achieve continuous tracking and accurate capture of the emotional change process, and effectively improve the continuity of emotional evaluation, the continuity of data, and the stability and accuracy of the results.

[0133] According to one embodiment, the method further includes step S107.

[0134] In step S107, based on the emotion evaluation result output by the emotion prediction model, corresponding evaluation feedback information is provided to the subject to be evaluated.

[0135] The evaluation feedback information includes emotion evaluation results and mental disorder assessment results.

[0136] Among them, the mental disorder assessment results include but are not limited to whether the patient suffers from anxiety disorder, depressive disorder, bipolar disorder, schizophrenia spectrum disorder, ADHD, cognitive disorder, etc.

[0137] According to one embodiment, the method trains the emotion prediction model through step S108.

[0138] In step S108 , the emotion prediction model is trained based on the emotion evaluation data of the plurality of sample objects, so that the emotion prediction model learns through training how to output corresponding emotion evaluation results based on the emotion evaluation data of the sample objects.

[0139] The preprocessing is the same or similar to the aforementioned preprocessing of the emotion evaluation data of the prediction object, and will not be described in detail here.

[0140] Among them, the enhancement processing includes but is not limited to synonym replacement or data expansion of text data, rotation, flipping, noise addition or color adjustment of image data, speed adjustment or spectrum adjustment of audio data, signal superposition or noise injection of physiological signal data.

[0141] In the embodiment of the present application, after preprocessing and enhancing the multimodal data, the model is initialized and pre-trained, the loss is calculated through forward propagation, the parameters are updated using the optimization algorithm, and the model performance is monitored and an early stopping mechanism is applied. When the new data accumulates to a certain level, online incremental training is performed to improve the performance and generalization ability of the model.

[0142] According to one embodiment, step S108 includes steps S1081 to S1084.

[0143] In step S1081, target training data is acquired, wherein the target training data includes multimodal emotion evaluation data corresponding to multiple sample objects and their annotation results.

[0144] The emotion assessment data includes but is not limited to voice data, facial video data, facial blood flow data, facial temperature data, eye movement data and / or cognitive behavior data.

[0145] The labeling results include but are not limited to the sample subject's emotional state type, mental disorder type, and information indicating the intensity of the emotion and the severity of the mental disorder.

[0146] Optionally, the emotion assessment results include information indicating transient emotional states and persistent emotional states. Transient emotional states are emotional states predicted based on transient data features, primarily basic emotional states such as happiness, sadness, anger, fear, disgust, surprise, and peace; persistent emotional states are emotional states predicted based on long-term and multiple accumulations of data features, primarily complex emotional states such as love, anxiety, depression, flow, ennui, jealousy, shame, anticipation, and worry.

[0147] According to one embodiment, multimodal training sample data, including speech data, facial video data, facial blood flow data, facial temperature data, eye movement data, and / or cognitive behavioral data, is loaded and preprocessed and enhanced. Next, the training sample data is annotated, including information indicating the sample subject's emotional state type, mental disorder type, emotional intensity, and mental disorder severity. The annotated dataset is then divided into a training dataset and a validation dataset according to a specific ratio.

[0148] During model training, the data set is divided into a training set and a validation set, and the accuracy of the validation set can be monitored to prevent model overfitting.

[0149] Among them, in the construction of the data set, questionnaires and scales are used to evaluate the emotional state of the sample subjects at the moment and within a specified time, including immediate basic emotional states (such as happiness, sadness, anger, fear, disgust, surprise and peace) and complex emotional states that last for a period of time (such as love, anxiety, depression, flow, burnout, jealousy, shame, expectation, worry). In addition, through interviews and assessments with professionals, the mental disorders of the sample subjects are evaluated (such as whether they suffer from anxiety disorders, depressive disorders, bipolar disorders, schizophrenia spectrum disorders, ADHD, cognitive disorders). By organizing, summarizing and aligning the above-mentioned collected data, a multimodal and multidimensional emotion evaluation data set is formed, and it is divided into a training set for model training and a validation set for model validation.

[0150] In step S1082, feature extraction and feature fusion processing are performed on the emotion evaluation data of the multiple sample objects.

[0151] In step S1083, the emotion evaluation result output by the emotion prediction model based on the features corresponding to the input emotion evaluation data is obtained.

[0152] In step S1084, the emotion prediction model is continuously updated based on the difference between the emotion evaluation results output by the emotion prediction model and the actual emotion evaluation results of the sample subjects until the difference meets a predetermined requirement. Specifically, multiple iterations of training are performed, and the loss between the emotion evaluation results obtained in each iteration and the actual emotion evaluation results is calculated. The model parameters are updated based on the calculated loss.

[0153] The predetermined requirements include one or more conditions for determining whether the expected training effect is achieved.

[0154] Optionally, the predetermined requirements include but are not limited to at least any one of the following conditions: the value of the loss function between the predicted emotion assessment result and the actual emotion assessment result of the sample object is lower than a preset threshold; the value of the predetermined performance indicator of the model reaches a preset threshold, and the performance indicator includes but is not limited to accuracy or recall rate, etc.; the number of iterations reaches a preset threshold.

[0155] According to one embodiment, a trained emotion prediction model is validated and evaluated using a cross-validation strategy and independent samples, and the model is comprehensively evaluated using multiple evaluation metrics. The cross-validation strategy includes K-fold cross-validation, leave-one-out, and time series segmentation. Evaluation methods include, but are not limited to, accuracy, F1-score, AUC, ROC, MAE (mean absolute error), and RMSE (root mean square error).

[0156] According to one embodiment, the optimal emotion prediction model is determined by comparing the performance of unimodal and multimodal emotion prediction models.

[0157] According to one embodiment, a Monte Carlo Dropout method is used to generate confidence intervals for prediction results to quantify the uncertainty of the sentiment prediction model output. When the prediction confidence falls below a set threshold, a dynamic ecological instantaneous assessment (EMA) verification mechanism is triggered. This mechanism dynamically adjusts model parameters based on real-time verification results and iteratively optimizes model performance to improve model prediction accuracy and generalization capabilities.

[0158] According to an embodiment of the present application, the emotion prediction model adopts a meta-learning algorithm for pre-training and automatic iterative update to adapt to different objects to be evaluated and improve the performance, efficiency and robustness of the model prediction.

[0159] The goal of meta-learning is to teach models how to learn quickly. In the context of emotional state classification, MAML (Model-Agnostic Meta-Learning) pre-trains a meta-model, enabling it to quickly adapt to individual differences when faced with new users (new tasks) using only a small amount of data (e.g., 10 samples) and a few gradient updates, thereby improving the model's classification accuracy.

[0160] The MAML architecture is designed to enable rapid adaptation to new tasks through an optimization process. The core idea of ​​MAML is to find a set of initialization parameters that allows the model to quickly adapt to new tasks with a small number of gradient updates.

[0161] According to one embodiment, pre-training and automatic iterative updating using a meta-learning algorithm are achieved through a meta-training phase (Meta-Training) and rapid adaptation to new users (Meta-Testing).

[0162] The meta-training phase includes the following steps:

[0163] Task definition: Each user is considered as an independent task, and each task is divided into a support set and a query set;

[0164] Data preparation: Input data includes multimodal features and emotional state labels. K samples are randomly sampled from each user as the support set, and the remaining samples are used as the query set. The multimodal features include but are not limited to features corresponding to speech, eye movements, facial video, facial blood flow, facial temperature, or cognitive behavior.

[0165] Model architecture construction: Build a shared feature extractor and a multi-task head. The shared feature extractor is used to process multimodal data, and the multi-task head is used for classification and regression dual-task output.

[0166] Execute the MAML training process: This includes executing the inner loop and the outer loop. The inner loop updates the model parameters for each task using the support set, while the outer loop updates the meta-model parameters based on the loss of the query set.

[0167] The rapid adaptation process includes the following steps:

[0168] Initialization: load pre-trained meta-model parameters;

[0169] Perform the adaptation process: Use a small amount of support data (e.g., 10 samples) for the new user and perform an inner loop gradient update. The adapted parameters are used for predictions for that user.

[0170] Optionally, the method evaluates the adaptation effect of the meta-training. Specifically, the performance of the adapted model is tested using a new user's query set. This evaluation can be achieved by assessing the accuracy of the classification task and the severity score prediction error of the regression task.

[0171] In the context of emotion assessment, since each user has different ways of expressing emotions when conducting emotion assessments, traditional machine learning models may need to collect a large amount of data to adapt to new users. However, the embodiments of the present application use a meta-learning algorithm to pre-train the model and automatically iterate and update it. During the meta-training phase, the model learns general knowledge and learning strategies. When facing new users, it can use a small amount of user data for adaptation, achieve rapid adaptation to new users (new tasks), and thus achieve rapid and accurate recognition of the new user's emotional state.

[0172] According to the method of the embodiment of the present application, by obtaining emotion assessment data of multiple modalities and using a trained emotion prediction model to output corresponding emotion assessment results based on the feature information of the emotion assessment data of multiple modalities, this multimodal data fusion method reduces the limitations of a single data source, thereby significantly improving the accuracy and reliability of the emotion assessment results; during the data collection process, a variety of methods are used to automatically verify the validity of the data to ensure data quality; the model of the embodiment of the present application uses a meta-learning algorithm for pre-training and automatic iterative updates, and only requires a small amount of data and a few steps of gradient updates to quickly adapt to individual differences, realize the personalization and dynamism of the assessment, and further improve the performance, efficiency and robustness of the model prediction.

[0173] Reference Figure 2 , Figure 2 A schematic diagram of an exemplary intelligent emotion assessment system according to an embodiment of the present application is shown. The system includes a data acquisition module, a data preprocessing and feature extraction module, a multimodal data feature fusion module, an emotion assessment module, and an emotion assessment result output and feedback module.

[0174] Among them, the data acquisition module is used to obtain the tester's voice data, facial video data, facial blood flow data, facial temperature data, eye movement data and cognitive behavior data.

[0175] This example uses a specially developed software system, combined with hardware devices (such as mobile phones, computers, tablets, smartwatches, and other terminal devices) to collect test-taker emotional assessment data. The emotional assessment data collection scheme is designed based on the basic principles of experimental psychology and cognitive psychology. The software system presents the corresponding emotional assessment tasks on the screen in sequence, such as facial capture, emotional image browsing, and emotional question-and-answer tasks. Test-takers complete these tasks in sequence according to the software's guidance.

[0176] While the tester is performing the task, the system will synchronously collect the following data: Voice data: The device's microphone is used to record the tester's voice performance when completing voice-related tasks (such as single-tone pronunciation tasks, voice question-and-answer tasks, etc.); Facial video data: The device's camera is used to record the tester's facial expression changes when completing the task; Facial blood flow data: During the presentation of pictures or video stimuli or the execution of cognitive tasks, the device camera (such as the front camera of a mobile phone) is used based on visible light imaging technology to capture facial blood flow signals through photoplethysmography (rPPG); Facial temperature data: During the presentation of pictures or video stimuli or the execution of cognitive tasks, the device camera (such as the front camera of a mobile phone) is used based on infrared thermal imaging technology to capture facial blood flow signals through infrared imaging technology. The software uses a mobile phone camera with a module to collect facial temperature distribution data; eye movement data: the tester needs to perform an eye movement calibration task, the gaze point of which is generated by the software system configuration. During the tester's eye movement correction process, the software system automatically records the tester's eye movement trajectory and matches it with the logical point (gaze point), and realizes eye movement calibration through a preset eye movement calibration algorithm; at the same time, the eye movement trajectory state of the subject to be evaluated is calculated and identified based on the predetermined eye movement trajectory recognition algorithm to collect, calculate and record the eye movement data of the subject to be evaluated; cognitive behavioral data: the software system records the tester's behavioral performance when completing cognitive tasks (such as point detection tasks, visual search tasks, etc.), including reaction time, reaction accuracy, task completion status, etc.

[0177] Among them, the data preprocessing and feature extraction module is used to perform preprocessing such as cleaning and standardization on multimodal data, and to extract features from the acquired data.

[0178] Among them, the multimodal data feature fusion module is used for feature fusion.

[0179] Among them, the emotion assessment module is used to output the emotional state classification and emotion intensity assessment results based on the input fusion features through a trained model. Among them, deep belief network (DBN), convolutional neural network (CNN), artificial neural network (ANN), support vector machine (SVM), logistic regression, linear regression algorithm, Transformer algorithm, and integrated machine learning algorithm are used to create the model. The created model can not only distinguish the type of emotion, but also assess the intensity of these emotions, such as mild, moderate, severe, etc. The model is trained in the manner of step S108 above, which will not be repeated here.

[0180] Among them, the emotion assessment result output and feedback module is used to feedback the emotion assessment results and mental disorder assessment results, including instantaneous emotional state and persistent emotional state. The instantaneous emotional state is the emotional state predicted based on short-term data features, mainly basic emotional state; the persistent emotional state is the emotional state obtained by calculating and predicting based on long-term and multiple data feature accumulation, mainly complex emotional state.

[0181] The intelligent emotion assessment system in this example uses multimodal data for emotion assessment. The data from each modality can verify and complement each other, making the assessment results output by the model more accurate and reliable.

[0182] Figure 3 A schematic structural diagram of a device for emotion assessment provided in an embodiment of the present application is shown.

[0183] The device includes: a device for obtaining multimodal emotion assessment data of an object to be evaluated (hereinafter referred to as "data acquisition device 101"), a device for performing feature extraction processing on each data contained in the emotion assessment data during the process of the object to be evaluated performing a predetermined plurality of emotion assessment tasks to obtain corresponding feature information (hereinafter referred to as "feature acquisition device 102"), a device for fusing the feature information corresponding to each data to obtain corresponding fused feature information (hereinafter referred to as "fusion processing device 103"), and a device for using a trained emotion prediction model to output a corresponding emotion assessment result based on the fused feature information (hereinafter referred to as "emotion assessment device 104").

[0184] Reference Figure 3 In the process of the subject to be evaluated performing a plurality of predetermined emotion evaluation tasks, the data acquisition device 101 acquires multimodal emotion evaluation data of the subject to be evaluated.

[0185] The subject to be evaluated refers to a subject that needs to undergo emotion evaluation.

[0186] The emotion assessment data includes various data that can reflect the emotions of the subject to be assessed. The emotion assessment data can come from a variety of sensor devices and data sources.

[0187] According to one embodiment, the emotion assessment data may include voice data, facial video data, facial blood flow data, facial temperature data, eye movement data and / or cognitive behavior data.

[0188] Among them, various data have been described in the previous article and will not be repeated here.

[0189] The data collection of voice data, facial video data, eye movement data, facial blood flow data, facial temperature data, and cognitive behavior data can be completed on a single hardware device, such as using only a computer to collect all the data. Alternatively, the above data can be collected simultaneously using multiple devices.

[0190] Optionally, the acquired voice data, facial video data, facial blood flow data, facial temperature data, eye movement data, and / or cognitive behavior data are algorithmically corrected based on factors such as ambient temperature, ambient noise, lighting, and motion to eliminate interference from factors such as ambient temperature, ambient noise, lighting, motion, and makeup. Furthermore, device sensor data (such as a mobile phone gyroscope) is used to assist in tracking voice, facial images, blood flow, temperature, eye movement, and / or behavioral characteristics to address data distortion caused by device movement or sudden changes in lighting.

[0191] According to one embodiment, the data acquisition device 101 is used to perform the following operations: presenting a plurality of predetermined emotion assessment tasks, wherein the emotion assessment tasks include general emotion assessment tasks and cognitive emotion assessment tasks; while the subject to be assessed is performing the emotion assessment task, the voice data, facial video data, facial blood flow data, facial temperature data, eye movement data and / or cognitive behavior data of the subject to be assessed are synchronously collected as emotion assessment data.

[0192] The meanings of the general emotion assessment task and the cognitive emotion assessment task have been described in the previous text and will not be repeated here.

[0193] According to one embodiment, before the subject to be evaluated performs the emotion evaluation task, the device performs eye movement calibration processing based on the image or video data of the subject to be evaluated.

[0194] Specifically, an eye movement calibration task is configured, and the gaze points of the eye movement calibration task are generated by a software system included in the equipment for performing the emotion assessment task; then, during the process of the eye movement calibration of the subject to be assessed, the eye movement trajectory of the subject to be assessed is automatically recorded; then, the recorded eye movement trajectory is matched with the logical points, and the eye movement calibration is achieved through a preset eye movement calibration algorithm; then, the eye movement trajectory state of the subject to be assessed is calculated and identified based on a predetermined eye movement trajectory recognition algorithm, so as to collect, calculate and record the eye movement data of the subject to be assessed.

[0195] Among them, the eye movement calibration algorithm and eye movement trajectory recognition algorithm have been described in the previous article and will not be repeated here.

[0196] According to one embodiment the method comprises a data verification device.

[0197] The data verification device automatically verifies the validity of the collected emotion assessment data.

[0198] The data verification device verifies the validity of the collected emotion assessment data online and automatically through multiple methods.

[0199] Among them, the method of verifying the validity of voice data, facial video data, eye movement data, facial blood flow data, facial temperature data and cognitive behavior data has been described in the previous article and will not be repeated here.

[0200] If the validity verification fails, the data acquisition device 101 reacquires the corresponding emotion assessment data.

[0201] By conducting validity verification, we ensure that the collected data is of high quality and high reliability during the emotion assessment process, thereby ensuring the accuracy of subsequent emotion assessments.

[0202] According to one embodiment, the device comprises pre-processing means.

[0203] The preprocessing device preprocesses the acquired emotion assessment data.

[0204] The preprocessing includes, but is not limited to, data cleaning, data standardization and normalization, data rectification, data quality assessment, and data preservation. Data cleaning includes, but is not limited to, data review, downsampling, noise reduction (artifact processing, filtering, outlier processing, and useless data processing), missing value processing, data alignment, and time synchronization (based on the PTP protocol to ensure the alignment of multimodal data timestamps).

[0205] Continue to refer to Figure 3 To illustrate, the feature acquisition device 102 performs feature extraction processing on each data included in the emotion evaluation data to obtain corresponding feature information.

[0206] Specifically, the feature acquisition device 102 determines the type of emotion assessment data to be processed and adopts a feature extraction operation that matches the type.

[0207] Specifically, the feature acquisition device 102 calculates and extracts one or more of statistics, frequency domain features, time domain features, nonlinear dynamic features, or waveform features from the emotion assessment data based on different data feature types. Statistics include, but are not limited to, mean and variance, and frequency domain features include fast Fourier transform (FFT) energy. Time domain features include, but are not limited to, peak values ​​and peak intervals within a sliding window. Nonlinear dynamic features include, but are not limited to, multiscale entropy and fractal dimension. Waveform features include, but are not limited to, rise time, number of peaks, and area integral.

[0208] Among them, the feature extraction operations for voice data, facial video data, eye movement data, facial blood flow data, facial temperature data and cognitive behavior data have been described in the previous article and will not be repeated here.

[0209] To enhance data privacy protection, the device in the embodiment of the present application uses biometric desensitization technology to convert raw blood flow signals into abstract vascular motion feature vectors. This conversion method can retain emotion-related features in the data while removing raw biometric details that may contain personal identification information, thereby effectively protecting the user's privacy. Specifically, for facial temperature data, the embodiment of the present application only collects and records relative temperature changes, not absolute temperature values. This processing method can reduce the personal identification information that may be contained in the data while still retaining features valuable for emotion analysis, such as temperature change trends and patterns.

[0210] Moreover, in terms of data processing architecture, the device of the embodiment of the present application is designed based on the edge computing framework. Based on this design, the original data will be stored on the user's local device and will not be transmitted to an external server or other device. All feature extraction and preliminary analysis work in the embodiment of the present application will be completed on the local device side, thereby minimizing the risk of data leakage. In certain scenarios that require multi-device collaboration, each terminal device will only upload encrypted parameters, which are difficult to decrypt and interpret during transmission, further ensuring the security and privacy of user data.

[0211] The fusion processing device 103 performs fusion processing on the feature information corresponding to each data item to obtain corresponding fusion feature information.

[0212] The fusion processing method has been described above and will not be repeated here.

[0213] The emotion evaluation device 104 uses the trained emotion prediction model to output corresponding emotion evaluation results based on the fused feature information.

[0214] The emotion prediction model adopts a meta-learning algorithm for pre-training and automatic iterative update to adapt to different objects to be evaluated and improve the performance, efficiency and robustness of the model prediction.

[0215] The emotion prediction model is a machine learning-based intelligent classification and evaluation model for intelligently classifying and evaluating emotional states. Optionally, the emotion prediction model is used to identify the type of emotion and assess the intensity of the emotion, such as mild, moderate, severe, or expressed as a percentage.

[0216] The emotion prediction model can be constructed using a variety of machine learning techniques. Optionally, the emotion prediction model is constructed using a deep learning model and a multi-task learning model.

[0217] Among them, the deep learning model includes but is not limited to at least any one of the following: long short-term memory network (LSTM): used to process sequence data and capture emotional changes in time series; convolutional neural network (CNN): used to extract local features in images or text, suitable for emotion recognition; graph convolutional network (GCN): used to process graph structure data and analyze the propagation of emotions in the network; CNN-LSTM hybrid model: combining the feature extraction capability of CNN and the sequence processing capability of LSTM to improve the robustness of the model; multimodal Transformer model: used to process multi-source data (such as speech, text, images) and capture the complex characteristics of emotions through the attention mechanism.

[0218] Among them, the multi-task learning model includes but is not limited to at least any one of the following: Cross-personal activity LSTM multi-task autoencoder (CALMNet): simultaneously predicting basic emotion probabilities and compound emotion scores to improve the generalization ability of the model; sub-indicator multi-task model: performing multi-task learning on different emotion indicators to improve the prediction accuracy of the model; personalized multi-task network model: optimizing emotion prediction based on individual differences to improve the personalization ability of the model.

[0219] Through the above technologies, the model can achieve comprehensive analysis and accurate evaluation of emotional states, providing reliable data support for emotional assessment.

[0220] The emotion assessment results include various information that can indicate the type and intensity of the emotional state of the subject to be assessed.

[0221] According to one embodiment, for each subject to be evaluated, the device stores the emotion evaluation data of each emotion evaluation performed on the subject to be evaluated.

[0222] The emotion assessment device 104 uses the emotion prediction model to perform weighted calculation based on the emotion assessment data of the object to be assessed and the emotion assessment data corresponding to one or more previous emotion assessments to obtain a corresponding emotion assessment result, wherein the weight of each emotion assessment data is dynamically determined based on a predetermined algorithm.

[0223] Specifically, when the subject to be evaluated is evaluated for the first time, the emotional state is predicted based on the collected data to obtain an emotional state classification result.

[0224] When the subject is evaluated a second time, the prediction model incorporates not only the data from the current evaluation but also the data from the first evaluation. Using a weighted approach, the data from both evaluations are combined to produce a classification of the current and ongoing emotional state.

[0225] For the third and subsequent assessments of the subject, the scope of data integration is further expanded to include data from all previous assessments in addition to the current assessment data. Using the same weighting method and algorithm combination, not only is the current emotional state classification result obtained, but also the continuous emotional state classification results over a longer period of time.

[0226] Among them, the predetermined algorithm includes but is not limited to a hierarchical adjustment algorithm of time series features, an algorithm based on an attention mechanism, and a Shapley value calculation model. The hierarchical adjustment algorithm of time series features fully considers the timeliness and relevance of the data. As the number of evaluations increases and time passes, the weight of each data is automatically adjusted, so that the recent data has a greater impact on the prediction results, thereby better reflecting the latest changes in the emotional state. The attention mechanism-based algorithm can automatically learn and focus on the key features and important information in different evaluation data, dynamically assign weights according to the contribution of different features to the prediction of emotional states, effectively improve the model's attention to important information, and enhance the accuracy of the prediction. The Shapley value calculation model, from the perspective of game theory, comprehensively evaluates the marginal contribution of each evaluation data to the final prediction result, providing a more scientific and reasonable quantitative basis for the determination of weights.

[0227] The device of this embodiment adopts a progressive data fusion strategy, combining old data and new data for comprehensive prediction calculations, which can dynamically evaluate the emotional state of the subject, achieve continuous tracking and accurate capture of the emotional change process, and effectively improve the continuity of emotional evaluation, the continuity of data, and the stability and accuracy of the results.

[0228] According to one embodiment, the method further includes an information feedback device.

[0229] The information feedback device provides corresponding evaluation feedback information to the subject to be evaluated based on the emotion evaluation result output by the emotion prediction model.

[0230] The evaluation feedback information includes emotion evaluation results and mental disorder assessment results.

[0231] Among them, the mental disorder assessment results include but are not limited to whether the patient suffers from anxiety disorder, depressive disorder, bipolar disorder, schizophrenia spectrum disorder, ADHD, cognitive disorder, etc.

[0232] According to one embodiment, the device comprises a model training device.

[0233] The model training device trains the emotion prediction model based on the emotion evaluation data of multiple sample objects, so that the emotion prediction model learns through training how to output corresponding emotion evaluation results based on the emotion evaluation data of the sample objects.

[0234] The preprocessing is the same or similar to the aforementioned preprocessing of the emotion evaluation data of the prediction object, and will not be described in detail here.

[0235] Among them, the enhancement processing includes but is not limited to synonym replacement or data expansion of text data, rotation, flipping, noise addition or color adjustment of image data, speed adjustment or spectrum adjustment of audio data, signal superposition or noise injection of physiological signal data.

[0236] In the embodiment of the present application, after preprocessing and enhancing the multimodal data, the model is initialized and pre-trained, the loss is calculated through forward propagation, the parameters are updated using the optimization algorithm, and the model performance is monitored and an early stopping mechanism is applied. When the new data accumulates to a certain level, online incremental training is performed to improve the performance and generalization ability of the model.

[0237] According to one embodiment, the model training device includes a training data acquisition device, a training data processing device, an output result acquisition device and a model updating device.

[0238] The training data acquisition device acquires target training data. The target training data includes multimodal emotion assessment data corresponding to multiple sample objects and their annotation results. The emotion assessment data includes but is not limited to voice data, facial video data, facial blood flow data, facial temperature data, eye movement data and / or cognitive behavior data. The annotation results include but are not limited to the sample object's emotional state type, mental disorder type and information indicating the intensity of emotion and the severity of mental disorder. Optionally, the emotion assessment results include information for indicating transient emotional state and persistent emotional state. The transient emotional state is an emotional state predicted based on short-term data features, mainly basic emotional states such as happiness, sadness, anger, fear, disgust, surprise and peace; the persistent emotional state is an emotional state predicted based on long-term and multiple accumulation of data features, mainly complex emotional states such as love, anxiety, depression, flow, burnout, jealousy, shame, expectation, worry, etc.

[0239] According to one embodiment, multimodal training sample data, including speech data, facial video data, facial blood flow data, facial temperature data, eye movement data, and / or cognitive behavioral data, is loaded and preprocessed and enhanced. Next, the training sample data is annotated, including information indicating the sample subject's emotional state type, mental disorder type, emotional intensity, and mental disorder severity. The annotated dataset is then divided into a training dataset and a validation dataset according to a specific ratio.

[0240] During model training, the data set is divided into a training set and a validation set, and the accuracy of the validation set can be monitored to prevent model overfitting.

[0241] Among them, in the construction of the data set, questionnaires and scales are used to evaluate the emotional state of the sample subjects at the moment and within a specified time, including immediate basic emotional states (such as happiness, sadness, anger, fear, disgust, surprise and peace) and complex emotional states that last for a period of time (such as love, anxiety, depression, flow, burnout, jealousy, shame, expectation, worry). In addition, through interviews and assessments with professionals, the mental disorders of the sample subjects are evaluated (such as whether they suffer from anxiety disorders, depressive disorders, bipolar disorders, schizophrenia spectrum disorders, ADHD, cognitive disorders). By organizing, summarizing and aligning the above-mentioned collected data, a multimodal and multidimensional emotion evaluation data set is formed, and it is divided into a training set for model training and a validation set for model validation.

[0242] The training data processing device performs feature extraction processing and feature fusion processing on the emotion evaluation data of the multiple sample objects.

[0243] The output result obtaining device obtains the emotion evaluation result output by the emotion prediction model based on the features corresponding to the input emotion evaluation data.

[0244] The model updating device continuously updates the emotion prediction model based on the difference between the emotion evaluation results output by the emotion prediction model and the actual emotion evaluation results of the sample subjects until the difference meets a predetermined requirement. Specifically, the model updating device performs multiple iterations of training, calculates the loss between the emotion evaluation results obtained in each iteration and the actual emotion evaluation results, and updates the model parameters based on the calculated loss.

[0245] The predetermined requirements include one or more conditions for determining whether the expected training effect is achieved.

[0246] Optionally, the predetermined requirements include but are not limited to at least any one of the following conditions: the value of the loss function between the predicted emotion assessment result and the actual emotion assessment result of the sample object is lower than a preset threshold; the value of the predetermined performance indicator of the model reaches a preset threshold, and the performance indicator includes but is not limited to accuracy or recall rate, etc.; the number of iterations reaches a preset threshold.

[0247] According to one embodiment, the device comprises a verification evaluation device.

[0248] The validation and evaluation device uses a cross-validation strategy and independent samples to validate and evaluate the trained emotion prediction model, comprehensively evaluating the model using multiple evaluation metrics. Cross-validation strategies include K-fold cross-validation, leave-one-out, and time series segmentation. Evaluation methods include, but are not limited to, accuracy, F1-score, AUC, ROC, MAE (mean absolute error), and RMSE (root mean square error).

[0249] According to one embodiment, the verification and evaluation device determines the optimal emotion prediction model by comparing the performance of single-modal and multi-modal emotion prediction models.

[0250] According to one embodiment, the verification and evaluation device uses the Monte Carlo Dropout method to generate confidence intervals for prediction results to quantify the uncertainty of the emotion prediction model output. When the prediction confidence falls below a set threshold, a dynamic ecological momentary assessment (EMA) verification mechanism is triggered. This mechanism dynamically adjusts model parameters based on real-time verification results and iteratively optimizes model performance to improve model prediction accuracy and generalization capabilities.

[0251] According to an embodiment of the present application, the emotion prediction model adopts a meta-learning algorithm for pre-training and automatic iterative update to adapt to different objects to be evaluated and improve the performance, efficiency and robustness of the model prediction.

[0252] The goal of meta-learning is to teach models how to learn quickly. In the context of emotional state classification, MAML (Model-Agnostic Meta-Learning) pre-trains a meta-model, enabling it to quickly adapt to individual differences when faced with new users (new tasks) using only a small amount of data (e.g., 10 samples) and a few gradient updates, thereby improving the model's classification accuracy.

[0253] According to one embodiment, pre-training using a meta-learning algorithm is achieved through a meta-training phase (Meta-Training) and rapid adaptation to new users (Meta-Testing).

[0254] The meta-training phase includes the following steps:

[0255] Task definition: Each user is considered as an independent task, and each task is divided into a support set and a query set;

[0256] Data preparation: Input data includes multimodal features and emotional state labels. K samples are randomly sampled from each user as the support set, and the remaining samples are used as the query set. The multimodal features include but are not limited to features corresponding to speech, facial blood flow, facial temperature, eye movements, and facial or cognitive behavior.

[0257] Model architecture construction: Build a shared feature extractor and a multi-task head. The shared feature extractor is used to process multimodal data, and the multi-task head is used for classification and regression dual-task output.

[0258] Execute the MAML training process: This includes executing the inner loop and the outer loop. The inner loop updates the model parameters for each task using the support set, while the outer loop updates the meta-model parameters based on the loss of the query set.

[0259] The rapid adaptation process includes the following steps:

[0260] Initialization: load pre-trained meta-model parameters;

[0261] Perform the adaptation process: Use a small amount of support data (e.g., 10 samples) for the new user and perform an inner loop gradient update. The adapted parameters are used for predictions for that user.

[0262] Optionally, the apparatus evaluates the adaptation effect of the meta-training. Specifically, the performance of the adapted model is tested using a new user's query set. This evaluation can be achieved by assessing the accuracy of the classification task and the severity score prediction error of the regression task.

[0263] According to the device of the embodiment of the present application, by acquiring emotion assessment data of multiple modalities and using a trained emotion prediction model to output corresponding emotion assessment results based on the feature information of the emotion assessment data of multiple modalities, this multimodal data fusion method reduces the limitations of a single data source, thereby significantly improving the accuracy and reliability of the emotion assessment results; during the data collection process, a variety of methods are used to automatically verify the validity of the data to ensure data quality; the model of the embodiment of the present application uses a meta-learning algorithm for pre-training and automatic iterative updates, and only requires a small amount of data and a few steps of gradient updates to quickly adapt to individual differences, realize the personalization and dynamism of the assessment, and further improve the performance, efficiency and robustness of the model prediction.

[0264] Based on the same inventive concept, an electronic device is also provided in an embodiment of the present application. The method corresponding to the electronic device may be the method for performing emotion assessment in the aforementioned embodiment, and its principle of solving the problem is similar to that of the method. The electronic device provided in an embodiment of the present application includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the methods and / or technical solutions of the aforementioned multiple embodiments of the present application.

[0265] The electronic device may be a user device, or a device formed by integrating a user device and a network device via a network, or an application running on the above device. The user device includes but is not limited to various terminal devices such as computers, mobile phones, tablets, smart watches, and bracelets. The network device includes but is not limited to network hosts, single network servers, multiple network server sets, or cloud computing-based computer collections, and can be used to implement some of the processing functions when setting an alarm. Here, the cloud is composed of a large number of hosts or network servers based on cloud computing (Cloud Computing), where cloud computing is a type of distributed computing, a virtual computer composed of a group of loosely coupled computers.

[0266] Figure 4 The structure of a device suitable for implementing the method and / or technical solution in the embodiment of the present application is shown. The device 1200 includes a central processing unit (CPU) 1201, which can perform various appropriate actions and processes according to the program stored in the read-only memory (ROM) 1202 or the program loaded from the storage part 1208 into the random access memory (RAM) 1203. Various programs and data required for system operation are also stored in RAM 1203. CPU 1201, ROM 1202 and RAM 1203 are connected to each other through a bus 1204. Input / output (I / O) interface 1205 is also connected to bus 1204.

[0267] The following components are connected to the I / O interface 1205: an input section 1206 including a keyboard, a mouse, a touch screen, a microphone, an infrared sensor, and the like; an output section 1207 including devices such as a cathode ray tube (CRT), a liquid crystal display (LCD), an LED display, an OLED display, and a speaker; a storage section 1208 including one or more computer-readable media such as a hard disk, an optical disk, a magnetic disk, and a semiconductor memory; and a communication section 1209 including a network interface card such as a LAN (Local Area Network) card, a modem, and the like. The communication section 1209 performs communication processing via a network such as the Internet.

[0268] In particular, the methods and / or embodiments of the present application can be implemented as computer software programs. For example, the embodiments disclosed in the present application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for executing the method shown in the flowchart. When the computer program is executed by the central processing unit (CPU) 1201, the above-mentioned functions defined in the method of the present application are performed.

[0269] Among them, the programs include mobile phone programs (APP programs, mini-programs, etc.) and smart device programs, and smart devices include watches, bracelets, helmets, smart hardware devices, etc.

[0270] Another embodiment of the present application further provides a computer-readable storage medium having computer program instructions stored thereon, which can be executed by a processor to implement the methods and / or technical solutions of any one or more embodiments of the present application.

[0271] Specifically, the present embodiment can adopt any combination of one or more computer-readable media. The computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or component, or any combination thereof. More specific examples (non-exhaustive list) of computer-readable storage media include: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In this document, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device.

[0272] A computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, which carries computer-readable program code. This propagated data signal may take a variety of forms, including, but not limited to, electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or transport a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0273] Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0274] Computer program code for performing the operations of the present application can be written in one or more programming languages, or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0275] The flow chart or block diagram in the accompanying drawings illustrate the possible architecture, functions and operations of the equipment, methods and computer program products according to various embodiments of the present application. In this regard, each box in the flow chart or block diagram can represent a module, program segment or a part of code, and the module, program segment or a part of code include one or more executable instructions for realizing the logical function of the specification. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a sequence different from that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be implemented with a dedicated system for hardware that performs the function or operation of the specification, or can be implemented with a combination of dedicated hardware and computer instructions.

[0276] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0277] In the several embodiments provided in this application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or page components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0278] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0279] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0280] The above-mentioned integrated unit implemented in the form of a software functional unit can be stored in a computer-readable storage medium. The above-mentioned software functional unit is stored in a storage medium and includes a number of instructions for causing a computer device (which can be a personal computer, server, or network device, etc.) or a processor to execute some steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and other media that can store program code.

[0281] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

[0282] Furthermore, it is clear that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. Multiple units or devices recited in a device claim may also be implemented by a single unit or device through software or hardware. Terms such as "first" and "second" are used to indicate names and do not imply any particular order.

Claims

1. A method for emotion assessment, wherein: The method comprises: Acquiring multimodal emotion assessment data of the subject to be assessed during the subject to be assessed performs a plurality of predetermined emotion assessment tasks; Performing feature extraction processing on each data item included in the emotion assessment data to obtain corresponding feature information; Fusing the feature information corresponding to each data item to obtain corresponding fused feature information; A trained emotion prediction model is used to output corresponding emotion evaluation results based on the fused feature information, wherein the emotion prediction model is pre-trained and automatically iteratively updated using a meta-learning algorithm.

2. The method according to claim 1, wherein The step of obtaining multimodal emotion assessment data of the subject to be assessed includes: Presenting a plurality of predetermined emotion assessment tasks, wherein the emotion assessment tasks include a general emotion assessment task and a cognitive emotion assessment task; While the subject to be evaluated is performing the emotion evaluation task, voice data, facial video data, facial blood flow data, facial temperature data, eye movement data and / or cognitive behavior data of the subject to be evaluated are synchronously collected as emotion evaluation data.

3. The method according to claim 1 or 2, wherein: The method further comprises: Automatically verify the validity of collected emotion assessment data; If the validity verification fails, the corresponding emotion assessment data will be collected again.

4. The method according to claim 1 or 2, wherein: The method performs eye movement calibration processing based on image or video data of the subject to be evaluated, and the method further includes: Configuring an eye movement calibration task, wherein the gaze points of the eye movement calibration task are generated by a software system included in the device performing the emotion assessment task; Automatically record the eye movement trajectory of the subject to be evaluated during the eye movement calibration process; Match the recorded eye movement trajectory with the logical points and implement eye movement calibration through the preset eye movement calibration algorithm; The eye movement trajectory state of the subject to be evaluated is calculated and identified based on a predetermined eye movement trajectory recognition algorithm to collect, calculate and record the eye movement data of the subject to be evaluated.

5. The method according to claim 1 or 2, wherein: The method further includes performing correction processing on the multimodal data, and the method further includes: Perform algorithmic correction on the acquired voice data, facial video data, facial blood flow data, facial temperature data, eye movement data, and / or cognitive behavior data based on factors such as ambient temperature, ambient noise, lighting, and motion, to eliminate interference from factors such as ambient temperature, ambient noise, lighting, motion, and makeup; Leverage device sensor data to assist in tracking voice, facial images, blood flow, temperature, eye movement, and / or behavioral characteristics, addressing data distortion caused by device movement or sudden changes in lighting.

6. The method according to any one of claims 1 to 3, wherein The method further comprises: Based on the emotion prediction model outputting the corresponding emotion evaluation result, corresponding evaluation feedback information is provided to the subject to be evaluated, wherein the evaluation feedback information includes the emotion evaluation result and the mental disorder evaluation result.

7. The method according to claim 1, wherein The method further comprises: For each subject to be evaluated, the emotion evaluation data of each emotion evaluation of the subject to be evaluated is stored; The method of using the trained emotion prediction model to output corresponding emotion evaluation results based on the fused feature information includes: Using the emotion prediction model, a weighted calculation is performed based on the current emotion evaluation data of the subject to be evaluated and the emotion evaluation data corresponding to one or more previous ones to obtain a corresponding emotion evaluation result, wherein the weight of each emotion evaluation data is dynamically determined based on a predetermined algorithm.

8. The method according to any one of claims 1 to 3, wherein The method further comprises: The emotion prediction model is trained based on the emotion evaluation data of a plurality of sample objects, so that the emotion prediction model learns through training how to output corresponding emotion evaluation results based on the emotion evaluation data of the sample objects.

9. The method according to claim 7, wherein: The training of the emotion prediction model based on the emotion evaluation data of the plurality of sample objects includes: Acquire target training data, wherein the target training data includes multimodal emotion assessment data corresponding to multiple sample objects and their annotation results; performing feature extraction and feature fusion processing on the emotion evaluation data of the multiple sample objects; Obtaining the emotion evaluation result output by the emotion prediction model based on the features corresponding to the input emotion evaluation data; According to the difference between the emotion evaluation result output by the emotion prediction model and the actual emotion evaluation result of the sample object, the emotion prediction model is continuously updated until the difference meets the predetermined requirement.

10. The method according to claim 7 or 8, wherein The use of meta-learning algorithms for pre-training and automatic iterative updates is achieved through the meta-training phase and rapid adaptation to new users.

11. The method according to claim 7 or 8, wherein: The method further comprises: Monte Carlo methods are used to generate confidence intervals for prediction results to quantify the uncertainty of the sentiment prediction model output; When the prediction confidence is lower than the set threshold, the dynamic ecological instantaneous evaluation and verification mechanism is triggered, the model parameters are dynamically adjusted through real-time verification results, and the model performance is iteratively optimized.

12. A device for emotional assessment, wherein: The device comprises: A device for obtaining multimodal emotion assessment data of a subject to be assessed; A device for performing feature extraction processing on each data item included in the emotion assessment data to obtain corresponding feature information; A device for fusing feature information corresponding to each data item to obtain corresponding fused feature information; A device for using a trained emotion prediction model to output corresponding emotion evaluation results based on the fused feature information, wherein the emotion prediction model adopts a meta-learning algorithm for pre-training and automatic iterative update.

13. An electronic device, comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 11.

14. A computer-readable medium having computer program instructions stored thereon, wherein the computer program instructions can be executed by a processor to implement the method according to any one of claims 1 to 11.

15. A computer program product comprising a computer program, wherein when the computer program is executed by a processor, the method according to any one of claims 1 to 11 is implemented.

Images