Multi-modal emotion recognition method based on AI vision

Abstract

The invention discloses a multi-modal emotion recognition method based on AI vision. The multi-modal emotion recognition method comprises the following steps: acquiring facial image data, voice data and text data of a user and preprocessing the facial image data, the voice data and the text data; performing feature extraction on the preprocessed facial image data by adopting an HOG algorithm to obtain visual emotion features; performing feature extraction on the preprocessed voice data by adopting a TEO method to obtain voice emotion features; performing feature extraction on the preprocessed text data based on a pre-training language model to obtain text emotion features; performing time sequence alignment to obtain a time sequence alignment multi-modal feature sequence; a fused multi-modal emotion feature is obtained; outputting an emotion recognition result by using the improved VA emotion model; according to the method, stable recognition and continuous tracking of the emotional state of the user are realized, and robustness and time sequence consistency of emotional recognition are improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of human-computer interaction, and in particular to a multi-modal emotion recognition method based on AI vision. BACKGROUND

[0002] With the popularity of intelligent terminals, service robots, intelligent cockpits, online education and remote interaction systems, human-computer interaction has gradually developed from "command interaction" to "understanding interaction". The system not only needs to recognize user input content, but also needs to perceive the user's emotional and emotional state, so as to make a response that is more in line with the user's state in terms of interaction strategy, feedback mode and service decision. Emotion recognition, as an important part of affective computing, usually relies on the analysis of user facial expressions, speech expressions and text semantics, etc. to infer user emotions. However, in actual application scenarios, user emotional expression has diversity, instantaneity and context relevance, and data collection is often accompanied by noise, occlusion, time sequence asynchronization and other problems, which makes the existing emotion recognition technology still have deficiencies in robustness, real-time performance and cross-scene adaptability.

[0003] In the prior art, single-modal emotion recognition methods are more common. Vision-based emotion recognition mainly obtains facial image data through a camera, extracts features in the facial region, and then classifies or regresses to obtain emotion results; speech-based emotion recognition extracts acoustic features from speech signals and outputs emotion categories or emotion dimension values combined with a classifier; text-based emotion recognition usually obtains emotion tendency judgment by tokenizing, encoding and inputting text into a language model. This kind of single-modal method has relatively simple implementation structure, but is easily affected by external interference in real applications: the visual modality may cause unstable expression features due to changes in lighting, occlusion, posture deflection or insufficient resolution; the speech modality may cause acoustic feature fluctuations due to environmental noise, speaker differences, speech rate changes or emotional expression restraint; the text modality may cause semantic feature deviation due to speech recognition errors, colloquial omissions, irony expressions or context missing. Therefore, single-modal methods often have unstable recognition results, weak cross-person generalization ability, and high misjudgment rate in complex scenarios, which makes it difficult to meet the needs of high-reliability interaction systems for emotion perception. SUMMARY

[0004] One purpose of the present application is to provide an AI vision-based multi-modal emotion recognition method, which fully utilizes computer vision technology, speech signal processing technology and natural language processing technology, and realizes dynamic recognition and time sequence tracking of user emotional state through multi-modal feature extraction and fusion modeling of user facial image data, speech data and text data. The present application introduces a time sequence alignment mechanism of multi-modal features and a majority voting-based fusion strategy, which effectively reduces the influence of single modal anomaly on the recognition result while ensuring the realizability of the system, and combines the improved VA emotion model to output continuous emotion dimension values, so that the emotion recognition result is more stable, robust and has good continuity, and has the advantages of adapting to complex interactive scenarios, flexible deployment and strong emotion state tracking capability.

[0005] An AI vision-based multi-modal emotion recognition method according to an embodiment of the present application comprises the following steps: Obtain facial image data, speech data and text data of a user and preprocess them; Extract features from the preprocessed facial image data using the HOG algorithm to obtain visual emotion features; Extract features from the preprocessed speech data using the TEO method to obtain speech emotion features; Extract features from the preprocessed text data based on a pre-trained language model to obtain text emotion features; Time sequence align the visual emotion features, speech emotion features and text emotion features to obtain a time sequence aligned multi-modal feature sequence; Fuse the time sequence aligned multi-modal feature sequence using a majority voting mechanism to obtain fused multi-modal emotion features; Input the fused multi-modal emotion features into an improved VA emotion model to output an emotion recognition result; Track the emotional state based on the valence dimension value and the arousal dimension value of the emotion recognition result to obtain an emotional state change sequence.

[0006] Optionally, the acquisition process is to collect facial image data of a user through a visual sensor, collect speech data of a user through an audio sensor, and obtain text data by performing speech recognition on the speech data. The preprocessing includes performing face region positioning and alignment processing on the facial image data, performing noise reduction and framing processing on the speech data, and performing word segmentation and encoding processing on the text data.

[0007] Optionally, the visual emotion features are obtained specifically by: Converting the preprocessed facial image data into a grayscale image; Performing gradient calculation on the grayscale image to obtain horizontal direction gradient and vertical direction gradient; The gradient amplitude and the gradient direction are calculated based on the horizontal direction gradient and the vertical direction gradient; The HOG algorithm is used to divide the gray image into unit intervals, the gradient direction is quantized according to a preset direction interval in each unit, and the direction histogram of the unit is counted to obtain the direction histogram of the unit; Adjacent units are combined into a block, the direction histograms in each block are spliced to obtain a block vector, and the block vector is normalized to obtain a normalized block vector; All normalized block vectors are spliced to obtain a direction gradient histogram feature vector, and the direction gradient histogram feature vector is used as a visual emotion feature.

[0008] Optionally, the obtaining of the speech emotion feature specifically includes: The preprocessed speech data is subjected to frame processing to obtain a speech frame sequence arranged in time sequence; The Teager energy operator (TEO) is used to calculate each speech frame in the speech frame sequence to obtain a Teager energy sequence; Statistical quantity calculation is performed on the Teager energy sequence to obtain a Teager energy feature, the Teager energy feature including the mean value of the Teager energy sequence in the corresponding speech frame and the variance of the Teager energy sequence in the corresponding speech frame; The fundamental frequency, energy, pitch and speech rate of each speech frame in the speech frame sequence are extracted; The mel-frequency cepstral coefficient of each speech frame in the speech frame sequence is extracted; The Teager energy feature, fundamental frequency, energy, pitch, speech rate and mel-frequency cepstral coefficient are spliced to obtain a speech frame-level speech emotion feature vector; The speech frame-level speech emotion feature vectors corresponding to all speech frames are combined in time sequence to obtain a speech emotion feature.

[0009] Optionally, the obtaining of the text emotion feature specifically includes: The preprocessed text data is cut by characters to obtain a text unit sequence arranged in sequence; The text unit sequence is input into an embedding layer of a pre-trained language model to obtain a text embedding vector sequence corresponding to the text unit sequence in one-to-one correspondence, and the text embedding vector sequence is sequentially composed into a text embedding matrix; The text embedding matrix is subjected to position coding superposition processing to obtain a position-enhanced embedding matrix; The position-enhanced embedding matrix is input into an encoding network of the pre-trained language model to obtain a text context representation matrix; The text context representation matrix is subjected to pooling processing to obtain a text global representation vector; Linearly mapping the text global representation vector to obtain a text sentiment feature.

[0010] Optionally, obtaining the time sequence alignment multi-modal feature sequence specifically includes: timestamp sequences are established for the visual sentiment feature, the speech sentiment feature and the text sentiment feature respectively; alignment start time and alignment end time are obtained based on the timestamp sequences, and an alignment time axis is generated at fixed time intervals between the alignment start time and the alignment end time; the visual sentiment feature sequence is formed in the time sequence of the timestamp sequence, the speech sentiment feature sequence is formed in the time sequence of the timestamp sequence, and the text sentiment feature sequence is formed in the time sequence of the timestamp sequence; for each alignment time in the alignment time axis, linear interpolation processing is performed on the visual sentiment feature sequence based on the timestamp sequence to obtain an alignment visual sentiment feature; for each alignment time in the alignment time axis, linear interpolation processing is performed on the speech sentiment feature sequence based on the timestamp sequence to obtain an alignment speech sentiment feature; for each alignment time in the alignment time axis, nearest neighbor selection processing is performed on the text sentiment feature sequence based on the timestamp sequence to obtain an alignment text sentiment feature; the alignment visual sentiment feature, the alignment speech sentiment feature and the alignment text sentiment feature corresponding to the same alignment time are spliced to obtain the time sequence alignment multi-modal feature sequence.

[0011] Optionally, obtaining the fusion multi-modal sentiment feature specifically includes: for each alignment time in the time sequence alignment multi-modal feature sequence, a majority voting mechanism is used to generate a visual candidate voting result, a speech candidate voting result and a text candidate voting result based on the visual sentiment feature, the speech sentiment feature and the text sentiment feature corresponding to the alignment time respectively; a visual voting weight, a speech voting weight and a text voting weight are set for the visual candidate voting result, the speech candidate voting result and the text candidate voting result, and the visual candidate voting result, the speech candidate voting result and the text candidate voting result are weighted and counted to obtain a vote value corresponding to each candidate voting result, the vote value of each candidate voting result being the cumulative value of the visual voting weight, the speech voting weight and the text voting weight consistent with the candidate voting result; the candidate voting result with the largest vote value is selected as the voting fusion result of the alignment time, and the voting fusion result is taken as the fusion multi-modal sentiment feature corresponding to the alignment time.

[0012] Optionally, the output of the sentiment recognition result specifically includes: input the fused multi-modal emotion feature into an improved VA emotion model, the improved VA emotion model comprising an input representation mapping module, a valence dimension regression module, an arousal dimension regression module and a temporal consistency constraint module, the input representation mapping module being configured to perform representation mapping processing on the fused multi-modal emotion feature to obtain a unified input representation vector, the valence dimension regression module being configured to perform quantile regression calculation and truncation mapping on the unified input representation vector to obtain a valence dimension value, the arousal dimension regression module being configured to perform quantile regression calculation and truncation mapping on the unified input representation vector to obtain an arousal dimension value, and the temporal consistency constraint module being configured to perform smoothing processing on the valence dimension value and the arousal dimension value based on a total variation constraint; In the input representation mapping module, the fused multi-modal emotion feature is subjected to representation mapping processing, the representation mapping processing being linear mapping of the fused multi-modal emotion feature after component normalization processing to obtain an intermediate representation, and the intermediate representation is fused with the fused multi-modal emotion feature based on a gated residual connection mechanism to obtain a unified input representation vector corresponding to each alignment time; In the valence dimension regression module, quantile regression calculation is performed on the unified input representation vector to obtain a median estimate of the valence dimension value, and truncation mapping is performed on the median estimate of the valence dimension value to obtain the valence dimension value; In the arousal dimension regression module, quantile regression calculation is performed on the unified input representation vector to obtain a median estimate of the arousal dimension value, and truncation mapping is performed on the median estimate of the arousal dimension value to obtain the arousal dimension value; In the temporal consistency constraint module, the valence dimension value and the arousal dimension value are subjected to temporal consistency processing, the temporal consistency processing being smoothing processing on the valence dimension value based on a total variation constraint to obtain a temporally consistent valence dimension value, and smoothing processing on the arousal dimension value based on a total variation constraint to obtain a temporally consistent arousal dimension value, wherein the total variation constraint limits the sum of absolute values of differences between values corresponding to adjacent alignment times; The temporally consistent valence dimension value and the temporally consistent arousal dimension value are taken as emotion recognition results and output.

[0013] Optionally, the obtaining of the sequence of emotion state changes specifically comprises: The valence dimension values and the arousal dimension values of the emotion recognition results are arranged in time sequence to form a sequence of valence dimension values and a sequence of arousal dimension values; The valence dimension value and the arousal dimension value corresponding to the first alignment time are taken as an initial emotion state; For each alignment moment except the first alignment moment, a current alignment moment corresponding emotional state is generated, the valence dimension value of the current alignment moment and the valence dimension value in the emotional state of the previous alignment moment are proportionally weighted to form the valence dimension value of the current alignment moment, and the arousal dimension value of the current alignment moment and the arousal dimension value in the emotional state of the previous alignment moment are proportionally weighted to form the arousal dimension value of the current alignment moment; For each alignment moment except the first alignment moment, the emotional state change amount of adjacent alignment moments is calculated, the valence dimension change amount of the emotional state change amount is the valence dimension value in the emotional state of the current alignment moment minus the valence dimension value in the emotional state of the previous alignment moment, and the arousal dimension change amount of the emotional state change amount is the arousal dimension value in the emotional state of the current alignment moment minus the arousal dimension value in the emotional state of the previous alignment moment. The emotional state corresponding to each alignment moment and the emotional state change amount corresponding to each alignment moment are combined in time sequence to obtain an emotional state change sequence.

[0014] The present application has the following advantages: The present application effectively solves the problems of insufficient robustness of single modal emotion recognition, inconsistent multi-modal fusion timing and poor stability of continuous emotion output in the prior art by constructing an AI vision-based multi-modal emotion recognition method. By simultaneously introducing facial image, speech and text three modal information, the present application realizes the collaborative use of multi-source information at the emotional perception level, so that when the information quality of a certain modal is reduced due to environmental interference, shielding or noise influence, the remaining modal can still provide effective supplement for emotion recognition, which significantly improves the reliability and adaptability of emotion recognition results in complex real scenes as a whole.

[0015] At the feature extraction level, the present application respectively adopts an engineering feasible feature extraction method for the signal characteristics of different modalities, among which the visual side extracts facial local texture and structural change features through a direction gradient histogram algorithm, the speech side enhances the perception ability of emotional intensity change by introducing a non-linear energy analysis-based feature extraction method, and the text side obtains semantic layer emotional information by combining a pre-trained language model. The above multi-modal features realize effective representation of key information of emotional expression under the premise of controllable computing resources, avoid the problems of high deployment cost and poor real-time performance caused by excessive dependence on large-scale deep visual models in the prior art, and thus improve the practical value of the method in edge devices and real-time interactive systems.

[0016] In terms of multi-modal collaborative processing, the application unifies visual emotion features, speech emotion features and text emotion features with different sampling frequencies and generation times onto the same time axis by introducing a time alignment mechanism, effectively alleviating the common modal misplacement and information asynchronization problems in existing multi-modal emotion recognition methods. On this basis, by using a majority voting mechanism to fuse the time-aligned multi-modal features, the application can reduce the interference of single modal abnormality or short-time failure on the fusion result, making the fused multi-modal emotion features more stable and reliable, thereby improving the consistency of the overall emotion recognition result.

[0017] In addition, the application inputs the fused multi-modal emotion features into an improved VA emotion model, and through continuous output and time sequence constraint processing of the valence dimension value and the arousal dimension value, realizes smooth modeling and dynamic tracking of the user's emotional state. Compared with the discrete emotion classification or continuous output mode without time sequence constraint in the prior art, the application can more accurately reflect the trend of emotion change over time, reduce the jitter and mutation of the output result, and provide a continuous, stable and directly usable emotion state change sequence for subsequent human-computer interaction strategy adjustment, behavior feedback control and emotion perception application. In summary, the application has achieved significant beneficial effects in terms of emotion recognition accuracy, robustness, time sequence consistency and engineering deployability. BRIEF DESCRIPTION OF DRAWINGS

[0018] The accompanying drawings are included to provide a further understanding of the application, and constitute a part of the specification, illustrate the application together with the embodiments thereof, and explain the application, but do not constitute a limitation on the application. In the drawings: Fig. 1 A flowchart of a multi-modal emotion recognition method based on AI vision according to the application; Fig. 2 A construction schematic diagram of fused multi-modal emotion features of a multi-modal emotion recognition method based on AI vision according to the application; Fig. 3 A construction schematic diagram of an improved VA emotion model of a multi-modal emotion recognition method based on AI vision according to the application. DETAILED DESCRIPTION

[0019] The application will now be described in further detail with reference to the drawings. These drawings are simplified schematic diagrams, and only illustrate the basic structure of the application in a schematic manner, and therefore only show the components related to the application.

[0020] Reference Figs. 1-3 A multi-modal emotion recognition method based on AI vision, comprising the following steps: Obtaining facial image data, speech data and text data of a user and preprocessing; The HOG algorithm is used to extract features from the preprocessed facial image data to obtain visual emotional features. The TEO method is used to extract features from the preprocessed speech data to obtain speech emotional features. The pre-trained language model is used to extract features from the preprocessed text data to obtain text emotional features. The visual emotional features, speech emotional features, and text emotional features are time-aligned to obtain a time-aligned multi-modal feature sequence. The time-aligned multi-modal feature sequence is fused using a majority voting mechanism to obtain fused multi-modal emotional features. The fused multi-modal emotional features are input into the improved VA emotion model to output an emotional recognition result. Based on the valence dimension value and the arousal dimension value of the emotional recognition result, the emotional state tracking is performed to obtain an emotional state change sequence.

[0021] In this embodiment, the acquisition process includes collecting facial image data of a user through a visual sensor, collecting speech data of the user through an audio sensor, and performing speech recognition on the speech data to obtain text data. The preprocessing includes performing face region positioning and alignment processing on the facial image data, performing noise reduction and framing processing on the speech data, and performing word segmentation and encoding processing on the text data.

[0022] In this embodiment, the visual emotional features are obtained specifically as follows: The preprocessed facial image data is converted into a grayscale image. The conversion is specifically implemented as follows: the color image in the preprocessed facial image data is subjected to grayscale processing by pixel, the red channel component, the green channel component, and the blue channel component are read for each pixel point, the red channel component, the green channel component, and the blue channel component are weighted and summed according to a preset grayscale weight, the weighted sum result is taken as the grayscale value of the corresponding pixel point, and a grayscale image is obtained. When the preprocessed facial image data is a single-channel image, the single-channel image is taken as the grayscale image. The gradient of the grayscale image is calculated to obtain the horizontal direction gradient and the vertical direction gradient. The horizontal direction gradient is the grayscale difference between two adjacent points in the horizontal direction, and the vertical direction gradient is the grayscale difference between two adjacent points in the vertical direction. The gradient amplitude and the gradient direction are calculated based on the horizontal direction gradient and the vertical direction gradient. The gradient amplitude is the square root of the sum of the square of the horizontal direction gradient and the square of the vertical direction gradient, and the gradient direction is the inverse tangent value of the ratio of the vertical direction gradient to the horizontal direction gradient. The HOG algorithm is used to divide the gray image into unit intervals, the gradient direction is quantized according to a preset direction interval in each unit, and a direction histogram of the unit is obtained by direction histogram statistics, wherein an accumulated value of each direction interval in the direction histogram is obtained by weighted accumulation of the gradient amplitude of the pixels falling into the corresponding direction interval; Adjacent units are combined into blocks, the direction histograms in each block are spliced to obtain a block vector, and the block vector is normalized to obtain a normalized block vector, wherein the normalization is to divide each component of the block vector by a normalization factor, and the normalization factor is a square root of a sum of squares of a two-norm of the block vector and a square of a preset constant; All normalized block vectors are spliced to obtain a direction gradient histogram feature vector, and the direction gradient histogram feature vector is used as a visual emotion feature.

[0023] In the embodiment, the obtaining of the speech emotion feature specifically includes: The preprocessed speech data is subjected to frame processing to obtain a sequence of speech frames arranged in time sequence; The frame processing specifically includes: the preprocessed speech data is subjected to sampling point serialization, the preprocessed speech data is divided into continuous speech frames in time sequence, each speech frame contains the same number of continuous sampling points; the starting sampling point of the first speech frame is used as a frame starting point, and a frame window is sequentially moved from the starting position of the preprocessed speech data to generate subsequent speech frames, the fixed step length is smaller than the number of sampling points contained in each speech frame, so that there are overlapping sampling points between adjacent two speech frames; each speech frame is subjected to windowing processing, the windowing processing is to multiply each sampling point in the speech frame by a window function coefficient to obtain a windowed speech frame; all windowed speech frames are arranged in generation order to obtain a sequence of speech frames arranged in time sequence; The Teager energy operator (TEO) is used to calculate each speech frame in the sequence of speech frames to obtain a Teager energy sequence, wherein a Teager energy value corresponding to each sampling point is obtained by subtracting the product of the sampling value of the previous sampling point and the sampling value of the next sampling point from the square of the sampling value of the sampling point; The Teager energy sequence is subjected to statistical quantity calculation to obtain a Teager energy feature, wherein the Teager energy feature includes a mean value of the Teager energy sequence in the corresponding speech frame and a variance of the Teager energy sequence in the corresponding speech frame; The fundamental frequency, energy, pitch and speech rate of each speech frame in the sequence of speech frames are extracted; The process of the extraction is: short-time analysis is performed on each speech frame in the sequence of speech frames to obtain a fundamental frequency, energy, pitch and speech rate, the fundamental frequency is obtained through autocorrelation analysis, specifically, an autocorrelation sequence is calculated for the speech frame, a delay corresponding to a peak with the largest amplitude except for zero delay in the autocorrelation sequence is selected as a pitch period, and a ratio of a sampling frequency to the pitch period is taken as the fundamental frequency; the energy is obtained through short-time energy calculation, specifically, the amplitudes of the sampling points in the speech frame are squared and summed to obtain the short-time energy of the speech frame; the pitch is obtained through mapping from the fundamental frequency, specifically, the fundamental frequency is converted into a logarithmic pitch scale to obtain a pitch value; and the speech rate is obtained through speech activity segment statistics, specifically, threshold judgment is performed on the short-time energy of the continuous speech frames to distinguish voiced segments and unvoiced segments, the number of syllable peaks formed by energy fluctuations in the voiced segments is counted, and a ratio of the number of syllable peaks to the corresponding time length is taken as the speech rate; Mel-frequency cepstral coefficients are extracted from each speech frame in the sequence of speech frames; The process of the extraction is: spectrum analysis, Mel filtering, logarithmic compression and cepstrum transformation are sequentially performed on each speech frame in the sequence of speech frames to obtain Mel-frequency cepstral coefficients, specifically, the speech frame is windowed and then subjected to fast Fourier transform to obtain a spectrum amplitude; the spectrum amplitude is input into a Mel filter bank for filtering processing, the Mel filter bank is composed of triangular bandpass filters, each triangular bandpass filter is distributed on a frequency axis according to a Mel frequency scale, and the spectrum amplitudes in a frequency band corresponding to each triangular bandpass filter are weighted and accumulated to obtain a filter bank output; logarithmic operation is performed on the filter bank output to obtain a logarithmic Mel energy; discrete cosine transform is performed on the logarithmic Mel energy to obtain a cepstral coefficient set, and the cepstral coefficient set is taken as the Mel-frequency cepstral coefficients; The Teager energy feature, the fundamental frequency, the energy, the pitch, the speech rate and the Mel-frequency cepstral coefficients are spliced to obtain a speech frame-level speech emotion feature vector; The speech frame-level speech emotion feature vectors corresponding to all the speech frames are combined in time sequence to obtain a speech emotion feature.

[0024] In this embodiment, the text emotion feature is obtained specifically as follows: The preprocessed text data is divided by characters to obtain a sequence of text units arranged in order; The sequence of text units is input into an embedding layer of a pre-trained language model to obtain a sequence of text embedding vectors corresponding to the sequence of text units one by one, and the sequence of text embedding vectors is sequentially composed into a text embedding matrix; Position encoding superposition processing is performed on the text embedding matrix to obtain a position-enhanced embedding matrix, the position encoding superposition processing is to add a text embedding vector corresponding to each text unit to a position encoding vector corresponding to the position of the text unit; inputting the position-enhanced embedding matrix into an encoding network of a pre-trained language model to obtain a text context representation matrix; The encoding network comprises a multi-layer encoding structure, and each layer of the encoding structure performs self-attention calculation, which comprises: performing linear transformation on the input matrix to obtain a query matrix, a key matrix and a value matrix, calculating an attention weight matrix based on the query matrix and the key matrix, obtaining the attention weight matrix by scaling and normalizing the product of the query matrix and the key matrix, performing weighted summation on the value matrix by using the attention weight matrix to obtain an attention output matrix, and inputting the attention output matrix into a feedforward network to obtain an output matrix, thereby forming the text context representation matrix; performing pooling processing on the text context representation matrix to obtain a text global representation vector, the pooling processing being to take a vector corresponding to a first position in the text context representation matrix as the text global representation vector, or to perform mean value calculation on vectors corresponding to all positions in the text context representation matrix as the text global representation vector; performing linear mapping on the text global representation vector to obtain a text sentiment feature, the linear mapping being to perform matrix multiplication on the text global representation vector and superimpose a bias vector to obtain the text sentiment feature.

[0025] In the embodiment, obtaining the time-aligned multi-modal feature sequence specifically comprises: establishing a timestamp sequence for the visual sentiment feature, the speech sentiment feature and the text sentiment feature respectively; The establishment process specifically comprises: a manner of establishing a visual timestamp sequence for the visual sentiment feature is to record a capture time of each image frame as a timestamp of image frame corresponding visual input data when capturing the facial image data, and to associate the visual sentiment feature generated by the image frame with the timestamp to obtain the visual timestamp sequence in chronological order; a manner of establishing a speech timestamp sequence for the speech sentiment feature is to record a speech sampling start time in the speech data capturing process, and to determine a start time or a center time of a speech frame as a timestamp of speech frame corresponding speech input data by using the speech sampling start time and the position of the speech frame in the speech data when the speech data is framed, and to associate the speech sentiment feature generated by the speech frame with the timestamp to obtain the speech timestamp sequence in chronological order; a manner of establishing a text timestamp sequence for the text sentiment feature is to record a generation time of each text segment when the speech data is recognized to obtain the text data, and to associate the text sentiment feature generated by the text segment with the generation time to obtain the text timestamp sequence in chronological order; obtaining an alignment start time and an alignment end time based on the timestamp sequence, and generating an alignment time axis at a fixed time interval between the alignment start time and the alignment end time; form a visual emotion feature sequence in time sequence of the timestamp sequence, form a speech emotion feature sequence in time sequence of the timestamp sequence, and form a text emotion feature sequence in time sequence of the timestamp sequence; For each alignment time in the alignment time axis, linear interpolation processing is performed on the visual emotion feature sequence based on the timestamp sequence to obtain an aligned visual emotion feature. The linear interpolation processing specifically includes the following steps: for each alignment time in the alignment time axis, reading two adjacent timestamps of the alignment time in the visual timestamp sequence, the two adjacent timestamps being a previous visual timestamp less than the alignment time and a next visual timestamp greater than the alignment time respectively; obtaining a visual emotion feature corresponding to the previous visual timestamp and a visual emotion feature corresponding to the next visual timestamp; calculating a time proportion factor of the alignment time relative to the previous visual timestamp and the next visual timestamp, the time proportion factor being a time difference between the alignment time and the previous visual timestamp divided by a time difference between the next visual timestamp and the previous visual timestamp; and performing linear weighted summation on the visual emotion feature corresponding to the previous visual timestamp and the visual emotion feature corresponding to the next visual timestamp based on the time proportion factor to obtain an aligned visual emotion feature corresponding to the alignment time. For each alignment time in the alignment time axis, linear interpolation processing is performed on the speech emotion feature sequence based on the timestamp sequence to obtain an aligned speech emotion feature. The linear interpolation processing specifically includes the following steps: for each alignment time in the alignment time axis, reading two adjacent timestamps of the alignment time in the visual timestamp sequence, the two adjacent timestamps being a previous visual timestamp less than the alignment time and a next visual timestamp greater than the alignment time respectively; obtaining a visual emotion feature corresponding to the previous visual timestamp and a visual emotion feature corresponding to the next visual timestamp; calculating a time proportion factor of the alignment time relative to the previous visual timestamp and the next visual timestamp, the time proportion factor being a time difference between the alignment time and the previous visual timestamp divided by a time difference between the next visual timestamp and the previous visual timestamp; and performing linear weighted summation on the visual emotion feature corresponding to the previous visual timestamp and the visual emotion feature corresponding to the next visual timestamp based on the time proportion factor to obtain an aligned visual emotion feature corresponding to the alignment time. For each alignment time in the alignment time axis, nearest neighbor selection processing is performed on the text emotion feature sequence based on the timestamp sequence to obtain an aligned text emotion feature. The nearest neighbor selection process specifically comprises: for each alignment time in the alignment time axis, calculating absolute values of time differences between the alignment time and each text timestamp in the text timestamp sequence to obtain a time difference sequence; determining a text timestamp corresponding to a time difference with the smallest absolute value in the time difference sequence as a target text timestamp matched with the alignment time; obtaining a text sentiment feature corresponding to the target text timestamp, and determining the text sentiment feature as an alignment text sentiment feature corresponding to the alignment time; The alignment visual sentiment feature, the alignment speech sentiment feature and the alignment text sentiment feature corresponding to the same alignment time are spliced to obtain a time sequence alignment multi-modal feature sequence.

[0026] In this embodiment, the obtaining of the fused multi-modal sentiment feature specifically comprises: For each alignment time in the time sequence alignment multi-modal feature sequence, a majority voting mechanism is used to generate a visual candidate voting result, a speech candidate voting result and a text candidate voting result based on the visual sentiment feature, the speech sentiment feature and the text sentiment feature corresponding to the alignment time, respectively. The generation process is: based on the visual sentiment feature, the speech sentiment feature and the text sentiment feature, a visual candidate set, a speech candidate set and a text candidate set are established respectively, and each of the visual candidate set, the speech candidate set and the text candidate set is composed of candidate sentiment identifiers; the visual sentiment feature and each candidate sentiment identifier in the visual candidate set are sequentially matched and calculated, and the candidate sentiment identifier with the largest matching degree is selected as the visual candidate voting result; the speech sentiment feature and each candidate sentiment identifier in the speech candidate set are sequentially matched and calculated, and the candidate sentiment identifier with the largest matching degree is selected as the speech candidate voting result; the text sentiment feature and each candidate sentiment identifier in the text candidate set are sequentially matched and calculated, and the candidate sentiment identifier with the largest matching degree is selected as the text candidate voting result; the visual candidate voting result, the speech candidate voting result and the text candidate voting result are combined to form a voting set, the voting set is counted, and the candidate sentiment identifier with the most votes is determined as the majority voting result of the alignment time; when the candidate sentiment identifier with the most votes is not unique, the majority voting result of the alignment time is determined in the order of the visual candidate voting result first, the speech candidate voting result second and the text candidate voting result third; The visual voting weight, the speech voting weight and the text voting weight are set for the visual candidate voting result, the speech candidate voting result and the text candidate voting result respectively, the visual candidate voting result, the speech candidate voting result and the text candidate voting result are weighted and counted to obtain a vote value corresponding to each candidate voting result, and the vote value of each candidate voting result is the cumulative value of the visual voting weight, the speech voting weight and the text voting weight consistent with the candidate voting result. select the candidate voting result with the largest vote value as the voting fusion result of the alignment moment, and take the voting fusion result as the fusion multi-modal sentiment feature corresponding to the alignment moment.

[0027] In the embodiment, the output of the sentiment recognition result specifically includes: input the fusion multi-modal sentiment feature into an improved VA sentiment model, the improved VA sentiment model includes an input representation mapping module, a valence dimension regression module, an arousal dimension regression module, and a temporal consistency constraint module, the input representation mapping module refers to performing representation mapping processing on the fusion multi-modal sentiment feature to obtain a unified input representation vector, the valence dimension regression module refers to performing quantile regression calculation and truncation mapping on the unified input representation vector to obtain a valence dimension value, the arousal dimension regression module refers to performing quantile regression calculation and truncation mapping on the unified input representation vector to obtain an arousal dimension value, and the temporal consistency constraint module refers to performing smoothing processing on the valence dimension value and the arousal dimension value based on a total variation constraint; In the input representation mapping module, the fusion multi-modal sentiment feature is subjected to representation mapping processing, the representation mapping processing is linear mapping after component normalization processing of the fusion multi-modal sentiment feature to obtain an intermediate representation, and the intermediate representation is fused with the fusion multi-modal sentiment feature based on a gated residual connection mechanism to obtain a unified input representation vector corresponding to each alignment moment; In the valence dimension regression module, quantile regression calculation is performed on the unified input representation vector to obtain a median estimate of the valence dimension value, and truncation mapping is performed on the median estimate of the valence dimension value to obtain the valence dimension value; The quantile regression calculation is based on the unified input representation vector to obtain the median estimate of the valence dimension value through regression mapping, the regression mapping is linear transformation of the unified input representation vector and superposition of a bias to obtain a regression output; the regression output is subjected to parameter update by using a quantile regression loss, the quantile regression loss is the absolute value accumulation of the difference between the target valence dimension value of each training sample and the regression output by piecewise weighting, and when the difference is positive, the weight corresponding to the median quantile is accumulated, and when the difference is negative, another weight is accumulated; The truncation mapping processing is: when the median estimate of the valence dimension value is less than the valence dimension minimum value, the valence dimension value is determined as the valence dimension minimum value, when the median estimate of the valence dimension value is greater than the valence dimension maximum value, the valence dimension value is determined as the valence dimension maximum value, otherwise the valence dimension value is determined as the median estimate of the valence dimension value, to obtain the valence dimension value; In the arousal dimension regression module, quantile regression calculation is performed on the unified input representation vector to obtain a median estimate of the arousal dimension value, and truncation mapping is performed on the median estimate of the arousal dimension value to obtain the arousal dimension value. The quantile regression calculation refers to obtaining a median estimate of the arousal dimension value based on a unified input feature vector through a regression mapping, the regression mapping being a linear transformation of the unified input feature vector and superimposing a bias to obtain a regression output; and a quantile regression loss is used to update parameters of the regression output, the quantile regression loss being an absolute value accumulation of a segmented weighted difference between a target arousal dimension value of each training sample and the regression output, and when the difference is positive, a weight corresponding to a median quantile is accumulated, and when the difference is negative, another weight is accumulated; The truncation mapping processing is: when the median estimate of the arousal dimension value is less than the arousal dimension minimum value, the arousal dimension value is determined as the arousal dimension minimum value, when the median estimate of the arousal dimension value is greater than the arousal dimension maximum value, the arousal dimension value is determined as the arousal dimension maximum value, otherwise the arousal dimension value is determined as the median estimate of the arousal dimension value, to obtain the arousal dimension value; In the time sequence consistency constraint module, the valence dimension value and the arousal dimension value are subjected to time sequence consistency processing, the time sequence consistency processing being that the valence dimension value is subjected to smoothing processing based on a total variation constraint to obtain a time sequence consistent valence dimension value, and the arousal dimension value is subjected to smoothing processing based on a total variation constraint to obtain a time sequence consistent arousal dimension value, wherein the total variation constraint limits the sum of absolute values of differences between values corresponding to adjacent alignment moments; The time sequence consistent valence dimension value and the time sequence consistent arousal dimension value are taken as the emotion recognition result and output.

[0028] In the embodiment, the obtaining of the sequence of emotion state changes specifically includes: The valence dimension value and the arousal dimension value of the emotion recognition result are arranged in time sequence to form a valence dimension value sequence and an arousal dimension value sequence; The valence dimension value and the arousal dimension value corresponding to the first alignment moment are taken as an initial emotion state; For each alignment moment except the first alignment moment, an emotion state corresponding to the current alignment moment is generated, the emotion state being composed of a valence dimension value obtained by proportionally weighting the valence dimension value of the current alignment moment and the valence dimension value in the emotion state of the previous alignment moment, and being composed of an arousal dimension value obtained by proportionally weighting the arousal dimension value of the current alignment moment and the arousal dimension value in the emotion state of the previous alignment moment; For each alignment moment except the first alignment moment, a change amount of emotion state of adjacent alignment moments is calculated, a valence dimension change amount of the change amount of emotion state being the valence dimension value in the emotion state of the current alignment moment minus the valence dimension value in the emotion state of the previous alignment moment, and an arousal dimension change amount of the change amount of emotion state being the arousal dimension value in the emotion state of the current alignment moment minus the arousal dimension value in the emotion state of the previous alignment moment. The emotional state corresponding to each alignment time and the emotional state change amount corresponding to each alignment time are combined in time sequence to obtain an emotional state change sequence.

[0029] In the emotional interaction application of an intelligent service robot in a public service hall, the public service hall is a government service hall, the number of daily reception personnel is large, the service object age span is large, the emotional state difference is obvious, and the on-site environment is complex, there are crowd noise, light changes, and user expression methods are not unified, etc. The intelligent service robot is deployed in the hall entrance area and is mainly used to provide consultation guidance, process explanation and simple question answering for users coming to handle business. It is found in the actual operation process that the traditional interaction mode relying only on voice recognition or text semantic analysis is difficult to identify the real emotional state of the user in a timely and accurate manner, especially when the user is nervous, anxious or expresses unclearly, the system is easy to make inappropriate feedback, leading to a decline in interaction experience, and even causing user dissatisfaction.

[0030] In this scenario, the AI vision-based multi-modal emotion recognition method described in the application is integrated into the perception and decision-making module of the service robot. During the interaction with the user, the robot continuously collects the user's facial image data through the front camera, collects the user's voice data through the microphone array, and performs real-time voice recognition on the voice data to obtain the corresponding text data. After the data collection is completed, the facial image data is subjected to basic image preprocessing, including face region positioning, scale normalization and grayscale processing; the voice data is subjected to noise reduction, framing and windowing processing; the text data is subjected to word segmentation and normalization processing, providing stable input for subsequent feature extraction.

[0031] In the feature extraction stage, the system uses the direction gradient histogram algorithm to extract features from the preprocessed facial image data, obtaining visual emotional features that can reflect the local texture changes and expression structure differences of the face, thereby maintaining good stability in the case of light changes and slight posture deflection. The speech side uses a feature extraction method based on the Teager energy operator to model the non-linear energy changes of the voice signal, while combining the fundamental frequency, energy, speech rate and other conventional acoustic information to enhance the sensitivity of the system to emotional intensity changes. The text side extracts features from the text data through a pre-trained language model to obtain semantic-level sentiment information. The above three modal features are independent of each other in the extraction process, avoiding the direct impact of missing or abnormal single modal features on the overall system.

[0032] Due to the natural differences in the generation time and sampling frequency of visual frames, speech frames and text segments, the embodiment further performs time alignment processing on visual emotional features, speech emotional features and text emotional features, maps the three modal features to a unified time axis, and ensures that the information fused at the same alignment time is derived from the user emotional expression in the same time period. After completing the time alignment, the system uses a majority voting mechanism to fuse the time-aligned multi-modal feature sequence, determines the fused multi-modal emotional feature through the voting consistency of the candidate emotional results of each modality, thereby reducing the influence of single modal short-time abnormality or noise interference on the fusion result.

[0033] The fused multi-modal emotional feature is input into the improved VA emotional model, the model outputs continuous valence dimension values and arousal dimension values respectively, and the outputs at continuous time are smoothed and constrained through time consistency processing, and finally a sequence of emotional state changes is formed. In the actual operation of the robot, when the system identifies that the arousal dimension value of the user is continuously rising and the valence dimension value is significantly decreasing, the robot will automatically adjust the interaction strategy, for example, reducing the speech speed, using more soothing language or actively guiding the user to the manual window, thereby effectively relieving the user's emotion and improving the service experience.

[0034] In order to verify the beneficial effects of the method of the application in this scene, three different emotional recognition schemes were compared and tested in the actual operation for two weeks: a method using only the speech modality, a traditional multi-modal method without time alignment, and a multi-modal emotional recognition scheme using the method of the application. A total of 1260 valid interaction records were collected, including 438 groups of scenes with obvious user emotional fluctuations. By manually labeling the emotional trend as a reference, the emotional recognition accuracy, emotional state continuity and abnormal scene robustness of each method were statistically analyzed, and the results are shown in the following table.

[0035] Table 1 Performance comparison results of different emotional recognition methods in the public service hall scene

[0036] As can be seen from Table 1, in terms of overall emotion recognition accuracy, the method of the present application improves about 16.9 percentage points compared with the single speech modal method, and improves about 7.7 percentage points compared with the traditional multi-modal method, indicating that the multi-modal feature extraction combined with the time alignment and the majority voting fusion strategy can significantly improve the reliability of emotion recognition. In a high-noise environment, the method of the present application still maintains a high accuracy, verifying the effectiveness of the introduction of visual emotion features and nonlinear acoustic features in resisting environmental interference. In terms of emotion state continuity score, the method of the present application is significantly better than the comparative methods, indicating that the output valence dimension value and arousal dimension value change more smoothly over time, which is conducive to subsequent emotion state tracking and interaction strategy adjustment. In addition, in the single modal abnormal scene, the misjudgment rate of the method of the present application is significantly reduced, which reflects the advantage of the majority voting fusion mechanism in suppressing abnormal modal. In summary, the above experimental results can prove that the present application can effectively solve the problems of unstable emotion recognition, insufficient robustness and poor continuity in the prior art in complex real scenes, and has significant practical application value.

[0037] The above description is merely preferred specific embodiments of the present application, but the protection scope of the present application is not limited thereto, and any person skilled in the art can make equivalent replacements or changes to the technical solutions and inventive concepts of the present application within the technical scope disclosed by the present application, which should be covered within the protection scope of the present application.

Claims

1. A multimodal emotion recognition method based on AI vision, characterized in that, Includes the following steps: Acquire and preprocess the user's facial image data, voice data, and text data; The HOG algorithm was used to extract features from the preprocessed facial image data to obtain visual emotion features. The TEO method is used to extract features from the preprocessed speech data to obtain speech emotion features; Based on a pre-trained language model, feature extraction is performed on the pre-processed text data to obtain text sentiment features; Temporal alignment of visual sentiment features, speech sentiment features, and text sentiment features yields a temporally aligned multimodal feature sequence. A majority voting mechanism is used to fuse time-aligned multimodal feature sequences to obtain fused multimodal sentiment features; The VA emotion model is improved by integrating multimodal emotion features as input, and the emotion recognition result is output. Emotional state tracking is performed based on the value and arousal dimensions of the emotion recognition results to obtain a sequence of emotional state changes.

2. The multimodal emotion recognition method based on AI vision according to claim 1, characterized in that, The acquisition process involves collecting facial image data from a visual sensor, collecting voice data from an audio sensor, and performing speech recognition on the voice data to obtain text data. The preprocessing includes performing face region localization and alignment on the facial image data, noise reduction and frame segmentation on the voice data, and word segmentation and encoding on the text data.

3. The multimodal emotion recognition method based on AI vision according to claim 1, characterized in that, The acquisition of the visual emotional features specifically includes: Convert the preprocessed facial image data into a grayscale image; Gradient calculation is performed on the grayscale image to obtain the horizontal and vertical gradients; Calculate the gradient magnitude and gradient direction based on the horizontal and vertical gradient directions; The HOG algorithm is used to divide the grayscale image into cell intervals. Within each cell, the gradient direction is quantized according to the preset direction interval and the direction histogram is statistically analyzed to obtain the direction histogram of the cell. Adjacent units are combined into blocks, the direction histograms within each block are spliced ​​together to obtain a block vector, and the block vector is normalized to obtain a normalized block vector. All normalized block vectors are concatenated to obtain the directional gradient histogram feature vector, which is then used as the visual emotion feature.

4. The multimodal emotion recognition method based on AI vision according to claim 1, characterized in that, The acquisition of the voice emotion features specifically includes: The preprocessed speech data is segmented into frames to obtain a sequence of speech frames arranged in chronological order. The Teager energy sequence is obtained by calculating the Teager energy using the Teager energy operator TEO for each speech frame in the speech frame sequence. Statistical calculations are performed on the Teager energy sequence to obtain the Teager energy features, which include the mean of the Teager energy sequence within the corresponding speech frame and the variance of the Teager energy sequence within the corresponding speech frame. Extract fundamental frequency, energy, pitch, and speech rate from each speech frame in the speech frame sequence; Extract Mel-frequency cepstral coefficients for each speech frame in the speech frame sequence; The Teager energy features, fundamental frequency, energy, pitch, speech rate, and Mel frequency cepstral coefficients are concatenated to obtain the speech frame-level speech emotion feature vector. The speech emotion features are obtained by combining the speech frame-level speech emotion feature vectors corresponding to all speech frames in chronological order.

5. The multimodal emotion recognition method based on AI vision according to claim 1, characterized in that, The acquisition of the text sentiment features specifically includes: The preprocessed text data is segmented by character to obtain a sequence of text units arranged in order; The text unit sequence is input into the embedding layer of the pre-trained language model to obtain a text embedding vector sequence that corresponds one-to-one with the text unit sequence, and the text embedding vector sequences are arranged in order to form a text embedding matrix. The text embedding matrix is ​​subjected to positional encoding overlay processing to obtain the position-enhanced embedding matrix; The position-enhanced embedding matrix is ​​input into the encoding network of the pre-trained language model to obtain the text context representation matrix; The text context representation matrix is ​​pooled to obtain the text global representation vector. By performing a linear mapping on the global representation vector of the text, the sentiment features of the text can be obtained.

6. The multimodal emotion recognition method based on AI vision according to claim 1, characterized in that, The specific steps for obtaining the temporally aligned multimodal feature sequence include: Establish timestamp sequences for visual emotion features, voice emotion features, and text emotion features respectively; The alignment start time and alignment end time are obtained based on the timestamp sequence, and an alignment time axis is generated between the alignment start time and the alignment end time at a fixed time interval; Visual emotion features are arranged in chronological order according to timestamp sequences to form a visual emotion feature sequence; speech emotion features are arranged in chronological order according to timestamp sequences to form a speech emotion feature sequence; and text emotion features are arranged in chronological order according to timestamp sequences to form a text emotion feature sequence. For each alignment moment in the alignment timeline, the visual emotion feature sequence is linearly interpolated based on the timestamp sequence to obtain the aligned visual emotion features. For each alignment moment in the alignment timeline, the speech emotion feature sequence is linearly interpolated based on the timestamp sequence to obtain the aligned speech emotion features; For each alignment moment in the alignment timeline, the text sentiment feature sequence is processed by nearest neighbor selection based on the timestamp sequence to obtain the aligned text sentiment features; By concatenating the alignment visual sentiment features, alignment speech sentiment features, and alignment text sentiment features corresponding to the same alignment time, a temporal alignment multimodal feature sequence is obtained.

7. The multimodal emotion recognition method based on AI vision according to claim 1, characterized in that, The specific steps involved in obtaining the fusion of multimodal emotional features include: For each alignment time in the temporal alignment multimodal feature sequence, visual candidate voting results, speech candidate voting results, and text candidate voting results are generated by majority voting based on the visual sentiment features, speech sentiment features, and text sentiment features corresponding to the alignment time, respectively. Visual voting weights, voice voting weights, and text voting weights are set for visual candidate voting results, voice candidate voting results, and text candidate voting results respectively. The votes for visual candidate voting results, voice candidate voting results, and text candidate voting results are weighted and counted to obtain the vote value corresponding to each candidate voting result. The vote value for each candidate voting result is the sum of the visual voting weight, voice voting weight, and text voting weight that are consistent with the candidate voting result. The candidate voting result with the highest vote value is selected as the voting fusion result at the alignment time, and the voting fusion result is used as the fused multimodal sentiment feature corresponding to the alignment time.

8. The multimodal emotion recognition method based on AI vision according to claim 1, characterized in that, The output of the emotion recognition result specifically includes: The improved VA sentiment model incorporates multimodal sentiment features as input. This improved VA sentiment model includes an input representation mapping module, a valence dimension regression module, an arousal dimension regression module, and a temporal consistency constraint module. The input representation mapping module performs representation mapping on the fused multimodal sentiment features to obtain a unified input representation vector. The valence dimension regression module performs quantile regression calculation and truncation mapping on the unified input representation vector to obtain valence dimension values. The arousal dimension regression module performs quantile regression calculation and truncation mapping on the unified input representation vector to obtain arousal dimension values. The temporal consistency constraint module smooths the valence dimension values ​​and arousal dimension values ​​based on total variational constraints. In the input representation mapping module, the fused multimodal emotion features are processed by representation mapping. The representation mapping process is to perform component normalization on the fused multimodal emotion features and then perform linear mapping to obtain intermediate representations. The intermediate representations are then fused with the fused multimodal emotion features based on the gated residual connection mechanism to obtain a unified input representation vector corresponding to each alignment time. In the valence dimension regression module, quantile regression is performed on the unified input representation vector to obtain the median estimate of the valence dimension value, and the median estimate of the valence dimension value is truncated and mapped to obtain the valence dimension value. In the wake-up dimension regression module, quantile regression is performed on the unified input representation vector to obtain the median estimate of the wake-up dimension value, and the median estimate of the wake-up dimension value is truncated and mapped to obtain the wake-up dimension value. In the temporal consistency constraint module, temporal consistency processing is performed on the valence dimension value and the wake-up dimension value. The temporal consistency processing refers to smoothing the valence dimension value based on the total variation constraint to obtain the temporally consistent valence dimension value, and smoothing the wake-up dimension value based on the total variation constraint to obtain the temporally consistent wake-up dimension value. The total variation constraint restricts the sum of the absolute values ​​of the numerical differences corresponding to adjacent alignment times. The temporal consistency value and temporal consistency arousal value are used as the emotion recognition results and output.

9. The multimodal emotion recognition method based on AI vision according to claim 1, characterized in that, The specific steps to obtain the emotional state change sequence include: Arrange the valence dimension values ​​and arousal dimension values ​​of the emotion recognition results in chronological order to form valence dimension value sequences and arousal dimension value sequences; The value of the first alignment moment and the arousal value are used as the initial emotional state; For each alignment moment except the first alignment moment, generate the emotional state corresponding to the current alignment moment. The emotional state is composed of the value dimension value obtained by proportionally weighting the value dimension value of the current alignment moment and the value dimension value of the emotional state of the previous alignment moment, and is composed of the arousal dimension value obtained by proportionally weighting the arousal dimension value of the current alignment moment and the arousal dimension value of the emotional state of the previous alignment moment. For each alignment moment other than the first alignment moment, calculate the change in emotional state between adjacent alignment moments. The change in the valence dimension of the change in emotional state is the valence dimension value in the emotional state at the current alignment moment minus the valence dimension value in the emotional state at the previous alignment moment. The change in the arousal dimension of the change in emotional state is the arousal dimension value in the emotional state at the current alignment moment minus the arousal dimension value in the emotional state at the previous alignment moment. The emotional state at each alignment time and the change in emotional state at each alignment time are combined in chronological order to obtain the emotional state change sequence.

Images