Emotion recognition method based on interaction acoustic signal generated when touching a companion agent

Abstract

The present application relates to the technical field of human-computer interaction, and particularly relates to an emotion recognition method for generating interactive acoustic signals based on touch accompanying intelligent bodies, contact sound and structure conduction vibration sound generated when a user touches a shell or mechanical structure of an accompanying intelligent body are collected by the accompanying intelligent body, emotion state and adaptive feedback control instructions are generated through effective contact event screening, cross-time scale feature extraction, gesture-assisted emotion calibration and cross-window stable determination, and the method is applicable to accompanying robots, vehicle-mounted interactive terminals, desktop intelligent bodies and service equipment with touchable shells. The present application distinguishes between effective touch and device self-noise, takes into account different time scales of transient contact and continuous friction, uses touch gestures to constrain emotion results, and suppresses false state updates when there is low confidence or continuous window conflict, so that it is not necessary to collect user facial images or voice content, and emotion states can be stably recognized from non-voice acoustic signals and interactive feedback can be driven.

Description

Technical Field

[0001] This invention relates to the field of human-computer interaction technology, and in particular to an emotion recognition method based on interactive acoustic signals generated when a companion smart agent is touched. Background Technology

[0002] Interactive devices with tactile shells need to sense the user's emotional state during interaction and select an appropriate response. Existing emotion recognition solutions mostly analyze facial expressions, posture, or gaze through cameras, or analyze semantically meaningful speech through far-field microphones. Other solutions deploy pressure, capacitance, or high-density tactile sensor arrays on the device surface to identify touch actions based on contact location, pressure, and duration. When a user touches the shell of the intelligent device, the contact impact, surface friction, and mechanical vibrations of the shell and internal framework generate non-speech acoustic signals. These signals contain information such as action intensity, repetition rhythm, duration, and spectral changes, and can be used to describe touch methods such as patting, tapping, poking, scratching, rubbing, and holding. They can also reflect the toucher's arousal level and valence tendency. Existing research indicates the feasibility of using acoustic channels to identify touch actions and emotional dimensions.

[0003] However, visual methods are susceptible to occlusion, lighting, and installation angle, and involve facial image acquisition; voice emotion recognition is easily affected by background noise, multiple speakers, and semantic content; large-area haptic arrays increase wiring, manufacturing, and maintenance costs. Acoustic schemes that directly classify fixed-length audio segments still have the following shortcomings: First, standby noise, robot actuator noise, and non-contact environmental noise may be fed into the model, leading to misjudgments; second, gesture classification and emotion classification are independent of each other, failing to utilize the conditional association between touch methods and emotional states; third, the classification results of a single window are prone to jumps with instantaneous disturbances and cannot be directly used as the basis for robot feedback control; fourth, the lack of confidence rejection, state maintenance, and feedback recovery mechanisms makes it difficult to form a safe and coherent online interactive closed loop. Summary of the Invention

[0004] To address the shortcomings of existing technologies, this invention provides an emotion recognition method based on interactive acoustic signals generated when a companion robot is touched. The companion robot collects contact sounds and structural vibration sounds generated when a user touches its shell or mechanical structure. After effective contact event screening, cross-timescale feature extraction, gesture-assisted emotion calibration, and cross-window stability determination, an emotional state and adaptive feedback control commands are generated. This method is applicable to companion robots, in-vehicle interactive terminals, desktop robots, and service devices with touchable shells.

[0005] To address the aforementioned technical problems, this invention provides an emotion recognition method based on interactive acoustic signals generated when a companion smart agent is touched. The method includes the following steps:

[0006] The system responds to the main acoustic sensor installed on the companion smart body, collecting the contact sound and structural vibration sound generated when the user touches its shell or mechanical structure, and obtaining continuous contact acoustic signals.

[0007] The contact acoustic signal is divided into short-time frames, and the normalized short-time energy reflecting the signal power level of the current frame is calculated. And positive spectral flux, which reflects the positive spectral change between adjacent frames. And based on a dual-threshold hysteresis mechanism, effective contact event detection is performed;

[0008] Normalized short-time energy and positive spectral flux Together they describe the intensity and dynamic characteristics of the contact acoustic signal;

[0009] A time-frequency feature map is constructed for the signal segments corresponding to the detected valid contact events. The time-frequency feature map is then input into a convolutional network with multiple parallel branches. After feature fusion, a shared embedding is generated from each branch. ;

[0010] Based on the shared embedding Touch gesture probabilities are output through independent touch gesture classification heads, arousal classification heads, and valence classification heads. Arousal probability and valence probability ;

[0011] Based on the probability of touch gestures Arousal probability and valence probability Combined with gesture-emotion compatibility matrix and state transition matrix Joint calibration of emotional states generates an emotional state probability distribution;

[0012] Only when the highest probability in the said emotional state probability distribution Not lower than the confidence threshold Highest probability With the second highest probability The difference is not less than the category interval threshold. And N consecutive windows obtain the same candidate state When using candidate states Update to a stable emotional state; otherwise, maintain the previous stable emotional state.

[0013] Based on a stable emotional state, control commands are generated that include feedback type, feedback intensity, duration, recovery conditions, and cooldown time, driving the interactive device to execute the corresponding feedback.

[0014] Furthermore, at least one of the main acoustic sensors is disposed inside the touchable shell of the companion smart entity, on the shell connector, or on the internal mechanical skeleton. The main acoustic sensor is used to collect contact sound and structurally transmitted vibration sound caused by touch, and simultaneously performs analog-to-digital conversion on the contact acoustic signal and adds a timestamp.

[0015] Furthermore, the detection of the effective contact event includes:

[0016] The continuous contact acoustic signal is divided into short time frames, and the first frame is... Frame Samples Calculate normalized short-time energy and positive spectral flux ;

[0017] Normalized short-time energy during standby background segment mean and standard deviation and positive spectral flux mean and standard deviation Set the startup thresholds respectively Maintain threshold and flux threshold Among them, the start threshold Greater than the holding threshold ;

[0018] When normalized short-time energy Not lower than the startup threshold And positive spectral flux Not lower than the flux threshold And the recording of events begins when the preset number of frames is reached;

[0019] Once the event state is entered, even if the spectral flux decreases, as long as the short-time energy is normalized... Still above the maintenance threshold The event will remain in the record.

[0020] When normalized short-time energy continuously below the holding threshold The event recording ends when the end retention time is reached.

[0021] Furthermore, the construction of the time-frequency feature map includes:

[0022] DC component removal and amplitude normalization are performed on the signal segments corresponding to valid contact events, and short-time Fourier transform is performed with a 32ms Hamming window and a 10ms preset frame shift.

[0023] The power spectrum is input into a 64-channel Mel filter bank to obtain the Mel energy. Taking the logarithm of the Mel energy yields a two-dimensional feature map formed by multiplying the number of time frames by the number of Mel channels. ;

[0024] The online phase continues to use the mean and standard deviation fixed during the training phase for the two-dimensional feature map. Standardize it.

[0025] Furthermore, the convolutional network includes three parallel branches with kernel sizes of 3×3, 5×5, and 7×7, respectively, to respond to three types of time-scale features: short-duration impact, medium-frequency rhythm, and continuous friction.

[0026] Each branch is configured with three single-layer two-dimensional convolutional modules. Each convolutional module performs convolution, batch normalization, ReLU activation, 2×2 average pooling, and Dropout operations in sequence, with the number of output channels being 16, 32, and 64 in sequence. The time axis dilation rate is 1, 2, and 3 in sequence, while the frequency axis dilation rate remains at 1.

[0027] Furthermore, the maximum and mean values ​​of the third-layer output of each branch are calculated along the time dimension and then summed, so that burst peaks and sustained energy enter the branch representation simultaneously.

[0028] Subsequently, the branch features were compressed into 64 dimensions using frequency-dimensional linear mapping and ReLU activation. Three branch characteristics The vectors are concatenated into a 192-dimensional vector, and then mapped through a fully connected layer and GELU activation to a shared 128-dimensional embedding. .

[0029] Furthermore, the touch gesture classification head outputs the probabilities of six types of touch gestures: hold, tap, poke, scratch, tap, and rub. ;

[0030] The wakefulness classification header outputs three levels of wakefulness probability: low, medium, and high. ;

[0031] The valence classification header outputs valence probabilities at three levels: negative, neutral, and positive. .

[0032] Furthermore, generating the probability distribution of the emotional state includes:

[0033] Emotional states are divided into five categories, Q0-Q4, namely:

[0034] Q0 represents moderate arousal and neutral valence; Q1 represents high arousal and positive valence; Q2 represents high arousal and negative valence; Q3 represents low arousal and negative valence; Q4 represents low arousal and positive valence.

[0035] Based on the arousal probability corresponding to the emotional state and valence probability Calculate the base score The baseline score was normalized across the five emotional states. The calculation formula is:

[0036]

[0037] in, , These represent the arousal level and valence level corresponding to the emotional state q, respectively.

[0038] Establish a gesture-emotion compatibility matrix Calculate the gesture compatibility score for each emotional state. and joint calibration score The calculation formulas are as follows:

[0039]

[0040]

[0041] in, The first output of the touch gesture classification head Gesture probability, ; For the gesture-emotion compatibility matrix, the first... Gestures and the first The compatibility coefficient between different emotional states ; A positive number smoothing constant set to prevent taking the logarithm of zero; The state transition matrix is ​​for five types of emotional states; Indicates the change from the previous stable state Transition to candidate state The transition probability; and These are the weights for the gesture compatibility term and the time continuity term, respectively.

[0042] Joint calibration score By taking the exponent and normalizing, we obtain the probability distribution of emotional states. ,Right now:

[0043]

[0044] in, For the first Joint calibration scores for emotion-like states, ; This represents an exponential function with the natural constant as its base, and the denominator is used to normalize the exponential scores for the five emotional states.

[0045] Furthermore, the confidence threshold =0.60, the category interval threshold =0.15, and 2 for N consecutive windows.

[0046] The present invention also provides a system for implementing the above-described emotion recognition method, comprising:

[0047] An acoustic acquisition module is installed in the touchable housing of the interactive device to collect the contact acoustic signals generated by touch.

[0048] The effective contact event gating module is signal-connected to the acoustic acquisition module and is used to calculate the short-time energy and positive spectral flux of the contact acoustic signal in short-time frames. It detects effective contact events based on a dual-threshold hysteresis mechanism where the activation threshold is greater than the hold threshold. If there is no effective contact event, subsequent emotion inference is not triggered.

[0049] The feature extraction module is connected to the effective contact event gating module and is used to construct a time-frequency feature map for the signal segment corresponding to the effective contact event. The time-frequency feature map is then input into a convolutional network with multiple parallel branches. Each parallel branch uses a different convolutional kernel size to cover different time receptive fields. The features of each branch are fused to generate a shared embedding.

[0050] The multi-task classification module is data-connected to the feature extraction module and includes independent touch gesture classification heads, arousal classification heads, and valence classification heads, which are used to output touch gesture probabilities, arousal probabilities, and valence probabilities based on the shared embedding, respectively.

[0051] The joint calibration module is connected to the multi-task classification module and is used to jointly calibrate the emotional state based on the touch gesture probability, arousal probability and valence probability, combined with the gesture-emotion compatibility matrix and the state transition matrix, to generate an emotional state probability distribution.

[0052] The stability determination module is connected to the joint calibration module and is used to update the stable emotional state only when the highest probability in the probability distribution of the emotional state is not lower than the confidence threshold, the difference between the highest probability and the second highest probability is not lower than the category interval threshold, and the same candidate state is obtained in a consecutive preset number of windows; otherwise, the previous stable emotional state is maintained.

[0053] The feedback control module is data-connected to the stability determination module and is used to generate control instructions containing feedback type, feedback intensity, duration and recovery conditions according to the stable emotional state, so as to drive the interactive device to execute the corresponding feedback.

[0054] The state memory unit, connected to the joint calibration module and the stability determination module, is used to store the previous stable emotional state and the gesture-emotion compatibility matrix. State transition matrix Confidence threshold, category interval threshold, and cooling time parameters.

[0055] By employing the above technical solution, the present invention provides an emotion recognition method based on interactive acoustic signals generated when touching a companion smart agent, which has at least the following beneficial effects:

[0056] 1. This invention distinguishes between effective touch and device self-noise, takes into account different time scales of transient contact and continuous friction, uses touch gestures to constrain emotional outcomes, and suppresses erroneous state updates when there is low confidence or continuous window conflict. Therefore, it does not require the collection of user facial images or voice content, and can stably identify emotional states and drive interactive feedback from non-voice acoustic signals generated by the touch companion agent.

[0057] 2. This invention uses a dual-threshold hysteresis effective contact event gating mechanism based on short-time energy and positive spectral flux to accurately distinguish effective touch events from standby noise, actuator self-noise, and non-contact ambient sound before the signal enters the classification model. When there is no effective contact event, emotion inference is not initiated, fundamentally eliminating the misjudgment problem caused by arbitrary audio segments being directly sent into the classification model in existing solutions.

[0058] 3. This invention sets up three parallel branches with convolution kernel sizes of 3×3, 5×5 and 7×7, and combines them with the time axis mixed dilation rate to simultaneously cover three time scale features: short impact, medium rhythm and continuous friction. This solves the problem that existing single time scale feature representations do not cover different touch gesture types, and significantly improves the ability to represent various touch methods such as holding, patting, poking, scratching, tapping and rubbing.

[0059] 4. This invention sets up independent gesture classification heads, arousal classification heads, and valence classification heads in parallel on a shared embedding, so that the gesture probability, arousal probability, and valence probability of the same contact event share a unified representation basis. The three types of probability outputs have consistent semantic alignment in the same feature space, providing a high-quality multi-task probability interface for subsequent joint calibration.

[0060] 5. This invention establishes a gesture-emotion compatibility matrix and uses a log-linear fusion method to jointly calculate the probability distribution of emotional states by combining gesture compatibility scores, basic emotion scores, and state transition constraints. It uses the conditional correlation between touch methods and emotional states to impose interpretable prior constraints on emotional conclusions, overcoming the shortcomings of existing schemes where gesture classification and emotion classification are independent and cannot utilize the conditional correlation between the two.

[0061] 6. This invention employs a stability determination and rejection mechanism based on three conditions: confidence threshold, category interval threshold, and continuous window consistency. It updates the stable emotional state only when all three conditions are met simultaneously. When confidence is continuously low, it automatically enters an unknown state or a safe standby state. This effectively suppresses state jumps caused by instantaneous disturbances in a single window and solves the instability problem of feedback control directly driven by the classification result of a single window in existing solutions.

[0062] 7. This invention, through a combination of cooling timer and feedback period gating mechanism, restricts the repeated triggering of similar feedback after state update, and increases the activation threshold of effective contact events during feedback action to prevent the feedback sound of the interactive device itself from being identified as user touch again, thus forming a safe and coherent online interaction closed loop and solving the technical defects of existing solutions that lack feedback recovery mechanism and anti-self-excitation protection. Attached Figure Description

[0063] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0064] Figure 1 This is a flowchart of the emotion recognition method in this invention;

[0065] Figure 2 This is a block diagram illustrating the principle of the emotion recognition system in this invention.

[0066] Figure 3 This is a diagram of the cross-timescale shared representation network and multi-task output structure in this invention;

[0067] Figure 4 This is a schematic diagram of the gesture-assisted emotion calibration, stability determination, and feedback state machine in this invention. Detailed Implementation

[0068] To make the above-mentioned objects, features, and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. This will allow for a full understanding of how the present application uses technical means to solve technical problems and achieve technical effects, and to facilitate its implementation.

[0069] This embodiment proposes an emotion recognition method based on interactive acoustic signals generated when touching a companion intelligent agent. By distinguishing between effective touch and device self-noise, and considering different time scales of transient contact and continuous friction, it uses touch gestures to constrain the emotional outcome and suppresses erroneous state updates during low-confidence or continuous window conflicts. Therefore, it does not require collecting user facial images or voice content, and can stably identify emotional states and drive interactive feedback from non-voice acoustic signals generated by the companion intelligent agent. Figure 1As shown, the method includes the following steps:

[0070] S1. The main acoustic sensor installed on the companion smart body collects the contact sound and structural vibration sound generated when the user touches its shell or mechanical structure, and obtains continuous contact acoustic signals.

[0071] This embodiment places at least one primary acoustic sensor on the inside of the touchable shell of the companion intelligent agent, on the shell connector, or on the internal mechanical skeleton. The primary acoustic sensor can be an electret microphone, MEMS microphone, contact microphone, piezoelectric pickup, or accelerometer-type vibration sensor. It is used to collect contact sound caused by touch and structurally transmitted vibration sound, while simultaneously performing analog-to-digital conversion on the contact acoustic signal and adding a timestamp. To avoid collecting identifiable voice content, the sensor preferentially adopts a mounting method that is mechanically coupled to the shell and relatively insensitive to airborne voice. This mechanically coupled mounting method makes the sensor highly sensitive to contact sound and structurally transmitted vibration sound, while being relatively insensitive to airborne voice and ambient sound, fundamentally reducing the risk of collecting identifiable voice content.

[0072] In an optional dual-sensor implementation, a reference sensor is additionally provided, mechanically isolated from the touchable surface and adjacent to the actuator. When the interactive device is not in external contact and performs a preset action, the noise transfer function from the reference channel to the main channel is estimated. During online interaction, the main channel spectrum is corrected according to the noise transfer function to obtain the spectrum after suppressing self-noise. Then, effective contact event detection is performed; when the reference channel energy increases abnormally but the main channel does not meet the spectral flux condition, the signal segment is marked as device self-noise and updating the emotional state is prohibited. The main channel spectrum is denoted as... The reference channel spectrum is denoted as Through noise transfer function Obtain the spectrum after suppressing self-noise ,Right now:

[0073]

[0074] Noise transfer function Calibration can be performed using a least mean square adaptive algorithm when the agent executes a preset action without external contact; alternatively, Wiener filtering or spectral reduction can be applied based on the noise power spectrum of the reference channel. As an option, this embodiment does not require a reference sensor.

[0075] S2. Divide the contact acoustic signal into short-time frames and calculate the normalized short-time energy to reflect the signal power level of the current frame. And positive spectral flux, which reflects the positive spectral change between adjacent frames. It also performs effective contact event detection based on a dual-threshold hysteresis mechanism.

[0076] This embodiment divides continuous contact acoustic signals into short time frames. For the first... Frame Samples Calculate normalized short-time energy and positive spectral flux Normalized short-time energy and positive spectral flux Together, they describe the intensity and dynamic characteristics of the contact acoustic signal. The calculation formula is:

[0077]

[0078]

[0079] in, The number of temporal sampling points contained in each short-time frame; For the first The first frame Each time-domain sampled value, ; For the first After the short-time Fourier transform, the frame is at the 1st Complex spectral coefficients at each frequency point Its amplitude; The frequency point index is used, and the summation range is the frequency points of the single-sided spectrum used. A positive smoothing constant set to prevent the denominator from being zero.

[0080] Normalized short-time energy during standby background segment mean and standard deviation and positive spectral flux mean and standard deviation Set the startup thresholds respectively Maintain threshold and flux threshold Among them, the start threshold Greater than the holding threshold .

[0081] In the embodiment, the activation threshold Maintain threshold and flux threshold The calculation formulas are as follows:

[0082]

[0083]

[0084]

[0085] Among them, the start threshold Set as the energy mean plus 3 times the energy standard deviation, and maintain the threshold. Set the flux threshold to the energy mean plus 1.5 times the energy standard deviation. The coefficients are set as the mean of the spectral flux plus twice the standard deviation of the spectral flux. These coefficients, configured under typical accompanying device housing materials and installation conditions, provide a good balance between detection sensitivity and resistance to noise-induced false triggering. These coefficients can be calibrated and adjusted according to the device material, sensor installation location, and ambient noise.

[0086] When normalized short-time energy Not lower than the startup threshold And positive spectral flux Not lower than the flux threshold And while continuously recording the preset number of frames, the event begins; after entering the event state, even if the spectral flux decreases, as long as the normalized short-time energy is recorded... Still above the maintenance threshold The event is then retained; when the short-time energy is normalized... continuously below the holding threshold The event recording ends when the end retention time is reached.

[0087] This dual-threshold hysteresis mechanism prevents critical noise from causing event boundaries to repeatedly open and close, thus accurately distinguishing valid touch events from standby noise, actuator self-noise, and non-contact ambient noise. The minimum required segment for the model is set to 1.10s; if insufficient, the context before and after the start and end is incorporated and zero-padding or mirror padding is applied. If the maximum window length is exceeded, an overlapping sliding window is used. No signals are passed to subsequent modules when there are no valid events, and sentiment inference is not initiated.

[0088] S3. Construct a time-frequency feature map for the signal segments corresponding to the detected valid contact events, and input the time-frequency feature map into a convolutional network with multiple parallel branches. After feature fusion, each parallel branch generates a shared embedding. .

[0089] The time-frequency feature map is constructed as follows: DC component removal and amplitude normalization are performed on the signal segments corresponding to the effective contact events, and a short-time Fourier transform is performed using a 32ms Hamming window and a 10ms preset frame shift. The Hamming window is as follows:

[0090]

[0091] in, For the Hanming window in the first Window function coefficients at each sampling point; The total number of sampling points contained in a short-time frame; For the intra-frame sampling point index, and .

[0092] The power spectrum is input into a 64-channel Mel filter bank to obtain the Mel energy. Taking the logarithm of the Mel energy yields a two-dimensional feature map formed by multiplying the number of time frames by the number of Mel channels. The expression is:

[0093]

[0094] in, This represents the total number of time frames corresponding to the signal segment. For time frame index, and ; For Mel channel index, and ; A positive smoothing constant set to prevent zero Mel energy from entering logarithmic operations.

[0095] The online phase continues to use the mean and standard deviation fixed during the training phase for the two-dimensional feature map. Standardize it.

[0096] like Figure 3 As shown, the two-dimensional feature map Transform into shape The tensor is then normalized to the input batch, where This represents the number of samples in the batch.

[0097] The convolutional network comprises three parallel branches with kernel sizes of 3×3, 5×5, and 7×7, respectively, to respond to three temporal scale features: short-duration impact, moderate rhythm, and continuous friction. Each branch sequentially incorporates three single-layer two-dimensional convolutional modules with output channels of 16, 32, and 64, respectively; the temporal dilation rates are 1, 2, and 3, respectively, while the frequency dilation rate remains at 1. This hybrid dilation rate reduces the number of layers while expanding the temporal receptive field and decreasing the possibility of grid omissions caused by consecutively using the same dilation rate.

[0098] The processing of the b-th branch at the l-th layer can be written as follows:

[0099] ;

[0100] in, The branch index is identified by the side length of the convolution kernel, and its value can be 3, 5 or 7; This is the index of the convolutional layer within the branch, with a value of 1, 2, or 3; For the first The first branch road The feature tensor output by the layer The feature tensor after input batch normalization; Indicates the kernel size as Expansion rate The two-dimensional convolution operation has kernel sizes of 3×3, 5×5, and 7×7 for the three branches; the dilation rates of the three layers are respectively... , and ; Indicates batch normalization, Indicates linear rectification activation. Indicates the pooling window is Average pooling, This indicates a regularization operation that randomly masks features according to a preset inactivation rate during the training phase.

[0101] Each convolutional module sequentially performs convolution, batch normalization, ReLU activation, 2×2 average pooling, and Dropout operations. 2×2 average pooling is used to smooth local fluctuations in the touch sound, and Dropout is used to reduce overfitting during training.

[0102] The third-layer output of each branch is maximized and averaged along the time dimension and then summed, so that burst peaks and sustained energy are simultaneously included in the branch representation; subsequently, it is compressed into 64-dimensional branch features through frequency-dimensional linear mapping and ReLU activation. Three branch characteristics The vectors are concatenated into a 192-dimensional vector, and then mapped through a fully connected layer and GELU activation to a shared 128-dimensional embedding. ,Right now:

[0103]

[0104]

[0105] in, For the first The frequency dimension linear mapping weights of each branch are used to compress the frequency dimension of the time-statistical features from 6 positions to 1 position; and These represent calculating the maximum and average values ​​element-wise along the time dimension, respectively. For the first The feature tensor output by the third layer of each branch; The bias of the frequency-dimensional linear mapping; , and The 64-dimensional feature vectors generated for the three branches are respectively; This represents a 192-dimensional vector obtained by concatenating the features. To map a 192-dimensional concatenated vector to a fully connected layer weight matrix in a 128-dimensional shared embedding space; This is the bias vector of the fully connected layer; This represents the Gaussian error linear unit activation function.

[0106] S4, Based on the shared embedding Touch gesture probabilities are output through independent touch gesture classification heads, arousal classification heads, and valence classification heads. Arousal probability and valence probability .

[0107] This implementation uses shared embedding Subsequently, independent touch gesture classification heads, arousal classification heads, and valence classification heads are set up. The touch gesture classification head outputs the probabilities of six types of touch gestures: hold, tap, poke, scratch, tap, and rub. The wakefulness classification header outputs wakefulness probabilities at three levels: low, medium, and high. The valence classification head outputs the valence probabilities at three levels: negative, neutral, and positive. The expressions are as follows:

[0108]

[0109]

[0110]

[0111] in, , and These are the weight matrices for the touch gesture classifier, arousal classifier, and valence classifier, respectively, with dimensions of [dimensions to be filled in]. , and ; , and These are the bias vectors for the three classification heads, with dimensions respectively. , and ; This represents the normalization exponential function, used to convert the category scores output by each classifier into a probability vector with non-negative elements that sum to 1.

[0112] In this embodiment, the three classification heads share the same embedding representation generated by the same feature extraction module, thus increasing the probability of touch gestures for the same valid contact event. Arousal probability and valence probability It has a unified representation basis. During training, the cross-entropy loss is calculated for each of the three classifiers and then summed according to configurable weights. The shared encoder provides a probability interface for the same contact event to subsequent joint decisions by simultaneously learning the action rhythm and emotion dimensions. The classifiers only output probabilities during inference and do not directly trigger robot feedback at this layer.

[0113] S5. Based on touch gesture probability Arousal probability and valence probability Combined with gesture-emotion compatibility matrix and state transition matrix Emotional states are jointly calibrated to generate an emotional state probability distribution.

[0114] like Figure 4 As shown, this embodiment divides emotional states into five categories: Q0-Q4, namely: Q0 is moderate arousal with neutral valence; Q1 is high arousal with positive valence; Q2 is high arousal with negative valence; Q3 is low arousal with negative valence; and Q4 is low arousal with positive valence.

[0115] Based on the arousal probability corresponding to emotional state q and valence probability Calculate the base score And normalized among the five emotional states; among them, , These represent the arousal level and valence level corresponding to the emotional state q. The probability quality of not falling into any of the five typical combinations is characterized by low confidence or a small class interval, thus entering the rejection path.

[0116] S6. Establish a gesture-emotion compatibility matrix Gesture-Emotion Compatibility Matrix The gesture compatibility score can be obtained from training samples with paired gesture and emotion annotations using Laplace smoothing statistics, or it can be updated using user calibration data. The gesture compatibility score and joint calibration score for each emotion state are calculated, and then the gesture compatibility score for the q-th emotion state is obtained. and joint calibration score They are respectively:

[0117]

[0118]

[0119] in, The first output of the touch gesture classification head Gesture probability, ; For the gesture-emotion compatibility matrix, the first... Gestures and the first The compatibility coefficient between different emotional states ; A positive number smoothing constant set to prevent taking the logarithm of zero; The state transition matrix is ​​for five types of emotional states. Indicates the change from the previous stable state Transition to candidate state The transition probability; and These are the weights for the gesture compatibility term and the time continuity term, respectively.

[0120] Joint calibration score By taking the exponent and normalizing, we obtain the probability distribution of the q-th emotional state. ,Right now:

[0121]

[0122] in, For the first Joint calibration scores for emotion-like states, ; This represents an exponential function with the natural constant as its base, and the denominator is used to normalize the exponential scores for the five emotional states.

[0123] In this invention, the gesture-emotion compatibility matrix State transition matrix The weights are determined solely by training data, validation data, or the user calibration process, without using the true label of the current test segment. This embodiment simultaneously introduces gesture priors and historical state continuity constraints into the joint calibration score, giving emotion determination an interpretable multidimensional basis and overcoming the deficiency that gesture classification and emotion classification are independent of each other.

[0124] Take the probability distribution of emotional states The highest probability in The second highest probability and candidate status Only when , And N consecutive windows obtain the same candidate state Candidate states are only used when... Update to a stable emotional state; otherwise, maintain the previous stable emotional state.

[0125] An easy-to-implement initial configuration is: confidence threshold. Category Interval Threshold Preset windows The system is adjusted through validation sets or device calibration. When multiple consecutive events fall below a threshold, it enters an unknown state or a safe standby state, without interpreting uncertain results as user emotions. The joint determination of the three conditions effectively suppresses state jumps caused by instantaneous disturbances in a single window, preventing low-confidence results from directly triggering feedback.

[0126] S7. Generate control commands based on a stable emotional state, including feedback type, feedback intensity, duration, recovery conditions, and cooldown time, to drive the interactive device to execute the corresponding feedback.

[0127] In this embodiment, The state corresponds to positive and more active light or motion feedback; The corresponding soothing feedback is to reduce the amplitude of movements and slow down the pace. The appropriate state is gentle companionship and low-stimulation response; The state corresponds to gentle, positive feedback; The state corresponds to normal standby or minimal feedback; when entering an unknown state or with insufficient confidence, it also maintains safe standby or minimal feedback. The voice feedback preferentially uses preset phrases or non-content-based sounds that do not require continuous acquisition of user semantic content.

[0128] After the emotional state is stabilized and updated, a cooldown timer is initiated, limiting the repeated triggering of similar feedback during the cooldown period. During the feedback action, the threshold for effective touch events is raised to prevent the interactive device's own feedback sound from being identified as a user touch again, thus blocking the feedback self-excitation path at the signal level. When the touch event ends, the stable emotional state returns to Q0, there is continuous low confidence, or the longest duration of feedback is reached, the controller smoothly reduces the feedback intensity with a preset ramp and returns to the normal standby mode. The combined mechanism of cooldown timer and feedback period gating prevents feedback self-excitation from both time and signal dimensions, ensuring the safety and continuity of the interaction loop.

[0129] This embodiment also proposes an emotion recognition system for implementing the above-described emotion recognition method, comprising:

[0130] An acoustic acquisition module is installed in the touchable housing of the interactive device to collect the contact acoustic signals generated by touch.

[0131] The effective contact event gating module is signal-connected to the acoustic acquisition module and is used to calculate the short-time energy and positive spectral flux of the contact acoustic signal in short-time frames. It detects effective contact events based on a dual-threshold hysteresis mechanism where the activation threshold is greater than the hold threshold. If there is no effective contact event, subsequent emotion inference is not triggered.

[0132] The feature extraction module is connected to the effective contact event gating module and is used to construct a time-frequency feature map for the signal segment corresponding to the effective contact event. The time-frequency feature map is then input into a convolutional network with multiple parallel branches. Each parallel branch uses a different convolutional kernel size to cover different time receptive fields. The features of each branch are fused to generate a shared embedding.

[0133] The multi-task classification module is data-connected to the feature extraction module and includes independent touch gesture classification heads, arousal classification heads, and valence classification heads, which are used to output touch gesture probabilities, arousal probabilities, and valence probabilities based on the shared embedding, respectively.

[0134] The joint calibration module is connected to the multi-task classification module and is used to jointly calibrate the emotional state based on the touch gesture probability, arousal probability and valence probability, combined with the gesture-emotion compatibility matrix and the state transition matrix, to generate an emotional state probability distribution.

[0135] The stability determination module is connected to the joint calibration module and is used to update the stable emotional state only when the highest probability in the probability distribution of the emotional state is not lower than the confidence threshold, the difference between the highest probability and the second highest probability is not lower than the category interval threshold, and the same candidate state is obtained in a consecutive preset number of windows; otherwise, the previous stable emotional state is maintained.

[0136] The feedback control module is data-connected to the stability determination module and is used to generate control instructions containing feedback type, feedback intensity, duration and recovery conditions according to the stable emotional state, so as to drive the interactive device to execute the corresponding feedback.

[0137] The state memory unit is connected to the joint calibration module and the stability determination module for storing the gesture-emotion compatibility matrix, state transition matrix, confidence threshold, category interval threshold, and cooldown timing parameters.

[0138] Combination Figure 2The complete signal flow of the emotion recognition system executing the emotion recognition method is as follows: The acoustic acquisition module continuously acquires contact acoustic signals; the effective contact event gating module calculates the short-time energy and positive spectral flux of each frame of signal and determines whether there is an effective contact event through a dual-threshold hysteresis mechanism; when there is no effective contact event, the system maintains a safe standby and does not initiate emotion inference; after an effective contact event is detected, the feature extraction module preprocesses the corresponding signal segment and extracts a 64-channel log-Melogram feature map, inputs it into a three-branch convolutional network to extract cross-timescale features, and generates a 128-dimensional shared embedding; the multi-task classification module outputs gesture probability, arousal probability, and valence probability in parallel based on the shared embedding; the joint calibration module calculates the probability distribution of the emotion state by combining the gesture-emotion compatibility matrix and the state transition matrix; the stability determination module determines whether to update the stable emotion state based on three conditions; the feedback control module generates control commands based on the stable emotion state and drives the interactive device to execute feedback, while simultaneously starting the cooling timer and feedback period gating; after completing one feedback cycle, the system continues to loop online and returns to the acoustic signal acquisition stage.

[0139] In this invention, the main contact microphone is fixed to the inside of the forearm housing of the companion intelligent agent and isolated from airborne sound sources by an elastic pad. After the device is started, background frames are first acquired during the standby phase without external contact to obtain the gating threshold. When the user performs actions such as holding, patting, poking, scratching, tapping, or rubbing, the system locates valid events. For events with a duration of less than 1.10 seconds, the buffer before and after the event is added to the window; if it is still insufficient, it is mirrored at the end of the signal. Continuous events are segmented according to sliding windows, and adjacent windows retain overlapping areas to maintain the continuity of actions.

[0140] Each window is converted to a 64-channel log-Mel feature, input Figure 3 The network shown. Three classification heads are obtained. , and The five-state probabilities are then calculated. The system updates the state only when the confidence level, class interval, and continuous window condition are all met simultaneously. If the current candidate is Q2 and the stability condition is met, the controller selects a low-intensity, low-speed soothing feedback; during the feedback, a cooling timer is started and the gating threshold is increased. After touch stops or the feedback times out, the device reduces the feedback intensity with a preset ramp and returns to standby.

[0141] In addition to the main contact sensor, a reference sensor is placed near the robot actuator, with a mechanical isolation layer between the reference sensor and the user-touchable housing. When the device performs preset actions such as head turning, arm swinging, or speaker playback without human touch, the main and reference channels are collected, and the noise transfer function from the reference channel to the main channel is estimated. A calibration spectrum is generated during online interaction, and event gating is then applied. If the energy in the reference channel increases abnormally but the main channel does not meet the spectral flux condition for a contact event, this segment is marked as device self-noise, and updating the emotional state is prohibited. When the reference sensor fails, the system degenerates into the single-sensor mode of Embodiment 1.

[0142] Upon initial use, the system guides users through a small number of voluntarily labeled touch interactions to obtain user-specific gesture-emotion pair samples. Based on the general compatibility matrix M, the system updates the user matrix using weighted counting; when individual samples are insufficient, higher weights are still allocated to the general matrix to prevent offsetting due to a few mislabeled samples. Users can turn off or clear the individualized parameters. This calibration only updates M, the threshold, and the state transition parameters, eliminating the need to retrain all convolutional layers.

[0143] This embodiment also provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, the processor reads from the storage medium and executes each step of the above-described emotion recognition method: the processor acquires contact acoustic signals through an interface corresponding to the acoustic acquisition module, performs short-time energy and spectral flux calculations and dual-threshold hysteresis gating judgment, and after detecting a valid contact event, performs time-frequency feature extraction, forward inference of a three-branch convolutional network, multi-task probability output, joint calibration calculation, stability determination, and feedback instruction generation, and outputs the generated control instructions to the actuator of the interactive device through the interface to drive the corresponding feedback action. The gesture-emotion compatibility matrix, state transition matrix, threshold parameters, and cooldown timing state are all stored in the storage medium, which the processor reads during each inference and writes back after calibration update.

[0144] It is understandable that the main acoustic sensor is not limited to the electret microphone mentioned above; MEMS microphones, contact microphones, piezoelectric pickups, or accelerometer-type vibration sensors can also be used, as long as they can form mechanical coupling with the shell and collect contact sound and structurally transmitted vibration sound. Time-frequency features are not limited to a 64-channel log-Mel spectrum; constant Q-transform, short-time Fourier amplitude spectrum, or time-frequency representation generated by a one-dimensional waveform encoder can also be used; the number of Mel channels can be adjusted according to computational resources. The number of branches in the multi-branch parallel convolutional network is not limited to three; the kernel size, number of channels, and dilation rate can be adjusted according to the shell material and computational resources; branch fusion methods can include splicing, weighted summation, gated fusion, or attention fusion, as long as features at different time scales can be integrated into a unified shared embedding. Emotional state classification is not limited to five categories from Q0 to Q4; the number of arousal and valence levels can be adjusted according to the application scenario; gesture categories are not limited to six categories and can be added or removed according to actual needs. Feedback actuators are not limited to lights or motions, but may also include displays of facial expressions, vibrations, non-content-based sounds, or preset phrases, and may limit the range of motion according to device safety constraints.

[0145] Those skilled in the art will understand that all or part of the steps in the methods of the above embodiments can be implemented by a program instructing related hardware. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Moreover, this application can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0146] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. Since the above embodiments are substantially similar to the method embodiments, their descriptions are relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0147] The above embodiments provide a detailed description of the present invention. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of the present invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. An emotion recognition method based on interactive acoustic signals generated when a companion agent is touched, characterized in that, The method includes the following steps: The system responds to the main acoustic sensor installed on the companion smart body, collecting the contact sound and structural vibration sound generated when the user touches its shell or mechanical structure, and obtaining continuous contact acoustic signals. The contact acoustic signal is divided into short-time frames, and the normalized short-time energy reflecting the signal power level of the current frame is calculated. And positive spectral flux, which reflects the positive spectral change between adjacent frames. And based on a dual-threshold hysteresis mechanism, effective contact event detection is performed; Normalized short-time energy and positive spectral flux Together they describe the intensity and dynamic characteristics of the contact acoustic signal; A time-frequency feature map is constructed for the signal segments corresponding to the detected valid contact events. The time-frequency feature map is then input into a convolutional network with multiple parallel branches. After feature fusion, a shared embedding is generated from each branch. ; Based on the shared embedding Touch gesture probabilities are output through independent touch gesture classification heads, arousal classification heads, and valence classification heads. Arousal probability and valence probability ; Based on the probability of touch gestures Arousal probability and valence probability Combined with gesture-emotion compatibility matrix and state transition matrix Joint calibration of emotional states generates an emotional state probability distribution; Only when the highest probability in the said emotional state probability distribution Not lower than the confidence threshold Highest probability With the second highest probability The difference is not less than the category interval threshold. And N consecutive windows obtain the same candidate state When using candidate states Update to a stable emotional state; otherwise, maintain the previous stable emotional state. Based on a stable emotional state, control commands are generated, including feedback type, feedback intensity, duration, recovery conditions, and cooldown time, to drive the interactive device to execute the corresponding feedback.

2. The emotion recognition method according to claim 1, characterized in that, The system includes at least one primary acoustic sensor disposed inside the touchable housing of the companion smart entity, on the housing connector, or on the internal mechanical frame. The primary acoustic sensor is used to collect contact sound and structurally transmitted vibration sound caused by touch, and simultaneously performs analog-to-digital conversion on the contact acoustic signal and adds a timestamp.

3. The emotion recognition method according to claim 1, characterized in that, The detection of the effective contact event includes: The continuous contact acoustic signal is divided into short time frames, and the first frame is... Frame Samples Calculate normalized short-time energy and positive spectral flux ; Normalized short-time energy during standby background segment mean and standard deviation and positive spectral flux mean and standard deviation Set the startup thresholds respectively Maintain threshold and flux threshold Among them, the start threshold Greater than the holding threshold ; When normalized short-time energy Not lower than the startup threshold And positive spectral flux Not lower than the flux threshold And the recording of events begins when the preset number of frames is reached; Once the event state is entered, even if the spectral flux decreases, as long as the short-time energy is normalized... Still above the maintenance threshold The event will remain in the record. When normalized short-time energy continuously below the holding threshold The event recording ends when the end retention time is reached.

4. The emotion recognition method according to claim 1, characterized in that, The construction of the time-frequency feature map includes: DC component removal and amplitude normalization are performed on the signal segments corresponding to valid contact events, and short-time Fourier transform is performed with a 32ms Hamming window and a 10ms preset frame shift. The power spectrum is input into a 64-channel Mel filter bank to obtain the Mel energy. Taking the logarithm of the Mel energy yields a two-dimensional feature map formed by multiplying the number of time frames by the number of Mel channels. ; The online phase continues to use the mean and standard deviation fixed during the training phase for the two-dimensional feature map. Standardize it.

5. The emotion recognition method according to claim 1, characterized in that, The convolutional network includes three parallel branches with kernel sizes of 3×3, 5×5 and 7×7, respectively, to respond to three types of time-scale features: short-duration impact, medium-frequency rhythm and continuous friction. Each branch is configured with three single-layer two-dimensional convolutional modules. Each convolutional module performs convolution, batch normalization, ReLU activation, 2×2 average pooling, and Dropout operations in sequence, with the number of output channels being 16, 32, and 64 in sequence. The time axis dilation rate is 1, 2, and 3 in sequence, while the frequency axis dilation rate remains at 1.

6. The emotion recognition method according to claim 5, characterized in that, The third-level output of each branch is calculated along the time dimension, and the maximum and mean values ​​are added together, so that burst peaks and sustained energy enter the branch representation simultaneously; Subsequently, the branch features were compressed into 64 dimensions using frequency-dimensional linear mapping and ReLU activation. Three branch characteristics The vectors are concatenated into a 192-dimensional vector, and then mapped through a fully connected layer and GELU activation to a shared 128-dimensional embedding. .

7. The emotion recognition method according to claim 1, characterized in that, The touch gesture classification head outputs the probabilities of six types of touch gestures: hold, tap, poke, scratch, tap, and rub. ; The wakefulness classification header outputs three levels of wakefulness probability: low, medium, and high. ; The valence classification head outputs valence probabilities at three levels: negative, neutral, and positive. .

8. The emotion recognition method according to claim 7, characterized in that, Generating the probability distribution of the emotional state includes: Emotional states are divided into five categories, Q0-Q4, namely: Q0 represents moderate arousal and neutral valence; Q1 represents high arousal and positive valence; Q2 represents high arousal and negative valence; Q3 represents low arousal and negative valence; Q4 represents low arousal and positive valence. Based on the arousal probability corresponding to emotional state and valence probability Calculate the base score The baseline score was normalized across the five emotional states. The calculation formula is: ； in, , These represent the arousal level and valence level corresponding to the emotional state q, respectively. Establish a gesture-emotion compatibility matrix Calculate the gesture compatibility score for each emotional state. and joint calibration score The calculation formulas are as follows: ； ； in, The first output of the touch gesture classification head Gesture probability, ; For the gesture-emotion compatibility matrix, the first... Gestures and the first The compatibility coefficient between different emotional states ; A positive number smoothing constant set to prevent taking the logarithm of zero; The state transition matrix is ​​for five types of emotional states; Indicates the change from the previous stable state Transition to candidate state The transition probability; and These are the weights for the gesture compatibility term and the time continuity term, respectively. Joint calibration score By taking the exponent and normalizing, we obtain the probability distribution of emotional states. ,Right now: ； in, For the first Joint calibration scores for emotion-like states, ; This represents an exponential function with the natural constant as its base.

9. The emotion recognition method according to claim 1, characterized in that, The confidence threshold =0.60, the category interval threshold =0.15, and 2 for N consecutive windows.

10. A system for implementing the emotion recognition method according to any one of claims 1-9, characterized in that, include: An acoustic acquisition module is installed in the touchable housing of the interactive device to collect the contact acoustic signals generated by touch. The effective contact event gating module is signal-connected to the acoustic acquisition module and is used to calculate the short-time energy and positive spectral flux of the contact acoustic signal in short-time frames. It detects effective contact events based on a dual-threshold hysteresis mechanism where the activation threshold is greater than the hold threshold. If there is no effective contact event, subsequent emotion inference is not triggered. The feature extraction module is connected to the effective contact event gating module and is used to construct a time-frequency feature map for the signal segment corresponding to the effective contact event. The time-frequency feature map is then input into a convolutional network with multiple parallel branches. Each parallel branch uses a different convolutional kernel size to cover different time receptive fields. The features of each branch are fused to generate a shared embedding. The multi-task classification module is data-connected to the feature extraction module and includes independent touch gesture classification heads, arousal classification heads, and valence classification heads, which are used to output touch gesture probabilities, arousal probabilities, and valence probabilities based on the shared embedding, respectively. The joint calibration module is connected to the multi-task classification module and is used to jointly calibrate the emotional state based on the touch gesture probability, arousal probability and valence probability, combined with the gesture-emotion compatibility matrix and the state transition matrix, to generate an emotional state probability distribution. The stability determination module is connected to the joint calibration module and is used to update the stable emotional state only when the highest probability in the probability distribution of the emotional state is not lower than the confidence threshold, the difference between the highest probability and the second highest probability is not lower than the category interval threshold, and the same candidate state is obtained in a consecutive preset number of windows; otherwise, the previous stable emotional state is maintained. The feedback control module is data-connected to the stability determination module and is used to generate control instructions containing feedback type, feedback intensity, duration and recovery conditions according to the stable emotional state, so as to drive the interactive device to execute the corresponding feedback. The state memory unit, connected to the joint calibration module and the stability determination module, is used to store the previous stable emotional state and the gesture-emotion compatibility matrix. State transition matrix Confidence threshold, category interval threshold, and cooling timing parameters.

Images