Intelligent household electrical appliance interaction experience evaluation method and system

Abstract

The invention provides an intelligent household electrical appliance interaction experience evaluation method and system. The method comprises the following steps: collecting a physiological signal and a behavior signal of a user; processing the physiological signal and the behavior signal according to a preset rule to obtain a physiological signal feature and a behavior signal feature; inputting the obtained physiological signal features and behavior signal features into a preset multi-mode deep belief network to obtain a multi-mode fusion feature vector; and performing interactive experience category label labeling on the obtained multi-modal fusion feature vector by using a preset classifier to obtain a final evaluation result. Physiological data and behavior data in the interactive experience process of the user and the intelligent household electrical appliance are collected and analyzed in real time, the modal features of the physiological signals and the behavior signals are fused by adopting the learning characteristics of the deep belief network, the complementarity advantage between different modal feature data is fully played, and the user experience is improved. And comprehensive, scientific and accurate intelligent household electrical appliance interaction experience comprehensive evaluation is realized.

Description

Technical Field

[0001] The present invention belongs to the technical field of human-computer interaction, and in particular relates to a method and system for evaluating interactive experience of smart home appliances. Background Art

[0002] With the rapid development of intelligent technology and the improvement of people's quality of life, the categories of smart home appliances and their intelligent application forms are constantly enriched, causing changes in functional content, interactive forms, etc., making the information characteristics of the interactive interface more and more significant. With the increase in information content, users are prone to information asymmetry and unclear meaning in the process of interacting with the interface. Especially in the soft interface of smart home appliances and related APP interfaces, the information content is rich and highly concentrated, but the presentation space is very limited. If the arrangement and distribution of various information elements do not take into account the user's logical needs, cognitive characteristics, interactive behaviors and habits, characteristic preferences and other human factors, it will increase the user's resistance in the process of use. If users are required to accumulate experience through continuous trial and error, it will inevitably cause them to feel frustrated, greatly reduce the experience, and weaken the product advantages. In addition, in the use environment of home scenes, smart home appliances are not only a simple mechanical combination of technology and function, but also need to pay attention to the emotional interaction of users. Only when users maintain a pleasant mood can the overall user experience be improved. Therefore, the more the interaction of the smart home appliance interface conforms to the user's cognitive, psychological, physiological and behavioral characteristics, the more it can improve the user experience and greatly improve user satisfaction, thereby increasing the user stickiness of the product. Summary of the invention

[0003] The inventors found that in the past, whether the human-computer interaction of home appliances is reasonable or successful was mainly determined by open means, such as interviews, questionnaires, attitude surveys, focus groups, cognitive path methods, heuristic evaluations, etc., or strict behavioral performance evaluations, such as reaction time and error rate. Other researchers have proposed a comprehensive evaluation model for user experience quality, which builds a model from the perspective of human-computer interaction, user perspective, and observer perspective. The 360° indicator system of user experience quality is then used to determine the weight of each factor using the analytic hierarchy process, and finally the fuzzy evaluation method is used to evaluate the user experience quality. However, such methods tend to be subjective evaluations and are greatly affected by individual differences, such as perception, age, education level, appreciation ability, psychology, and differences in different regions, so the accuracy of the results is subject to certain limitations. People are constantly exploring and improving the user interaction experience evaluation methods. At present, many objective evaluation methods have been introduced, which are mostly based on single-mode user data. However, the user-smart home appliance interaction process is a complex task system, which is potentially driven by multiple processes such as emotions, attention, and mental workload. Single-mode information often cannot provide complete knowledge of the phenomenon of interest, that is, it cannot fully reflect the diversified user needs and characteristics, and has a certain one-sidedness. Therefore, how to build a scientific, comprehensive, and objective smart home appliance interaction experience evaluation method and system is a technical problem that needs to be solved urgently.

[0004] In order to comprehensively, scientifically and accurately evaluate the interactive experience of smart home appliances, so as to improve user satisfaction, increase user stickiness of products and enhance product competitiveness, the present invention proposes a method and system for evaluating the interactive experience of smart home appliances.

[0005] A method for evaluating interactive experience of smart home appliances comprises the following steps:

[0006] Collecting physiological signals and behavioral signals of the user, wherein the physiological signals include at least one of electroencephalogram data, electrocardiogram data, and electrodermal conductivity data, and the behavioral signals include at least eye movement data;

[0007] Processing physiological signals and behavioral signals according to preset rules to obtain physiological signal features and behavioral signal features;

[0008] Input the obtained physiological signal features and behavioral signal features into a preset multimodal deep belief network to obtain a multimodal fusion feature vector;

[0009] The obtained multimodal fusion feature vector is annotated with an interactive experience category label using a preset classifier to obtain the final evaluation result. The classifier is trained based on the user experience classification in a pre-built smart home appliance user interactive experience database.

[0010] Furthermore, the smart home appliance interactive experience evaluation method further includes preprocessing the collected physiological signals and behavioral signals of the user, specifically:

[0011] For EEG signals, use filters to perform bandpass filtering and notch processing on the EEG signals, and use independent component analysis to remove eye artifacts; or,

[0012] For ECG signals, perform power frequency notch processing and use wavelet transform denoising method to remove burrs; or,

[0013] For skin electrical signals, use a smoothing filter to remove high-frequency burrs; or,

[0014] For eye movement signals, missing values ​​and outliers caused by the acquisition process are processed.

[0015] Furthermore, for the ECG signal, the shape of the baseline drift is extracted by using a median filter method, and then the extracted baseline is subtracted from the original signal to solve the baseline drift problem.

[0016] Furthermore, for eye movement signals, missing values ​​and outliers caused by the collection process are processed, including: using regression analysis or Bayesian offline regression methods to fill missing values ​​based on reasoning or decision-making methods, and processing outliers through data binning, regression analysis or outlier analysis methods.

[0017] Furthermore, the physiological signal and the behavioral signal are processed according to preset rules to obtain physiological signal features and behavioral signal features, including:

[0018] The key parameters of each physiological signal and behavioral signal are obtained respectively;

[0019] The key parameter data of physiological signals and behavioral signals of multiple users at various user experience levels are collected, and combined with the users' evaluation levels of the experience results, the physiological signal characteristics and behavioral signal characteristics are obtained through variance analysis and correlation analysis respectively.

[0020] Furthermore, the key parameters of each physiological signal and behavioral signal are obtained, including:

[0021] Using wavelet transform to calculate the relative energy value of each rhythm wave of the EEG signal and the wavelet entropy characteristic parameters as the key parameters of the EEG signal; or,

[0022] The differential threshold method and the amplitude threshold method are used to locate the R wave of the QRS complex information of the electrocardiogram, and the RR interval sequence is calculated based on the R wave peak. Then, the RR interval is subjected to mathematical statistical analysis and power spectrum estimation analysis to obtain the time domain characteristic parameters, frequency domain characteristic parameters, and sample entropy as the key parameters of the electrocardiogram signal; or,

[0023] The power spectrum energy value of the skin electrical signal in the effective frequency range is calculated by Fourier transform to serve as the key parameter of the skin electrical signal; or,

[0024] The movement trajectory of the user's eyes when performing tasks is calculated to evaluate the user's attention and mental workload status, which serves as the key parameter of the eye movement signal.

[0025] Furthermore, the obtained physiological signal features and behavioral signal features are input into a preset multimodal deep belief network to obtain a multimodal fusion feature vector, including:

[0026] The obtained physiological signal features and behavioral signal features are input into a preset multi-modal deep belief network to obtain high-level physiological signal features and high-level behavioral signal features respectively;

[0027] The dual-mode deep belief network is used to fuse high-level physiological features and high-level behavioral features to obtain a multi-modal fusion feature vector.

[0028] Furthermore, the obtained multimodal fusion feature vector is labeled with the interactive experience category label using a preset classifier to obtain the final evaluation results, including:

[0029] After standardizing the samples of the obtained multimodal fusion feature vector, the samples are divided into training samples and test samples, wherein the training samples are used to train the classifier and the test samples are used to verify the performance level of the classifier;

[0030] After automatically optimizing the parameters of the training samples, the training samples are trained to obtain a classifier;

[0031] Use the acquired classifier to classify the test samples, obtain the test accuracy, and then evaluate the performance of the established classification model;

[0032] The classification model is used to predict the interactive experience status category of the sample to be tested.

[0033] On the other hand, the present invention also discloses a smart home appliance interactive experience evaluation system, including a signal acquisition module, a signal feature generation module, a multimodal fusion feature vector generation module, and an evaluation module, wherein:

[0034] A signal acquisition module, used to collect physiological signals and behavioral signals of the user, wherein the physiological signals include at least one of EEG data, ECG data, and skin conduction data, and the behavioral signals include at least eye movement data;

[0035] A signal feature generation module is used to process physiological signals and behavioral signals according to preset rules to obtain physiological signal features and behavioral signal features;

[0036] A multimodal fusion feature vector generation module is used to input the obtained physiological signal features and behavioral signal features into a preset multimodal deep belief network to obtain a multimodal fusion feature vector;

[0037] The evaluation module is used to use a preset classifier to annotate the obtained multimodal fusion feature vector with an interactive experience category label to obtain the final evaluation result. The classifier is trained by the user experience classification in the pre-built smart home appliance user interactive experience database.

[0038] Based on the above technical solution, the beneficial effects of the present invention compared with the prior art are as follows:

[0039] The present invention is mainly guided by multidisciplinary cross-theories such as engineering psychology, human factors engineering, cognitive neuroscience, computer science, etc., to collect and analyze multimodal data of users in the process of interactive experience of smart home appliances, involving user data at various levels including sensory layer, cognitive layer, behavioral layer and memory layer, such as eye tracking, skin conductance, EEG, ECG and other data, which can comprehensively quantify the user's overt behavior, internal emotions during the interaction process, cognitive load, attention level and other dimensions, and by jointly analyzing these multimodal heterogeneous data with the help of deep learning methods, the complementary advantages between feature data of different modalities can be fully utilized to achieve a comprehensive evaluation of the interactive experience of smart home appliances, which can improve user experience and satisfaction, thereby increasing user stickiness of the product. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0041] Figure 1 This is a flow chart of a method for evaluating interactive experience of smart home appliances in Embodiment 1 of the present invention;

[0042] Figure 2 It is a schematic diagram of key technical points applied in the method for evaluating interactive experience of smart home appliances in the first embodiment of the present invention;

[0043] Figure 3 This is a flow chart of step S120 in the first embodiment of the present invention;

[0044] Figure 4 This is a schematic diagram of determining the number of decomposition layers based on the frequency division ranges of four rhythmic waves in the EEG signal and the sampling frequency in the first embodiment of the present invention;

[0045] Figure 5 is a schematic diagram of an electrocardiogram in Embodiment 1 of the present invention;

[0046] Figure 6 This is a flow chart of using a differential threshold method and an amplitude threshold method to locate an R wave in Embodiment 1 of the present invention;

[0047] Figure 7 This is a schematic diagram of the R wave check flow in the first embodiment of the present invention;

[0048] Figure 8 This is a flow chart of step S130 in the first embodiment of the present invention;

[0049] Fig. 9 This is a schematic diagram of the structure of a preset deep belief network model in the first embodiment of the present invention;

[0050] Fig.10 This is a flow chart of step S140 in the first embodiment of the present invention;

[0051] Fig.11 This is a schematic diagram of the structure of a smart home appliance interactive experience evaluation system in Embodiment 2 of the present invention. DETAILED DESCRIPTION

[0052] The inventors have found through extensive research that user interaction experience is a multi-layer feedback of cognition, emotion and behavior generated by the interconnection of multi-channel sensory perception interaction. When users interact with smart home appliances, they receive information stimulation from smart home appliances from sensory receivers and convert it into sensory signals through the five senses, that is, the perception process. Users can trigger cognitive behavior by mapping information into short-term memory and directly enter working memory for information processing and storage. Some information beyond the user's cognitive ability needs to be introduced into long-term memory for query and extraction of relevant cognition. This process is accompanied by the user's attention and cognitive load, that is, the load comes from the cognitive difficulty of the information obtained by the user through attention in the use path, the complexity and ambiguity of contact interaction, unreasonable function distribution, cumbersome and tedious operations, excessive stimulation of sensory recognition and excessive interference of "Hick's Law" during use. In addition, the process of generating information feedback and corresponding behavior is accompanied by the user's emotions, which are mainly divided into positive and negative emotions. The former is mainly manifested as pleasure, residence, immersion, etc., and the latter is manifested as resistance, wandering, retreat or escape, etc. The inventors believe that the user's overt behavior, internal emotions during the interaction, cognitive load, attention level and other states should be comprehensively quantified from various levels, including the sensory layer, cognitive layer, behavioral layer and memory layer, and these data should be jointly analyzed to give full play to the complementary advantages of different data and achieve a more comprehensive, scientific and accurate comprehensive evaluation of the interactive experience of smart home appliances.

[0053] In order to better understand the above technical solution, the above technical solution will be described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0054] Embodiment 1

[0055] A smart home appliance interactive experience evaluation method, combined with Figure 1 , Figure 2 As shown, steps S110-S140 are included, specifically:

[0056] Step S110, collecting the user's physiological signals and behavioral signals.

[0057] The physiological signal may be at least one of electroencephalogram (EEG), electrocardiogram (ECG), electrodermal conductivity (GSR), and electromyographic (EMG), and the behavioral signal may include at least eye movement (EM), lip movement, and the like. The electroencephalogram (EEG) data may reflect the state of the central nervous system, and the electrocardiogram (ECG), electrodermal conductivity (GSR), and electromyographic (EMG) data. The electroencephalogram (EEG) data may reflect the state of the central nervous system, and the electrocardiogram (ECG) data, electrodermal conductivity (GSR), and electromyographic (EMG) data may reflect the state of the peripheral nervous system. In order to facilitate the introduction of the specific technical solutions of the embodiments of the present invention, in the present embodiment, the physiological signals include electroencephalogram (EEG), electrocardiogram (ECG), and electrodermal conductivity (GSR), and the behavioral signal includes eye movement (EM).

[0058] In practical applications, it is necessary to first design user experience test tasks, analyze the specific situations in which users use specific smart home appliances, product functional features, and interaction touchpoints, extract typical task profiles and key interaction nodes, and design test tasks in accordance with the requirements of psychophysical experimental design. The tasks should fully cover factors such as attention, load, and emotion involved in users' sensory cognitive levels at all levels.

[0059] Then, recruit user subjects, screen them according to the target user group of smart products and the physiological conditions of the users themselves, and provide guidance on experimental tasks. Since this evaluation method and test tasks have high requirements for subjects, when recruiting subjects, in addition to being the target users of smart home appliances, they also need to be strictly screened according to their physiological conditions and task experience. In addition, it is necessary to impose certain restrictions on the state of the subjects before the formal experiment, such as normal sleep and diet, and prohibition of drinking stimulating beverages. Provide guidance on experimental tasks for the screened subjects, and explain the experimental process and precautions in detail.

[0060] Carry out formal user experience testing tasks to collect users' physiological and behavioral signals. Physiological signals can include EEG data, ECG data, and skin conduction data, and behavioral signals can include eye movement data. When collecting physiological signals during the formal experiment, certain requirements must be placed on the experimental environment, such as selecting an environment with good electromagnetic shielding and no noise.

[0061] Since the physiological signal itself is weak, it is very susceptible to various interferences during the acquisition process, such as baseline drift caused by electrode failure, movement, and breathing, power frequency interference, etc. Preprocessing is one of the essential links, which can remove the noise generated by the physiological signal during the acquisition process and ensure the accuracy of the user status analysis of the physiological signal in the later stage. Due to the different characteristics of different signals, including the frequency range of the signal itself, such as EEG signals are mainly concentrated between 0.5 and 35 Hz, the useful frequency range of ECG signals is concentrated between 0.01 and 100 Hz, and the skin electrical signal is mainly concentrated between 0 and 2 Hz, so the preprocessing process of different signals is also different.

[0062] Specifically, for EEG signals, in addition to baseline drift, power frequency interference, electromyography and other interferences, electrooculography artifacts are the main interference, so the EEG signal denoising is divided into two aspects. On the one hand, based on the frequency range of normal brain signals, the EEG signal is subjected to 0.5-40Hz bandpass filtering and 50Hz notch processing using a filter to remove electromyography, slow voltage drift artifacts and power frequency interference; on the other hand, since the frequency bands of electrooculography and electroencephalography overlap, the independent component analysis (ICA) method is used here to remove electrooculography artifacts. The main idea of ​​independent component analysis ICA is to classify mixed signals composed of several relatively independent signal sources, and then implement non-Gaussian measurement on the results. If the non-Gaussianity is the largest, the mixed signal is considered to have achieved optimal separation. Since the generation of artifacts such as electrooculography and brain activity is a different physiological process, that is, it meets the requirements of independent signal sources, the independent component analysis ICA method can be used to remove electrooculography artifacts.

[0063] There are three main aspects for ECG signal denoising: on the one hand, ECG signal contains a large amount of 50Hz and 100Hz power frequency noise, so we first filter it out by designing 50Hz and 100Hz power frequency notch filters; on the other hand, after removing the power frequency interference, the signal still contains many glitches caused by myoelectric interference and other reasons. Considering that the traditional low-pass filter is easy to weaken the ECG R wave while removing the glitches, the wavelet transform denoising method is generally used to remove the glitches; sometimes there will be serious baseline drift in the ECG signal, and the median filter method is generally used to extract the shape of the baseline drift, and then the extracted baseline is subtracted from the original signal.

[0064] For denoising of the electrodermal signal, a smoothing filter (such as a 3rd-order Butterworth filter) can be used to remove high-frequency glitches in the electrodermal signal. Since the change of the electrodermal signal is generally slow, the cutoff frequency of the low-pass filter is usually set to 0.3 Hz.

[0065] The preprocessing of eye movement signals mainly focuses on dealing with missing values ​​and outliers caused by the acquisition process. Missing values ​​are filled in by regression, Bayesian and other methods based on reasoning or decision-making methods, and outliers are processed by binning, regression or outlier analysis.

[0066] Step S120, processing the physiological signal and the behavioral signal according to preset rules to obtain physiological signal features and behavioral signal features.

[0067] like Figure 3 As shown, this step includes sub-steps S121 and S122:

[0068] Step S121, obtaining key parameters of each physiological signal and behavioral signal respectively.

[0069] (1) The relative energy value and wavelet entropy characteristic parameters of the EEG signal in each rhythmic wave are calculated using wavelet transform as the key parameters of the EEG signal.

[0070] Since EEG signals have random non-stationary characteristics, traditional time domain and frequency domain methods cannot fully and accurately reflect the EEG characteristics of different emotions. Time-frequency analysis can make up for the shortcomings of the above two methods. Common time-frequency analysis methods include short-time Fourier transform, wavelet transform and wavelet packet transform. Among them, although short-time Fourier transform can reflect the information of time and frequency, it is difficult to synchronously process the high-frequency and low-frequency parts of the signal due to the fixed resolution. For wavelet transform, its time-frequency rectangular window will change with the change of frequency, which is more suitable for analyzing EEG signals containing multiple frequency components. Wavelet packet transform is proposed on the basis of wavelet transform. Compared with wavelet transform, its analysis is more refined and can effectively increase the time-frequency resolution. Wavelet entropy is an entropy value calculated from the signal sequence after wavelet decomposition. It is a nonlinear dynamic method and is often used to analyze short-time non-stationary signals. Entropy value can be used to measure the disorder degree of signal energy in subspace, which can accurately reflect signals containing multiple frequency components. Therefore, it is often used to estimate the complexity and uncertainty of signals. Therefore, wavelet transform is used here to calculate the relative energy values ​​of each rhythmic wave of EEG signal δ, θ, α, and β as well as the wavelet entropy characteristic parameters. These key parameters have also been proven in previous studies to reflect a person's mental workload, emotional state, and attention state.

[0071] For ease of understanding, the specific implementation method of wavelet transform to calculate the key parameters of EEG signals is introduced.

[0072] First, when performing wavelet packet decomposition, the number of decomposition layers is determined according to the frequency division range of the four rhythmic waves in the EEG signal and the sampling frequency. Assuming that a 6-layer decomposition is used, Figure 4As shown. Secondly, the selection of wavelet function is very important, but there is no good pre-judgment standard yet. The quality of the wavelet function used is often judged by comparing the actual results obtained after wavelet transform with the theoretical results. Combined with the experience of previous researchers, and because the Daubechies wavelet system has excellent properties and a canonical orthogonal wavelet with a support set, it can effectively analyze EEG signals. Therefore, the db wavelet function can be used.

[0073] After extracting the four rhythmic waves using wavelet packet technology, it is necessary to further study their characteristics. Since the strength of the signal can be represented by the wavelet packet decomposition coefficient, the wavelet packet decomposition coefficients contained in each rhythmic signal can be squared separately and then summed to obtain the energy of each frequency band. Assume that the wavelet decomposition coefficient corresponding to the i-th wavelet node in the j-th layer is The energy of a specific frequency band can be calculated using the following formula:

[0074]

[0075] The wavelet entropy analysis method based on EEG rhythm obtains wavelet entropy based on the relative energy of each rhythm wave. The wavelet entropy obtained in this way can reflect the distribution of the energy of the four rhythm waves in the total energy. The logarithmic energy entropy in the wavelet packet entropy is used here, that is, the wavelet entropy is defined by combining the energy distribution of the wavelet packet decomposition coefficients with the information entropy. The algorithm formula is as follows:

[0076] SE=-∑RE m oeLh m (2)

[0077] Among them, m represents the four waves of δ, θ, α, and β, and RE m is the relative energy value of each rhythmic wave.

[0078] (2) The differential threshold method and amplitude threshold method are used to locate the R wave of the QRS complex information of the electrocardiogram. The RR interval sequence is calculated based on the R wave peak. Then, the RR interval is subjected to mathematical statistical analysis and power spectrum estimation analysis to obtain the time domain characteristic parameters, frequency domain characteristic parameters, and sample entropy as the key parameters of the electrocardiogram signal.

[0079] Figure 5 The schematic diagram of the electrocardiogram is shown in Figure 1. The extraction of QRS complex information is the key part of the extraction of electrocardiogram feature information. The position of the R wave must be determined before subsequent parameter measurement. Considering the real-time and accuracy of the system, the differential threshold method and the amplitude threshold method are used to locate the R wave. The process is shown in Figure 1. Figure 6As shown. Among them, X(n) is the voltage value of the nth sampling point in the ECG signal segment. According to the characteristic of the largest wave slope, the ECG amplitude array is differentially calculated to generate the differential sequence Z(n): Z(n) = X(n+1)-X(n-1). The differential value is judged point by point. If the differential value of three consecutive points is greater than K1 (R wave rising slope threshold), then from 100ms after this point forward, it is judged whether there is such a point n, so that X(n) is greater than Xh (amplitude threshold), Z(n), Z(n-1) are less than K2 (R wave falling slope threshold). If it exists, the point with the largest amplitude between these two points is the R wave peak. In order to avoid interference from large T waves, a relative refractory period (200ms) is skipped after the wave is detected, and the next detection is performed. At the same time, in order to prevent missed detection, the RR interval should be judged for each R wave detected. If the RR interval is greater than the sampling period, there may be missed detection and recheck is required. The backcheck starts from the last R wave skipping the refractory period, and checks point by point whether there is such a point: the amplitude of this point is greater than the threshold, and higher than the amplitude of the previous and next points. If such a point is found, it is marked as the R wave peak, and returns to the main program to identify the next R wave. The R wave backcheck process is as follows Figure 7 As shown in the figure, the RR interval sequence is calculated according to the R wave peak, and then the RR interval is subjected to mathematical statistical analysis and power spectrum estimation analysis to obtain time domain characteristics (RR interval mean and STD, coefficient of variation between RRs, STD of continuous differences between RRs, etc.) and frequency domain characteristic parameters (LF, HF, LF / HF ratio, etc.), sample entropy, etc.

[0080] (3) The power spectrum energy value of the skin electrodermal signal in the effective frequency band is calculated using Fourier transform as the key parameter of the skin electrodermal signal.

[0081] For electrodermal signals, since their effective frequency is between 0.02 and 0.2 Hz, the power spectrum energy value of this frequency band is calculated mainly through Fourier transform to reflect the emotional state of the user during the experience.

[0082] (4) Calculate the movement trajectory of the user's eyes when performing tasks to evaluate the user's attention and mental workload status, which serves as the key parameter of the eye movement signal.

[0083] For eye movement features, the user's attention and mental workload are evaluated mainly by calculating the movement trajectory of the user's eyes when performing tasks. The main raw features calculated are pupil diameter, fixation time, scanning trajectory, eye saccade distance / speed / frequency, blinking frequency and duration.

[0084] Step S122, collecting key parameter data of physiological signals and key parameter data of behavioral signals of multiple users at multiple user experience levels, combining the user's evaluation level of the experience results, and obtaining physiological signal characteristics and behavioral signal characteristics through variance analysis and correlation analysis respectively.

[0085] The key parameter data of physiological signals and behavioral signals of multiple users at various user experience levels are collected, combined with the users' subjective evaluation levels of the experience results (excellent, good, average, poor, bad), and variance analysis and correlation analysis are used to obtain EEG, EKG, EGG and eye movement feature indicators that can be effectively used to characterize the cognitive load, attention and emotional processes in the user experience process. These multimodal physiological signal and behavioral signal feature indicators constitute a multidimensional feature sample set.

[0086] Step S130, input the obtained physiological signal features and behavioral signal features into a preset multimodal deep belief network to obtain a multimodal fusion feature vector.

[0087] like Figure 8 As shown, specifically, this step includes sub-steps S131-S132:

[0088] Step S131, input the obtained physiological signal features and behavioral signal features into a preset multi-modal deep belief network to obtain high-level physiological signal features and high-level behavioral signal features, respectively.

[0089] Step S132, using a dual-mode deep belief network to fuse high-level physiological features and high-level behavioral features into modal features to obtain a multi-modal fusion feature vector.

[0090] Since multimodal physiological and behavioral data have information redundancy, missing, heterogeneity and other problems in the collection process, that is, simple feature splicing often does not establish the interactive connection between the modes, ignores the semantic differences between different modes and has large noise interference, and each modal information is not equal to the state of the final user experience, so establishing a multimodal fusion mechanism becomes the key to multimodal user experience state recognition. In order to solve the problem of low efficiency and accuracy of the recognition process in the existing fusion technology, the present invention adopts a user experience state evaluation method based on a multimodal deep belief network fusion mechanism. The advantage of this fusion mechanism is that it can use the relationship between heterogeneous data for collaborative learning, and when the signal source is small, the multimodal features are fused to obtain stable recognition performance. Multimodal collaborative learning utilizes the relationship between different modes, fully complements the information between different features of each mode, eliminates the interference between different features of the same mode, removes redundancy, and reduces the mutual interference caused by irrelevant features between different modes when multimodal fusion is performed. In this way, the features of each modal signal will be more effectively combined, and a better recognition effect can be obtained through a deep network with fewer hidden layers. Specifically, in this model, the three physiological electrical signals and eye movement feature indicators all have independent hidden layers to obtain unified high-level representation features for each modality, remove redundant features in a single modality, and fuse features at different levels in the modality into a unified, standardized high-level feature. In the fusion stage of each modality feature, an attention mechanism is introduced to automatically learn the attention coefficient for each modality to indicate the importance of the modality to emotion recognition. Subsequently, each modality feature is multiplied by the attention coefficient to adaptively select key features. For the modality with missing signals, a graph convolutional network is used to model the complementary and mutually exclusive relationships between the features of each modality, and a prediction mechanism based on a generative adversarial network is introduced to complete the features of the missing signal. Finally, a bimodal deep belief network model (BDBN) is used to obtain deep multimodal features from the unified high-level features of the two modalities of physiological signals and behavioral signals. The structure of the deep belief network model in the embodiment of the present invention is referenced. Fig. 9 shown.

[0091] Step S140, using a preset classifier to annotate the obtained multimodal fusion feature vector with an interactive experience category label to obtain a final evaluation result, wherein the classifier is trained by user experience classification in a pre-constructed smart home appliance user interactive experience database.

[0092] like Fig.10 As shown, this step mainly uses a support vector machine (SVM) to train and classify the deep multimodal features, including sub-steps S141-S144, where:

[0093] Step S141 , after standardizing the obtained samples of the multimodal fusion feature vector, divide them into training samples and test samples, wherein the training samples are used to train the classifier, and the test samples are used to verify the performance level of the classifier.

[0094] In practical applications, we must first standardize the samples of the multimodal fusion feature vector. In this example, the sample set is divided into two parts of data, training samples and test samples, in a ratio of 7:3. The training samples are used to train the classifier, while the test samples are used to evaluate the performance of the classifier. The performance levels are divided into five categories: excellent, good, average, poor, and bad, represented by numbers 1 to 5 respectively.

[0095] Step S142, after automatically optimizing the parameters of the training samples, the training samples are trained to obtain a classifier.

[0096] After automatically optimizing the parameters of the training samples to obtain the best parameters, the training samples are trained to obtain the classifier. Automatic parameter optimization can be performed using random search or grid search. Random search is to sample hyperparameter combinations in the search space and then select the best hyperparameter combination in the sampled combination. Grid search is a brute force search that tries all hyperparameter combinations in a given hyperparameter search space and finally searches for the best hyperparameter combination.

[0097] Step S143, classify the test samples using the acquired classifier to obtain the test accuracy, and then evaluate the performance of the established classification model.

[0098] Step S144: using the classification model to predict the interactive experience state category of the sample to be tested.

[0099] This classification model is used to predict the interactive experience status category (excellent, good, average, poor, bad) of the samples to be tested. The evaluation results can be used to predict and evaluate the design level of the human-machine interface of smart home appliances, or to evaluate and optimize multiple interface solutions to provide the best solution.

[0100] Compared with the traditional user experience evaluation method, the present invention collects and analyzes the physiological data and behavioral data of users in the process of interactive experience with smart home appliances in real time, adopts the learning characteristics of deep belief networks, and fuses the various modal features of physiological signals and behavioral signals respectively. It has the characteristics of data perception and feature fusion, and solves the problem that multi-source heterogeneous data cannot be effectively fused, resulting in feature redundancy and insufficient key features. The present invention not only reduces subjectivity, but also comprehensively quantifies the state of various dimensions such as the user's explicit behavior, internal emotions in the interactive process, cognitive load, and attention level from various levels including the sensory layer, cognitive layer, behavioral layer, and memory layer. At the same time, with the help of deep learning methods, these multi-modal heterogeneous data are jointly analyzed, giving full play to the complementary advantages between different modal feature data, and realizing a comprehensive evaluation of the interactive experience of smart home appliances. Compared with the results obtained by traditional methods and technologies, it is more comprehensive, scientific, and accurate.

[0101] The present invention can be applied to the user's attention, mental workload and emotional cognition during the interaction with smart home appliances, and then predict and evaluate the design level of the human-machine interface of smart home appliances, or evaluate and optimize multiple interface solutions to provide the best solution. The present invention can improve user experience and satisfaction, thereby increasing the user stickiness of the product.

[0102] Embodiment 2

[0103] In other embodiments, the present invention also discloses a smart home appliance interactive experience evaluation system, such as Fig.11 As shown, the smart home appliance interactive experience evaluation system includes a signal acquisition module 10, a signal feature generation module 20, a multimodal fusion feature vector generation module 30, and an evaluation module 40, wherein:

[0104] The signal acquisition module 10 is used to collect physiological signals and behavioral signals of the user, wherein the physiological signal includes at least one of EEG data, ECG data, and electrodermal data, and the behavioral signal includes at least eye movement data.

[0105] The signal acquisition module 10 is also used to pre-process the collected physiological signals and behavioral signals, specifically:

[0106] (1) The EEG signal acquisition unit uses a filter to perform bandpass filtering and notch processing on the EEG signal, and uses independent component analysis to remove eye artifacts.

[0107] (2) The ECG signal acquisition unit performs power frequency notch processing and uses wavelet transform noise reduction method to remove burrs.

[0108] (3) The skin electrical signal acquisition unit uses a smoothing filter to remove high-frequency glitches.

[0109] (4) The eye movement signal acquisition unit processes the missing values ​​and abnormal values ​​caused by the acquisition process.

[0110] The signal feature generation module 20 is used to process the physiological signal and the behavioral signal according to preset rules to obtain physiological signal features and behavioral signal features.

[0111] Specifically, we first obtain the key parameters of each physiological signal and behavioral signal:

[0112] (1) The relative energy value and wavelet entropy characteristic parameters of the EEG signal in each rhythmic wave are calculated using wavelet transform as the key parameters of the EEG signal.

[0113] (2) The differential threshold method and amplitude threshold method are used to locate the R wave of the QRS complex information of the electrocardiogram. The RR interval sequence is calculated based on the R wave peak. Then, the RR interval is subjected to mathematical statistical analysis and power spectrum estimation analysis to obtain the time domain characteristic parameters, frequency domain characteristic parameters, and sample entropy as the key parameters of the electrocardiogram signal.

[0114] (3) The power spectrum energy value of the skin electrodermal signal in the effective frequency band is calculated using Fourier transform as the key parameter of the skin electrodermal signal.

[0115] (4) Calculate the movement trajectory of the user's eyes when performing tasks to evaluate the user's attention and mental workload status, which serves as the key parameter of the eye movement signal.

[0116] Then, we collect key parameter data of physiological signals and key parameter data of behavioral signals of multiple users at various user experience levels, and combine them with the users' evaluation levels of the experience results to obtain physiological signal characteristics and behavioral signal characteristics through variance analysis and correlation analysis respectively.

[0117] The multimodal fusion feature vector generation module 30 is used to input the obtained physiological signal features and behavioral signal features into a preset multimodal deep belief network to obtain a multimodal fusion feature vector.

[0118] Specifically: First, the obtained physiological signal features and behavioral signal features are input into the preset multimodal deep belief network to obtain the unified high-level representation features of each modality, remove the redundant features in a single modality, and fuse the features of different levels in the modality into a unified and standardized high-level feature. In the fusion stage of each modality feature, the attention mechanism is introduced to automatically learn the attention coefficient for each modality to indicate the importance of the modality for emotion recognition. Then, the features of each modality are multiplied by the attention coefficient to adaptively select the key features. For the modality with missing signals, the graph convolutional network is used to model the complementary and mutually exclusive relationship between the features of each modality, and a prediction mechanism based on the adversarial generative network is introduced to complete the features of the missing signal. Finally, the bimodal deep belief network model (BDBN) is used to obtain deep multimodal features from the unified high-level features of the two modalities of physiological signals and behavioral signals.

[0119] The evaluation module 40 is used to use a preset classifier to annotate the obtained multimodal fusion feature vector with an interactive experience category label to obtain a final evaluation result. The classifier is trained by the user experience classification in a pre-built smart home appliance user interactive experience database.

[0120] Specifically, first, the samples of the obtained multimodal fusion feature vector are standardized and divided into training samples and test samples. In this example, the sample set is divided into two parts of data, training samples and test samples, according to a ratio of 7:3. The training samples are used to train the classifier, and the test samples are used to verify the performance of the classifier. They are divided into five categories: excellent, good, general, poor, and bad, represented by numbers 1 to 5 respectively. Then, after the training samples are automatically optimized, the training samples are trained to obtain the classifier; the test samples are then classified using the obtained classifier to obtain the test accuracy, and then the performance of the established classification model is evaluated. Finally, the classification model can be used to predict the interactive experience status category (excellent, good, general, poor, and poor) of the sample to be tested. The evaluation results can be used as a prediction and evaluation of the design level of the human-machine interface of smart home appliances, or to evaluate and select multiple interface solutions to provide the best solution.

[0121] The specific working methods of each module of the smart home appliance interactive experience evaluation system in the embodiment of the present invention can be referred to in Example 1 and will not be repeated here.

[0122] In the above detailed description, various features are grouped together in a single embodiment to simplify the disclosure. This method of disclosure should not be interpreted as reflecting an intention that the embodiments of the claimed subject matter require more features than are clearly stated in each claim. On the contrary, as reflected in the appended claims, the invention is in a state of having less than all the features of the disclosed individual embodiments. Therefore, the appended claims are hereby expressly incorporated into the detailed description, with each claim standing on its own as a separate preferred embodiment of the invention.

[0123] The above description includes examples of one or more embodiments. Of course, it is not possible to describe all possible combinations of components or methods for the purpose of describing the above embodiments, but it should be recognized by those skilled in the art that the various embodiments may be further combined and arranged. Therefore, the embodiments described herein are intended to cover all such changes, modifications and variations that fall within the scope of protection of the appended claims. In addition, with respect to the term "comprising" used in the specification or claims, the word is encompassed in a manner similar to the term "including", as explained by "including," used as a transitional word in the claims. In addition, any term "or" used in the specification of the claims is intended to mean "non-exclusive or".

Claims

1. A method for evaluating the interactive experience of smart home appliances, characterized in that: The following steps are involved: Collecting physiological signals and behavioral signals of the user, wherein the physiological signals include at least one of electroencephalogram data, electrocardiogram data, and electrodermal conductivity data, and the behavioral signals include at least eye movement data; Processing physiological signals and behavioral signals according to preset rules to obtain physiological signal features and behavioral signal features; Input the obtained physiological signal features and behavioral signal features into a preset multimodal deep belief network to obtain a multimodal fusion feature vector; The obtained multimodal fusion feature vector is annotated with an interactive experience category label using a preset classifier to obtain the final evaluation result. The classifier is trained based on the user experience classification in a pre-built smart home appliance user interactive experience database.

2. The evaluation method according to claim 1, characterized in that: The smart home appliance interactive experience evaluation method further includes preprocessing the collected physiological signals and behavioral signals of the user, specifically: For EEG signals, use filters to perform bandpass filtering and notch processing on the EEG signals, and use independent component analysis to remove eye artifacts; or, For ECG signals, power frequency notch processing is performed and wavelet transform denoising method is used to remove burrs; or, For skin electrical signals, use a smoothing filter to remove high-frequency burrs; or, For eye movement signals, missing values ​​and outliers caused by the acquisition process are processed.

3. The evaluation method according to claim 2, characterized in that: For the ECG signal, the shape of the baseline drift is extracted by using the median filter method, and then the extracted baseline is subtracted from the original signal to solve the baseline drift problem.

4. The evaluation method according to claim 2, characterized in that: For eye movement signals, the missing values ​​and outliers caused by the collection process are processed, including: using regression analysis or Bayesian offline regression methods to fill in missing values ​​based on reasoning or decision-making methods, and processing outliers through data binning, regression analysis or outlier analysis.

5. The evaluation method according to claim 1, characterized in that: The physiological signals and behavioral signals are processed according to preset rules to obtain physiological signal features and behavioral signal features, including: The key parameters of each physiological signal and behavioral signal are obtained respectively; The key parameter data of physiological signals and behavioral signals of multiple users at various user experience levels are collected, and combined with the users' evaluation levels of the experience results, the physiological signal characteristics and behavioral signal characteristics are obtained through variance analysis and correlation analysis respectively.

6. The evaluation method according to claim 5, characterized in that: The key parameters of each physiological signal and behavioral signal are obtained respectively, including: Using wavelet transform to calculate the relative energy value of each rhythm wave of the EEG signal and the wavelet entropy characteristic parameters as the key parameters of the EEG signal; or, The differential threshold method and the amplitude threshold method are used to locate the R wave of the QRS complex information of the electrocardiogram, and the RR interval sequence is calculated based on the R wave peak. Then, the RR interval is subjected to mathematical statistical analysis and power spectrum estimation analysis to obtain the time domain characteristic parameters, frequency domain characteristic parameters, and sample entropy as the key parameters of the electrocardiogram signal; or, The power spectrum energy value of the skin electrical signal in the effective frequency range is calculated by Fourier transform to serve as the key parameter of the skin electrical signal; or, The movement trajectory of the user's eyes when performing tasks is calculated to evaluate the user's attention and mental workload status, which serves as the key parameter of the eye movement signal.

7. The evaluation method according to claim 1, characterized in that: The obtained physiological signal features and behavioral signal features are input into the preset multimodal deep belief network to obtain a multimodal fusion feature vector, including: The obtained physiological signal features and behavioral signal features are input into a preset multi-modal deep belief network to obtain high-level physiological signal features and high-level behavioral signal features respectively; The dual-mode deep belief network is used to fuse the high-level physiological features and high-level behavioral features to obtain a multi-modal fusion feature vector.

8. The evaluation method according to claim 1, characterized in that: Use the preset classifier to annotate the obtained multimodal fusion feature vector with the interactive experience category label to obtain the final evaluation results, including: After standardizing the samples of the obtained multimodal fusion feature vector, the samples are divided into training samples and test samples, wherein the training samples are used to train the classifier and the test samples are used to verify the performance level of the classifier; After automatically optimizing the parameters of the training samples, the training samples are trained to obtain a classifier; Use the acquired classifier to classify the test samples, obtain the test accuracy, and then evaluate the performance of the established classification model; The classification model is used to predict the interactive experience status category of the sample to be tested.

9. A smart home appliance interactive experience evaluation system, characterized in that: include: Signal acquisition module, signal feature generation module, multimodal fusion feature vector generation module, and evaluation module, including: The signal acquisition module is used to collect physiological signals and behavioral signals of the user, wherein the physiological signals include at least one of EEG data, ECG data, and electrodermal data, and the behavioral signals include at least eye movement data; The signal feature generation module is used to process the physiological signal and the behavioral signal according to preset rules to obtain physiological signal features and behavioral signal features; The multimodal fusion feature vector generation module is used to input the obtained physiological signal features and behavioral signal features into a preset multimodal deep belief network to obtain a multimodal fusion feature vector; The evaluation module is used to use a preset classifier to annotate the obtained multimodal fusion feature vector with an interactive experience category label to obtain a final evaluation result. The classifier is trained by the user experience classification situation in a pre-built smart home appliance user interactive experience database.

10. The evaluation system according to claim 9, characterized in that: The signal acquisition module is also used to pre-process the collected physiological signals and behavioral signals, specifically: For EEG signals, use filters to perform bandpass filtering and notch processing on the EEG signals, and use independent component analysis to remove eye artifacts; or, For ECG signals, power frequency notch processing is performed and wavelet transform denoising method is used to remove burrs; or, For skin electrical signals, use a smoothing filter to remove high-frequency burrs; or, For eye movement signals, missing values ​​and outliers caused by the acquisition process are processed.

11. The evaluation system according to claim 9, characterized in that: The signal feature generation module is specifically used for: The key parameters of each physiological signal and behavioral signal are obtained respectively; the key parameter data of physiological signals and key parameter data of behavioral signals of multiple users at various user experience levels are collected, and the physiological signal characteristics and behavioral signal characteristics are obtained respectively through variance analysis and correlation analysis in combination with the user's evaluation level of the experience results.

12. The evaluation system according to claim 9, characterized in that: The multimodal fusion feature vector generation module is specifically used for: The obtained physiological signal features and behavioral signal features are input into a preset multi-modal deep belief network to obtain high-level physiological signal features and high-level behavioral signal features respectively; The dual-mode deep belief network is used to fuse the high-level physiological features and high-level behavioral features to obtain a multi-modal fusion feature vector.

13. The evaluation system according to claim 9, characterized in that: The evaluation module is specifically used for: After standardizing the samples of the obtained multimodal fusion feature vector, the samples are divided into training samples and test samples, wherein the training samples are used to train the classifier and the test samples are used to verify the performance level of the classifier; After automatically optimizing the parameters of the training samples, the training samples are trained to obtain a classifier; Use the acquired classifier to classify the test samples, obtain the test accuracy, and then evaluate the performance of the established classification model; The classification model is used to predict the interactive experience status category of the sample to be tested.

Images