User experience evaluation method, device, system and equipment based on multi-modal data

Abstract

The invention discloses a user experience evaluation method, device, system and equipment based on multi-modal data, and relates to the technical field of artificial intelligence. The user experience evaluation method based on the multi-modal data comprises the steps that the multi-modal data of a user are acquired through a virtual reality head-mounted device, and the multi-modal data comprise at least two of electroencephalogram data, physiological data and eye movement data; performing feature extraction on the multi-modal data to obtain multi-modal features; and performing fusion processing on the multi-modal features by using a pre-trained evaluation model, and evaluating the experience information of the user according to fusion features obtained by the fusion processing to obtain an evaluation result. According to the method, the user experience is evaluated through the multi-modal data, the problems that a subjective evaluation method is high in subjectivity and poor in stability can be solved, continuous and objective evaluation of the XR user experience is achieved, meanwhile, the defect that single-modal data evaluation cannot comprehensively reflect the user experience state is overcome, and the accuracy and reliability of evaluation are improved.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a user experience evaluation method, device, system and equipment based on multimodal data. Background Art

[0002] In extended reality (XR) technology, the evaluation of user experience is crucial. At present, the methods for evaluating user experience include: subjective evaluation method and single modality data evaluation method. Among them, the subjective evaluation method is to conduct a subjective evaluation of the user experience based on the user's self-perception. It is highly subjective, lacks stability and accuracy, and is difficult to continuously evaluate changes in user experience in real time; the single modality data evaluation method, such as using EEG data or physiological data to evaluate user experience, cannot fully and accurately reflect the complex state of XR user experience. In addition, the two evaluation methods lack unified standards, making it difficult to effectively compare and promote them in different research and application scenarios. Summary of the invention

[0003] The present invention aims to solve at least one of the technical problems in the related art to a certain extent. To this end, the purpose of the present invention is to propose a user experience evaluation method, device, system and equipment based on multimodal data to solve the problem of strong subjectivity and poor stability of subjective evaluation methods, realize continuous and objective evaluation of XR user experience, and make up for the deficiency that single modality data evaluation cannot fully reflect the user experience status, so as to improve the accuracy and reliability of the evaluation.

[0004] In the first aspect, an embodiment of the present invention proposes a user experience evaluation method based on multimodal data, comprising: obtaining multimodal data of a user through a virtual reality head-mounted device, wherein the multimodal data includes at least two of EEG data, physiological data, and eye movement data; performing feature extraction on the multimodal data to obtain multimodal features; using a pre-trained evaluation model to fuse the multimodal features, and evaluating the user's experience information based on the fused features obtained by the fusion processing to obtain an evaluation result.

[0005] According to one embodiment of the present invention, the physiological data includes heart rate data, galvanic skin response data, surface electromyography data and respiratory data; the feature extraction of the multimodal data is performed separately to obtain multimodal features, including: extracting time-related potential components and / or time-frequency features from the EEG data; extracting heart rate variability indicators from the heart rate data, extracting the rate of change of galvanic skin level and / or galvanic skin response amplitude from the galvanic skin response data, extracting frequency features and / or root mean square values ​​of muscle activity from the surface electromyography data, and extracting breathing depth changes and / or breathing frequency variability from the respiratory data;

[0006] A gaze point feature and / or a glance feature is extracted from the eye movement data.

[0007] According to one embodiment of the present invention, before feature extraction is performed on the multimodal data respectively, the multimodal data are also preprocessed respectively, including: performing at least one of filtering processing, bad track detection, interpolation processing, re-referencing, segmentation processing and artifact removal on the EEG data; performing smoothing processing and / or mathematical statistics on the heart rate data, performing baseline correction and / or filtering processing on the galvanic skin response data, performing at least one of filtering processing, rectification processing, smoothing processing and normalization on the surface electromyography data, performing smoothing processing and / or outlier removal on the respiratory data; and performing at least one of calibration, drift correction, and invalid data removal on the eye movement data.

[0008] According to one embodiment of the present invention, the evaluation model includes a feature fusion module and an experience prediction module; the using of a pre-trained evaluation model to fuse the multimodal features and evaluating the user's experience information based on the fusion features obtained by the fusion processing includes: using a weighting function in a pre-trained feature fusion module to fuse the multimodal features and using a pre-trained experience prediction module to evaluate the user's experience information based on the fusion features.

[0009] According to an embodiment of the present invention, the evaluation result includes at least one of the immersion level, the emotion change situation, and the stress level, and the method further includes: quantifying and displaying the evaluation result.

[0010] According to an embodiment of the present invention, the method further comprises: generating contribution information of each of the modal features to the evaluation result.

[0011] According to an embodiment of the present invention, the contribution information is generated according to the weight corresponding to each of the modal features in the weighting function.

[0012] According to an embodiment of the present invention, the virtual reality head mounted device includes an EEG acquisition electrode, a heart rate sensor, a skin electrode sensor, a surface electromyography sensor, a respiratory monitoring device, and an eye tracking device; the EEG data is acquired by the EEG acquisition electrode, the acquisition site of the EEG acquisition electrode includes at least one of the frontal lobe, the parietal lobe, and the occipital lobe of the user, and the acquisition frequency is greater than or equal to 512 Hz; the heart rate data is acquired by the heart rate sensor, the heart rate sensor is attached to the earlobe of the user, and the acquisition frequency is greater than or equal to 1 Hz; the skin electrode response data is acquired by the skin electrode sensor, the skin electrode sensor is attached to the cheek of the user, and the acquisition frequency ranges from 5 Hz to 15 Hz; the surface electromyography data is acquired by the surface electromyography sensor, the surface electromyography sensor is attached to the cheek of the user, and the acquisition frequency is greater than or equal to 1000 Hz; the respiratory data is acquired by the respiratory monitoring device, and the sampling frequency of the respiratory monitoring device ranges from 1 Hz to 5 Hz; the eye movement data is acquired by the eye tracking device, and the acquisition rate of the eye tracking device is greater than or equal to 60 Hz.

[0013] In the second aspect, an embodiment of the present invention proposes a user experience evaluation device based on multimodal data, including: an acquisition unit, used to obtain a user's multimodal data through a virtual reality head-mounted device, wherein the multimodal data includes at least two of EEG data, physiological data and eye movement data; an extraction unit, used to perform feature extraction on the multimodal data respectively to obtain multimodal features; an evaluation unit, used to use a pre-trained evaluation model to fuse the multimodal features, and evaluate the user's experience information based on the fused features obtained by the fusion processing to obtain an evaluation result.

[0014] In a third aspect, an embodiment of the present invention proposes an electronic device, including a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the steps of the user experience evaluation method based on multimodal data described in the first aspect are implemented.

[0015] In a fourth aspect, an embodiment of the present invention proposes a user experience evaluation system based on multimodal data, comprising: a virtual reality head-mounted device; and a user experience evaluation device based on multimodal data according to the second aspect above, or an electronic device according to the third aspect above.

[0016] In a fifth aspect, an embodiment of the present invention proposes a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the steps of the user experience evaluation method based on multimodal data described in the first aspect are implemented.

[0017] The user experience evaluation method, device, system and equipment based on multimodal data of the embodiment of the present invention first obtains the user's multimodal data through a virtual reality head-mounted device, and the multimodal data includes at least two of EEG data, physiological data and eye movement data; then, feature extraction is performed on the multimodal data to obtain multimodal features; then, the multimodal features are fused using a pre-trained evaluation model, and the user's experience information is evaluated based on the fused features obtained by the fusion process to obtain an evaluation result. Therefore, by using multimodal data to evaluate user experience, the problems of strong subjectivity and poor stability of subjective evaluation methods can be solved, and continuous and objective evaluation of XR user experience can be achieved. At the same time, it makes up for the deficiency that single modality data evaluation cannot fully reflect the user experience status, and improves the accuracy and reliability of the evaluation. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 is a flow chart of a method for evaluating user experience based on multimodal data according to an embodiment of the present invention;

[0019] Figure 2 is a flowchart of a method for evaluating user experience based on multimodal data according to another embodiment of the present invention;

[0020] Figure 3 is a flowchart of a method for evaluating user experience based on multimodal data according to another embodiment of the present invention;

[0021] Figure 4 is a structural block diagram of a user experience evaluation device based on multimodal data according to an embodiment of the present invention;

[0022] Figure 5 is a structural block diagram of an electronic device according to an embodiment of the present invention;

[0023] Figure 6 is a structural block diagram of a user experience evaluation system based on multimodal data according to an embodiment of the present invention;

[0024] Figure 7 It is a structural block diagram of a user experience evaluation system based on multimodal data according to another embodiment of the present invention. DETAILED DESCRIPTION

[0025] Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present invention, and should not be construed as limiting the present invention.

[0026] The following describes a method, apparatus, system, and device for evaluating user experience based on multimodal data according to an embodiment of the present invention with reference to the accompanying drawings.

[0027] Figure 1 4 is a flowchart of a method for evaluating user experience based on multimodal data according to an embodiment of the present invention.

[0028] like Figure 1 As shown in Figure 2, the user experience evaluation method based on multimodal data includes:

[0029] S11, obtaining multimodal data of the user through a virtual reality head mounted device, where the multimodal data includes at least two of EEG data, physiological data, and eye movement data.

[0030] In some embodiments of the present invention, the physiological data may include at least one of heart rate data, galvanic skin response data, surface electromyography data, and respiration data. The virtual reality head-mounted device may include at least two of EEG acquisition electrodes, physiological data acquisition components, and eye tracking devices, and the physiological data acquisition components include at least one of a heart rate sensor, a galvanic skin sensor, a surface electromyography sensor, and a respiration monitoring device.

[0031] Among them, EEG data can be collected through EEG collection electrodes, and the collection sites of the EEG collection electrodes include at least one of the user's frontal lobe, parietal lobe, and occipital lobe, and the collection frequency is greater than or equal to 512Hz; heart rate data is collected through a heart rate sensor, and the heart rate sensor is attached to the user's earlobe, and the collection frequency is greater than or equal to 1Hz; skin electrode response data is collected through a skin electrode sensor, and the skin electrode sensor is attached to the user's cheek, and the collection frequency ranges from 5Hz to 15Hz; surface electromyography data is collected through a surface electromyography sensor, and the surface muscle sensor is attached to the user's cheek, and the collection frequency is greater than or equal to 1000Hz; breathing data is collected through a breathing monitoring device, and the sampling frequency range of the breathing monitoring device is 1Hz to 5Hz; eye movement data is collected through an eye tracking device, and the collection rate of the eye tracking device is greater than or equal to 60Hz.

[0032] Specifically, the EEG acquisition electrodes can be multi-channel, such as 32 channels or higher. The multi-channel EEG acquisition electrodes can be distributed on the user's scalp according to the international 10-20 system, and collect EEG signals at a high sampling rate (such as 512Hz or higher). The collection sites include the frontal lobe, parietal lobe, occipital lobe and other brain areas related to cognition, perception and attention. It should be noted that before the XR experience (such as gaming, education, training, medical rehabilitation and other fields) begins, the EEG acquisition electrodes can perform a short baseline signal acquisition for comparison and correction during subsequent data processing.

[0033] The heart rate sensor can be attached to the earlobe to collect heart rate data in real time. The sampling frequency can be set according to the accuracy of the device, generally not less than 1Hz; the skin electricity sensor can be attached to the cheek to measure the changes in skin electricity (i.e. skin conductivity), which can reflect the degree of activation of the user's autonomic nervous system. The sampling frequency can be set to about 10Hz; the surface electromyography sensor can be attached to the cheek, and the sampling frequency can be set to 1000Hz or higher to accurately capture changes in muscle electrical activity. Before the XR experience begins, the surface electromyography sensor can collect a baseline electromyography signal for subsequent data correction and comparison; the breathing detection device (such as a breathing belt or other breathing monitoring equipment) can measure the breathing frequency, and the sampling frequency is between 1 and 5Hz to reflect the user's body relaxation or tension.

[0034] Eye tracking devices are used to record the user's eye movement data in the XR experience, including gaze point location, gaze duration, scanning path and other data. The sampling frequency is not less than 60Hz to ensure that subtle eye movements can be accurately captured.

[0035] By integrating the acquisition devices of EEG data, physiological data, eye movement data, etc. into the virtual reality head-mounted device, it can be convenient to carry and use in the XR scene, and it can also facilitate the synchronous acquisition of multimodal data during the XR scene experience.

[0036] S12, extracting features from the multimodal data to obtain multimodal features.

[0037] In some embodiments, if the multimodal data includes EEG data, physiological data and eye movement data, and the physiological data includes heart rate data, galvanic skin response data, surface electromyography data and respiratory data, then feature extraction is performed on the multimodal data respectively to obtain multimodal features, which may include: extracting time-related potential components and / or time-frequency features from the EEG data; extracting heart rate variability indicators from the heart rate data, extracting the rate of change of skin electrode levels and / or the amplitude of galvanic skin response from the galvanic skin response data, extracting frequency features and / or the root mean square value of muscle activity from the surface electromyography data, extracting breathing depth changes and / or respiratory frequency variability from the respiratory data; extracting gaze point features and / or scanning features from the eye movement data.

[0038] Specifically, for EEG data, event-related potential (ERP) components can be extracted, such as N1 (100-200ms), mismatch negativity (MMN, 150-200ms), early slow wave (SW1, 400-650ms) and late slow wave (SW2, 650-900ms), and the average amplitude, latency and other parameters of these ERP components can be calculated. These parameters can reflect the brain's early perception of stimuli, attention allocation and cognitive processing. Time-frequency analysis methods (such as wavelet transform) can also be used to extract the time-frequency characteristics of EEG signals, such as the energy distribution of different frequency bands (such as theta, alpha, beta, gamma and other frequency bands), to reveal the changes in neural oscillation activity of the brain under different cognitive states.

[0039] For the heart rate data in the physiological data, the heart rate variability (HRV) index is extracted from the heart rate data. It can be a time domain index (such as the standard deviation of all sinus beat RR intervals (SDNN), the root mean square of the difference between adjacent normal cardiac cycles (RMSSD), etc.) and a frequency domain index (such as LF / HF, that is, the ratio of low-frequency power to high-frequency power, etc.). The HRV index can reflect the balance state of the autonomic nervous system and is related to the regulation of emotions and attention in immersion.

[0040] For the electrodermal response data in the physiological data, features such as the rate of change of electrodermal level and the amplitude of electrodermal response can be extracted. These features are closely related to the user's emotional arousal level.

[0041] For the surface electromyography data in physiological data, the root mean square (RMS) value of muscle activity can be extracted. The RMS value can reflect the overall intensity of muscle activity. In the XR experience, different levels of immersion may lead to different degrees of muscle tension, thus affecting the RMS value. The frequency characteristics of surface electromyography data can also be extracted, such as the median frequency (MF) and the mean power frequency (MPF). These frequency characteristics can reflect the degree of muscle fatigue and changes in neuromuscular control strategies. Continuous muscle activity or tension related to immersion may cause changes in these frequency characteristics.

[0042] For the breathing data in the physiological data, features such as changes in breathing depth and breathing frequency variability can be extracted to evaluate the user's degree of physical relaxation or tension.

[0043] For eye movement data, gaze point features can be extracted, such as average gaze time, first gaze time, gaze point distribution density, etc. These features can reflect the user's attention and interest in different areas or objects in the XR scene. Glance features can also be extracted, including average glance speed, glance amplitude, and the complexity of the glance path (such as the tortuosity of the glance path). Glance features can reveal the user's visual exploration strategy and attention transfer in the XR scene.

[0044] It should be understood that if the multimodal data includes any two of EEG data, physiological data and eye movement data, and the physiological data includes one, two or three of heart rate data, galvanic skin response data, surface electromyography data and respiratory data, then only the corresponding modal data needs to be feature extracted.

[0045] S13, using a pre-trained evaluation model to perform fusion processing on the multimodal features, and evaluating the user's experience information based on the fusion features obtained by the fusion processing to obtain an evaluation result.

[0046] In some embodiments of the present invention, the evaluation model includes a feature fusion module and an experience prediction module; multimodal features are fused using a pre-trained evaluation model, and the user's experience information is evaluated based on the fusion features obtained by the fusion process, including: multimodal features are fused using a weighting function in a pre-trained feature fusion module, and the user's experience information is evaluated based on the fusion features using a pre-trained experience prediction module.

[0047] Specifically, the feature fusion module can use a multimodal data fusion algorithm, such as a weighted fusion method or a principal component analysis (PCA) method, to fuse the features of the EEG data, physiological data, and eye movement data, and use the fused feature vector as the input of the experience prediction module. Taking the weighted fusion method as an example, when the evaluation model is trained, different weights can be assigned according to the importance and relevance of each modal data in reflecting the sense of immersion. The initial weight can be set according to prior knowledge or experience. For example, when there is no more information, the initial weights of the three can be set to 1 / 3, or set according to the preliminary estimate of the importance of each modal data. When the evaluation model is subsequently trained, the weights can be optimized, using optimization algorithms such as the gradient descent method, the stochastic gradient descent method, etc., and the weights of each modal data can be adjusted according to the value of the loss function. The direction of adjustment is to gradually reduce the value of the loss function, that is, to improve the accuracy of the evaluation model prediction. In each iteration, the gradient of the loss function to each weight is calculated, and then the weight is updated according to the direction and size of the gradient. For example, in the gradient descent method, the weight update formula is: Among them, wi is the weight of the i-th modal data, are the weights of the i-th modal data before and after updating, α is the learning rate (the step size for controlling weight update), and L is the loss function. The experience prediction module may include machine learning models such as support vector machines (SVMs), neural networks (such as multilayer perceptrons), deep learning models (such as convolutional neural networks), decision trees, Bayesian classifiers, etc.

[0048] When training the evaluation model, the evaluation model can be trained using training data (multimodal data) with labeled evaluation results (which can be immersion levels, such as high, medium, and low immersion, which can be labeled by combining subjective evaluation and expert evaluation), and the evaluation model parameters (including the aforementioned weights, machine learning model parameters, etc.) can be adjusted so that the evaluation model can accurately predict the user's immersion level. In the process of training the evaluation model, methods such as cross-validation can be used to prevent overfitting and improve the generalization ability of the evaluation model. Methods such as feature engineering can also be used to improve the performance of the evaluation model.

[0049] In some embodiments of the present invention, before feature extraction is performed on the multimodal data, the multimodal data are preprocessed separately, including: performing at least one of filtering, bad track detection, interpolation, re-referencing, segmentation and artifact removal on the EEG data; performing smoothing and / or mathematical statistics on the heart rate data, performing baseline correction and / or filtering on the galvanic skin response data, performing at least one of filtering, rectification, smoothing and normalization on the surface electromyography data, performing smoothing and / or outlier removal on the respiratory data; and performing at least one of calibration, drift correction and invalid data removal on the eye movement data.

[0050] Specifically, for EEG data, filtering is first performed, and a bandpass filter with a passband range of 0.1 to 40 Hz can be used to remove high-frequency noise and DC drift; then bad track detection and interpolation are performed, and channel statistical information (such as channel kurtosis, etc.) is used to automatically identify and remove bad tracks, and then the bad track data is repaired through interpolation methods (such as spherical interpolation methods). EEG data can also be re-referenced, using an average reference or a specific brain area (such as the mastoid) as a reference electrode to improve the accuracy and comparability of the data. EEG data can also be segmented, such as segmenting EEG data according to stimulation events (such as XR scene switching, task start, etc.), and the segment length can be determined according to the ERP component of the study, generally 100-900ms. Artifacts can also be removed from EEG data, including: first removing segments containing large-amplitude artifacts based on voltage thresholds (such as ±500μV), and then further removing artifacts to ensure data quality.

[0051] For the heart rate data in the physiological data, smoothing processing can be performed first to remove abnormal values ​​(such as instantaneous large fluctuations due to poor sensor contact, etc.), and then statistical indicators such as the mean value and standard deviation of the heart rate are calculated to reflect the stability and change trend of the heart rate.

[0052] For the skin electrical response data in the physiological data, baseline correction can be performed, and the skin electrical level before the XR experience begins is used as the baseline to calculate the skin electrical change value. The skin electrical response data can also be filtered to remove high-frequency noise interference to improve the reliability of the data.

[0053] For the surface electromyography data in physiological data, a bandpass filter can be used, and the bandpass range is set according to the characteristics of the electromyography signal, such as 20-500Hz, to remove low-frequency motion artifacts and high-frequency noise; the filtered surface electromyography data can then be rectified (such as taking the absolute value), and then smoothed (such as using the moving average method or low-pass filtering) to obtain the envelope of the surface electromyography data to reflect the intensity changes of muscle activity. The amplitude of the surface electromyography data can also be normalized (such as maximum voluntary contraction (MVC) normalization or other appropriate normalization methods) to facilitate data comparison between different muscle groups and different individuals.

[0054] For the respiratory frequency data in the physiological data, similar smoothing processing and outlier removal can be performed to calculate parameters such as the respiratory frequency variation range and the average respiratory cycle.

[0055] For eye movement data, the fixation point data in the eye movement data can be calibrated and drift corrected to ensure the accuracy of the fixation point position; invalid fixation point data can also be eliminated, such as abnormal fixation points caused by blinking or violent head movement. Eye movement parameters such as fixation time ratio (the ratio of fixation time to total experience time), average scanning speed, scanning path length, etc. can also be calculated for subsequent analysis.

[0056] It should be understood that if the multimodal data includes any two of EEG data, physiological data and eye movement data, and the physiological data includes one, two or three of heart rate data, galvanic skin response data, surface electromyography data and respiratory data, then only the corresponding modal data needs to be preprocessed.

[0057] In some embodiments of the present invention, the evaluation result includes at least one of the immersion level, the emotion change situation, and the stress level.

[0058] In this embodiment, if Figure 2 As shown, the user experience evaluation method based on multimodal data also includes:

[0059] S14, quantitatively displaying the evaluation results.

[0060] Specifically, the multimodal data collected and preprocessed in real time is input into the trained evaluation model, and the evaluation model outputs the user's current evaluation results, and the evaluation results can be displayed quantitatively. Taking the evaluation result as the level of immersion as an example, it can be presented in the form of a numerical value or level (such as 1-5 levels of immersion) and displayed visually. The evaluation results can also be presented to users or XR developers in the form of intuitive charts (such as line charts showing the trend of immersion over time, and bar charts comparing the immersion levels, emotional changes, stress, etc. of different users or different XR scenarios).

[0061] In some embodiments of the present invention, Figure 3 As shown, the user experience evaluation method based on multimodal data also includes:

[0062] S15, generating contribution information of each modal feature to the evaluation result.

[0063] In an embodiment of the present invention, after obtaining the evaluation results, a detailed data analysis report can also be provided, including information such as the contribution of each modal data feature to the immersive feeling, so that developers can optimize the XR experience in a targeted manner.

[0064] In some examples, contribution information is generated based on the weights corresponding to each modal feature in a weighting function.

[0065] For example, in the weighting function, the weights corresponding to the EEG features, physiological features, and eye movement features are a, b, and c, respectively. Then the contribution information (i.e., contribution value) of the EEG features, physiological features, and eye movement features to the evaluation results are a / (a+b+c), b / (a+b+c), and c / (a+b+c), respectively. If the weights corresponding to the heart rate features, galvanic skin response features, surface electromyography features, and breathing features are b1, b2, b3, and b4, respectively, b1+b2+b3+b4, then the contribution information of the heart rate features, galvanic skin response features, surface electromyography features, and breathing features to the evaluation results are b1 / (a+b+c), b2 / (a+b+c), b3 / (a+b+c), and b4 / (a+b+c), respectively.

[0066] In some examples, user experience evaluation can be performed based on each modal feature to obtain respective evaluation results, which are then compared for similarity with the evaluation results obtained based on multimodal data to obtain contribution information of each modal feature. The higher the similarity, the greater the contribution value of the corresponding modal feature.

[0067] In this example, as an implementation method, each evaluation result can be quantified first, such as high immersion is 100%, low immersion is 0, and medium immersion is 50%. The similarity between the two evaluation results can be directly represented by the product of the two corresponding quantized values. The contribution information of a modal feature can be the ratio between the product corresponding to the modal feature and the sum of the products corresponding to each modal feature. Taking multimodal data including EEG data, physiological data, and eye movement data as an example, if the evaluation result obtained based on the multimodal data is high immersion, and the evaluation results obtained by each modal feature are also high immersion, then the contribution information of each modal feature is 100%*100% / (100%*100%+100%*100%+100%*100%)=1 / 3; if the evaluation result obtained based on the multimodal data is high immersion, the evaluation result obtained by the EEG feature is high immersion, and the evaluation results obtained by the physiological features and the eye movement features are both medium immersion, then the contribution information of the EEG features, physiological features, and eye movement features are 100% respectively. *100% / (100%*100%+5%*100%+50%*100%)=1 / 2, 50%*100% / (100%*100%+5%*100%+50%*100%)=1 / 4, 50%*100% / (100%*100%+5%*100%+50%*100%)=1 / 4; if the evaluation result obtained based on multimodal data is high immersion, the evaluation result obtained by EEG characteristics is high immersion, and the evaluation results obtained by physiological characteristics and eye movement characteristics are both low immersion, then the contribution information of EEG characteristics, physiological characteristics and eye movement characteristics are 1, 0, and 0 respectively.

[0068] In summary, the user experience evaluation method based on multimodal data in the embodiment of the present invention can achieve the following effects:

[0069] 1) Portability: Using a virtual reality headset with integrated multi-modal data collection, it is possible to simultaneously collect skin electrical response data, heart rate data, surface electromyography data, breathing data, EEG data, eye movement data, etc. during the virtual scene experience;

[0070] 2) Improve the accuracy of evaluation: Comprehensive multimodal data such as EEG, physiology and eye movement can fully reflect the user's status in the XR experience from multiple angles, avoiding the one-sidedness of a single data source, thereby more accurately evaluating the user experience;

[0071] 3) Realize real-time continuous evaluation: The collection of multimodal data can be carried out continuously during the XR experience process, monitoring the dynamic changes of the user experience in real time, providing timely feedback to XR content creators and developers, and helping to optimize XR experience content and interaction design;

[0072] 4) Enhanced objectivity: The reliance on subjective evaluation is reduced, and the evaluation method based on physiological and behavioral data is more objective, and the evaluation results of different users and different experimental conditions are more comparable;

[0073] 5) Wide applicability: This method can be applied to various types of XR experiences, including games, education, training, medical rehabilitation and other fields, providing unified technical support for XR user experience evaluation in different application scenarios.

[0074] Figure 4 It is a structural block diagram of a user experience evaluation device based on multimodal data according to an embodiment of the present invention.

[0075] like Figure 4 As shown, the user experience evaluation device 400 based on multimodal data includes: an acquisition unit 410 , an extraction unit 420 and an evaluation unit 430 .

[0076] Among them, the acquisition unit 410 is used to obtain the user's multimodal data through a virtual reality head-mounted device, and the multimodal data includes at least two of EEG data, physiological data and eye movement data; the extraction unit 420 is used to perform feature extraction on the multimodal data respectively to obtain multimodal features; the evaluation unit 430 is used to use a pre-trained evaluation model to fuse the multimodal features, and evaluate the user's experience information based on the fused features obtained by the fusion processing to obtain an evaluation result.

[0077] In some embodiments of the present invention, the physiological data includes heart rate data, galvanic skin response data, surface electromyography data and respiratory data; the multimodal data are subjected to feature extraction to obtain multimodal features, including: extracting time-related potential components and / or time-frequency features from EEG data; extracting heart rate variability indicators from heart rate data, extracting the rate of change of galvanic skin level and / or galvanic skin response amplitude from galvanic skin response data, extracting frequency features and / or root mean square value of muscle activity from surface electromyography data, extracting respiratory depth changes and / or respiratory frequency variability from respiratory data; extracting gaze point features and / or scanning features from eye movement data.

[0078] In some embodiments of the present invention, before feature extraction is performed on the multimodal data, the multimodal data are preprocessed separately, including: performing at least one of filtering, bad track detection, interpolation, re-referencing, segmentation and artifact removal on the EEG data; performing smoothing and / or mathematical statistics on the heart rate data, performing baseline correction and / or filtering on the galvanic skin response data, performing at least one of filtering, rectification, smoothing and normalization on the surface electromyography data, performing smoothing and / or outlier removal on the respiratory data; and performing at least one of calibration, drift correction and invalid data removal on the eye movement data.

[0079] In some embodiments of the present invention, the evaluation model includes a feature fusion module and an experience prediction module; multimodal features are fused using a pre-trained evaluation model, and the user's experience information is evaluated based on the fusion features obtained by the fusion process, including: multimodal features are fused using a weighting function in a pre-trained feature fusion module, and the user's experience information is evaluated based on the fusion features using a pre-trained experience prediction module.

[0080] In some embodiments of the present invention, the evaluation result includes at least one of the immersion level, the emotion change, and the stress level. The user experience evaluation device 400 based on multimodal data also includes a display unit, which is used to quantitatively display the evaluation result.

[0081] In some embodiments of the present invention, the user experience evaluation device 400 based on multimodal data further includes a generation unit, which is used to generate contribution information of each modal feature to the evaluation result.

[0082] In some embodiments of the present invention, contribution information is generated according to the weights corresponding to each modal feature in a weighting function.

[0083] In some embodiments of the present invention, the virtual reality head-mounted device includes EEG acquisition electrodes, a heart rate sensor, a skin electrode sensor, a surface electromyography sensor, a respiratory monitoring device, and an eye tracking device; the EEG data is collected by the EEG acquisition electrodes, and the collection site of the EEG acquisition electrodes includes at least one of the user's frontal lobe, parietal lobe, and occipital lobe, and the collection frequency is greater than or equal to 512 Hz; the heart rate data is collected by the heart rate sensor, the heart rate sensor is attached to the user's earlobe, and the collection frequency is greater than or equal to 1 Hz; the skin electrode response data is collected by the skin electrode sensor, the skin electrode sensor is attached to the user's cheek, and the collection frequency ranges from 5 Hz to 15 Hz; the surface electromyography data is collected by the surface electromyography sensor, the surface electromyography sensor is attached to the user's cheek, and the collection frequency is greater than or equal to 1000 Hz; the respiratory data is collected by the respiratory monitoring device, and the sampling frequency range of the respiratory monitoring device is 1 Hz to 5 Hz; the eye movement data is collected by the eye tracking device, and the collection rate of the eye tracking device is greater than or equal to 60 Hz.

[0084] It should be noted that, for other specific implementations of the user experience evaluation device based on multimodal data in an embodiment of the present invention, reference may be made to the specific implementations of the user experience evaluation method based on multimodal data in the above-mentioned embodiment.

[0085] Based on the user experience evaluation method based on multimodal data of the above embodiment, the present invention proposes a computer-readable storage medium.

[0086] In this embodiment, a computer program is stored on a computer-readable storage medium. When the computer program is executed by a processor, the user experience evaluation method based on multimodal data of the above embodiment is implemented.

[0087] Based on the user experience evaluation method based on multimodal data in the above embodiment, the present invention also proposes an electronic device.

[0088] like Figure 5 As shown, the electronic device 500 includes: a processor 501 and a memory 503. The processor 501 and the memory 503 are connected, such as through a bus 502. Optionally, the electronic device 500 may also include a transceiver 504. It should be noted that in actual applications, the transceiver 504 is not limited to one, and the structure of the electronic device 500 does not constitute a limitation on the embodiments of the present invention.

[0089] Processor 501 may be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It may implement or execute various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of the present invention. Processor 501 may also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, and the like.

[0090] The bus 502 may include a path to transmit information between the above components. The bus 502 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus. The bus 502 may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 5 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0091] The memory 503 is used to store a computer program corresponding to the user experience evaluation method based on multimodal data of the above embodiment of the present invention, and the computer program is controlled and executed by the processor 501. The processor 501 is used to execute the computer program stored in the memory 503 to implement the content shown in the above method embodiment.

[0092] The electronic device 500 includes, but is not limited to, fixed terminals such as digital TVs and desktop computers, and mobile terminals such as smart phones and tablet computers. Figure 5 The electronic device 500 shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0093] Figure 6 It is a structural block diagram of a user experience evaluation system based on multimodal data according to an embodiment of the present invention.

[0094] like Figure 6 As shown, the user experience evaluation system 600 based on multimodal data includes: a virtual reality head mounted device 610 and the user experience evaluation device 400 based on multimodal data of the above embodiment.

[0095] Figure 7 It is a structural block diagram of a user experience evaluation system based on multimodal data according to another embodiment of the present invention.

[0096] like Figure 7 As shown, the user experience evaluation system 600 based on multimodal data includes: a virtual reality head mounted device 610 and the electronic device 500 of the above embodiment.

[0097] It should be noted that the logical steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be specifically implemented in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in combination with these instruction execution systems, devices or apparatuses. For the purposes of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in combination with these instruction execution systems, devices or apparatuses. More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection portion with one or more wirings (electronic device), a portable computer disk box (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium and then editing, interpreting or processing in other suitable ways if necessary, and then stored in a computer memory.

[0098] It should be understood that the various parts of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, a plurality of steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0099] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "examples", "specific examples", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0100] In the description of the present invention, it is to be understood that the terms “center”, “longitudinal”, “lateral”, “length”, “width”, “thickness”, “up”, “down”, “front”, “back”, “left”, “right”, “vertical”, “horizontal”, “top”, “bottom”, “inside”, “outside”, “clockwise”, “counterclockwise”, “axial”, “radial”, “circumferential”, etc., indicating orientations or positional relationships based on the orientations or positional relationships shown in the accompanying drawings, are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the referred device or element must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be understood as limiting the present invention.

[0101] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In the description of the present invention, the meaning of "plurality" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0102] In the present invention, unless otherwise clearly specified and limited, the terms "installed", "connected", "connected", "fixed" and the like should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediate medium, it can be the internal connection of two elements or the interaction relationship between two elements, unless otherwise clearly defined. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0103] In the present invention, unless otherwise clearly specified and limited, a first feature being "above" or "below" a second feature may mean that the first and second features are in direct contact, or the first and second features are in indirect contact through an intermediate medium. Moreover, a first feature being "above", "above" or "above" a second feature may mean that the first feature is directly above or obliquely above the second feature, or simply means that the first feature is higher in level than the second feature. A first feature being "below", "below" or "below" a second feature may mean that the first feature is directly below or obliquely below the second feature, or simply means that the first feature is lower in level than the second feature.

[0104] Although the embodiments of the present invention have been shown and described above, it is to be understood that the above embodiments are exemplary and are not to be construed as limitations of the present invention. A person skilled in the art may change, modify, replace and vary the above embodiments within the scope of the present invention.

Claims

1. A user experience evaluation method based on multimodal data, characterized in that: include: Acquiring multimodal data of the user through a virtual reality head mounted device, wherein the multimodal data includes at least two of EEG data, physiological data, and eye movement data; Extracting features from the multimodal data to obtain multimodal features; The multimodal features are fused using a pre-trained evaluation model, and the user's experience information is evaluated based on the fused features obtained by the fusion process to obtain an evaluation result.

2. The user experience evaluation method based on multimodal data according to claim 1 is characterized in that: The physiological data includes heart rate data, galvanic skin response data, surface electromyography data and respiration data; The extracting features from the multimodal data to obtain multimodal features includes: Extracting time-related potential components and / or time-frequency features from the EEG data; Extracting a heart rate variability index from the heart rate data, extracting a skin electrical level change rate and / or a skin electrical response amplitude from the skin electrical response data, extracting a frequency feature and / or a root mean square value of muscle activity from the surface electromyography data, and extracting a breathing depth change and / or a breathing frequency variability from the breathing data; A gaze point feature and / or a glance feature is extracted from the eye movement data.

3. The user experience evaluation method based on multimodal data according to claim 2 is characterized in that: Before extracting features from the multimodal data, the multimodal data are preprocessed, including: Perform at least one of filtering, bad track detection, interpolation, re-referencing, segmentation and artifact removal on the EEG data; Performing smoothing and / or mathematical statistics on the heart rate data, performing baseline correction and / or filtering on the galvanic skin response data, performing at least one of filtering, rectification, smoothing, and normalization on the surface electromyography data, and performing smoothing and / or outlier removal on the respiratory data; The eye movement data is subjected to at least one of calibration, drift correction, and invalid data elimination.

4. The user experience evaluation method based on multimodal data according to claim 1 is characterized in that: The evaluation model includes a feature fusion module and an experience prediction module; the multimodal features are fused using the pre-trained evaluation model, and the user's experience information is evaluated based on the fusion features obtained by the fusion process, including: The multimodal features are fused using a weighting function in a pre-trained feature fusion module, and the user's experience information is evaluated based on the fused features using a pre-trained experience prediction module.

5. The user experience evaluation method based on multimodal data according to claim 1 is characterized in that: The evaluation result includes at least one of the immersion level, the emotion change situation, and the stress level. The method further includes: The evaluation results are quantified and displayed.

6. The user experience evaluation method based on multimodal data according to claim 4 is characterized in that: The method further comprises: Generate contribution information of each of the modal features to the evaluation result.

7. The user experience evaluation method based on multimodal data according to claim 6 is characterized in that: The contribution information is generated according to the weight corresponding to each of the modal features in the weighting function.

8. The user experience evaluation method based on multimodal data according to claim 2 is characterized in that: The virtual reality head-mounted device includes an electroencephalogram (EEG) acquisition electrode, a heart rate sensor, a skin charge sensor, a surface electromyography sensor, a respiratory monitoring device, and an eye tracking device; The EEG data is collected by the EEG collection electrodes, the collection site of the EEG collection electrodes includes at least one of the frontal lobe, parietal lobe, and occipital lobe of the user, and the collection frequency is greater than or equal to 512 Hz; The heart rate data is collected by the heart rate sensor, which is attached to the earlobe of the user, and the collection frequency is greater than or equal to 1 Hz; The galvanic skin response data is collected by the galvanic skin sensor, which is attached to the cheek of the user, and the collection frequency range is 5Hz to 15Hz; The surface electromyography data is collected by the surface electromyography sensor, the surface electromyography sensor is attached to the cheek of the user, and the collection frequency is greater than or equal to 1000 Hz; The respiratory data is collected by the respiratory monitoring device, and the sampling frequency of the respiratory monitoring device is in the range of 1 Hz to 5 Hz; The eye movement data is collected by the eye tracking device, and the collection rate of the eye tracking device is greater than or equal to 60 Hz.

9. A user experience evaluation device based on multimodal data, characterized in that: include: An acquisition unit, configured to acquire multimodal data of a user through a virtual reality head mounted device, wherein the multimodal data includes at least two of EEG data, physiological data, and eye movement data; An extraction unit, used for performing feature extraction on the multimodal data to obtain multimodal features; The evaluation unit is used to use a pre-trained evaluation model to perform fusion processing on the multimodal features, and evaluate the user's experience information according to the fusion features obtained by the fusion processing to obtain an evaluation result.

10. An electronic device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the user experience evaluation method based on multimodal data described in any one of claims 1 to 8 are implemented.

11. A user experience evaluation system based on multimodal data, characterized in that: include: Virtual reality headsets; as well as The user experience evaluation device based on multimodal data according to claim 9, or the electronic device according to claim 10.

12. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the user experience evaluation method based on multimodal data described in any one of claims 1 to 8 are implemented.

Images