Emotion regulation method and system based on VR device

Abstract

The application belongs to the technical field of VR and provides a mood adjustment method and system based on a VR device, which comprises the following steps: device wearing, mood signal data set collection, feature extraction and feature fusion, mood effect value and mood activation degree calculation, temporary mood adjustment scheme matching, switching command acquisition, temporary mood adjustment scheme updating, mood adjustment effect score table collection and model and strategy table optimization. The application increases the analysis dimension of the model and strengthens the prediction effect of the model by collecting mood information of multiple users. The application improves the user experience and increases the acceptance of users by collecting mood adjustment methods and switching commands. The application improves the generalization and reliability of the model by collecting mood adjustment effect score tables and optimizing the model and strategy table, and the application realizes the improvement of the mood adjustment effect by continuously accepting feedback information.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of VR, in particular to a mood regulation method and system based on a VR device. BACKGROUND

[0002] In daily life and work, people will encounter various pressures, with the rapid development of modern society, people are facing more and more pressure, people often need an effective way to relieve stress, regulate emotions. Common mood regulation methods include: shouting, crying, exercise, etc. In recent years, virtual reality technology has gradually integrated into the lives of the public, and VR devices provide wearers with a virtual three-dimensional environment that gives them a sense of being there, which has a greater effect on user mood regulation.

[0003] Traditional mood regulation methods are limited by people's own personality, physical condition, and the influence of less self-selected direction, resulting in short-term or poor results. Some existing technologies also propose methods and devices for mood regulation through VR, but these methods all have the problems of inaccurate mood analysis, slow or no feedback adjustment, and single adjustment method, resulting in an unobvious mood regulation effect. SUMMARY

[0004] In order to overcome the shortcomings of the prior art, the purpose of the present application is to provide a mood regulation method and system based on a VR device to improve the effect of user mood regulation.

[0005] To achieve the above-mentioned purpose, the present application provides the following solutions:

[0006] A mood regulation method based on a VR device, comprising:

[0007] According to a predetermined procedure, a user wears a VR mood regulation device, and records the user's selected mood regulation method; the mood regulation method includes: music regulation, video regulation, and game regulation;

[0008] In the VR virtual environment, the user is guided to explain personal circumstances and collect the user's emotional signal data set; the emotional signal data set includes: an electroencephalogram signal data set, an electrocardiogram signal data set, a body node motion signal data set, an eye image data set, a facial expression image data set, a tone signal data set, a volume signal data set, and a keyword detection signal data set;

[0009] Feature extraction and feature fusion are performed on the emotional signal data set to obtain a multi-dimensional fused emotional feature;

[0010] The multi-dimensional fused emotional feature is input into a pre-trained emotional determination model for emotional analysis to obtain an emotional efficiency value and an emotional activation degree;

[0011] matching the emotion effect value and the emotion activation degree with the pre-arranged emotion regulation scheme library to obtain a temporary emotion regulation scheme;

[0012] updating the temporary emotion regulation scheme to the pre-worn VR headset to regulate the user's emotion, and collecting the user's switching command during the emotion regulation process;

[0013] when the switching command is received, re-matching the emotion effect value and the emotion activation degree with the pre-arranged emotion regulation scheme library to update the temporary emotion regulation scheme, and returning to the step of updating the temporary emotion regulation scheme to the pre-worn VR headset to regulate the user's emotion, and collecting the user's switching command during the emotion regulation process;

[0014] after the emotion regulation is completed, recording the user's emotion regulation effect score table, and when the number of the emotion regulation effect score table reaches a feedback threshold, using the emotion regulation effect score table to optimize the emotion determination model and the emotion regulation scheme library.

[0015] Preferably, the user is guided to explain personal situation in the VR virtual environment and the user's emotion signal data set is collected, including:

[0016] detecting the working state of the VR headset, motion capture device, EEG device, ECG device, short-focus camera, high-definition camera and microphone worn by the user, and if the working state is not normal, prompting the user to check the device wearing or report the device abnormality to the staff;

[0017] collecting the user's head EEG signal through the electrodes arranged on the user's scalp by the EEG device to obtain the EEG signal data set;

[0018] collecting the user's ECG signal through the ECG device worn on the user's wrist side to obtain the ECG signal data set;

[0019] collecting the user's motion signal through the motion sensors arranged on the user's back of the hand and feet by the motion capture device to obtain the body node motion signal data set;

[0020] collecting the user's eye image through the short-focus camera arranged inside the VR headset to obtain the eye image data set;

[0021] collecting the user's facial image through the high-definition camera placed opposite to the user's head to obtain the facial expression image data set;

[0022] The voice information of the user is collected through the microphone arranged at the side of the VR head-mounted device, and the tonal signal data set and the volume signal data set are obtained;

[0023] The keyword detection signal data set is obtained through matching of the voice information and a preset keyword database.

[0024] Preferably, feature extraction and feature fusion are performed on the emotion signal data set to obtain multi-dimensional fused emotion features, including:

[0025] Statistical features, activity features, mobility features, complexity features, differential entropy features, power spectral density features and wavelet transform features are extracted from the electroencephalogram signal data set to obtain an electroencephalogram signal feature set;

[0026] RR interval features, heart rate features, QRS duration features and power spectral density features are extracted from the electrocardiogram signal data set to obtain an electrocardiogram signal feature set;

[0027] Fundamental frequency features, pitch features, audio energy features, short-time zero-crossing rate features, tonal amplitude features and tonal change rate features are extracted from the tonal signal data set to obtain a tonal signal feature set;

[0028] Linear acceleration, angular velocity, displacement and velocity are extracted from the body node motion signal data set to obtain a body node motion feature set.

[0029] Preferably, feature extraction and feature fusion are performed on the emotion signal data set to obtain multi-dimensional fused emotion features, further including:

[0030] The eye image data set is input into a preset feature extraction unit to obtain a feature map; the calculation formula of the feature map is:

[0031]

[0032] Wherein,

[0033]

[0034] I t is the feature map of the current cycle; I t (i,j) is the value of the feature map at (i,j); H, W and C are the dimensions of the convolution kernel; is the weight matrix of (i,j) generated randomly by the feature extraction unit for the to-be-identified image data; t is the reference coefficient of (i,j) generated randomly by the feature extraction unit for the to-be-identified image data; t ​​

[0035] inputting the feature map into a preset nonlinear activation unit for nonlinear transformation to obtain a transformation result; a calculation formula of the transformation result is:

[0036]

[0037] wherein, Z t is the transformation result of the current loop; Z t (i,j) is a value of the transformation result at (i,j); ReLU(·) is a nonlinear activation function;

[0038] inputting the transformation result into a preset feature down-sampling unit for dimension reduction operation to obtain a feature down-sampling result; a calculation formula of the feature down-sampling result is:

[0039]

[0040] wherein,

[0041] is the feature down-sampling result; is a value of the feature down-sampling result at (i,j); and are sizes of a down-sampling window.

[0042] flattening the feature down-sampling result to obtain the eye image feature set.

[0043] Preferably, the emotion determination model comprises:

[0044] an input layer for performing standardization processing on the multi-dimensional fusion emotion feature;

[0045] an image feature extraction layer for performing feature extraction on an image feature part in the multi-dimensional fusion emotion feature through 3 convolution modules with a convolution kernel size of 3*3, 3 maximum pooling modules with a convolution kernel size of 2*2, and 1 flattening module;

[0046] a data feature extraction layer for performing feature extraction on a time series data feature part in the multi-dimensional fusion emotion feature through 2 GRU modules each containing 64 units, and 1 fully connected layer containing 64 units;

[0047] a feature fusion layer for performing normalization processing on outputs of the image feature extraction layer and the data feature extraction layer;

[0048] a fully connected layer for performing nonlinear transformation on the output of the feature fusion layer through a first fully connected module containing 128 units, a Dropout module, and a second fully connected module containing 64 units.

[0049] an output layer configured to perform probability analysis on the output of the full connection layer by the two classification modules to obtain a predicted emotional valence and a predicted emotional arousal;

[0050] The loss function in the emotional determination model training process is L total =αL valence +βL arousal ; wherein,

[0051]

[0052] L total is a model training loss value; α and β are respectively an emotional valence loss weight value and an emotional arousal loss weight value; L valence and L arousal are respectively an emotional valence loss value and an emotional arousal loss value; y valence,i and y arousal,i are respectively an ith emotional valence real value and an ith emotional valence predicted value; y t and y t are respectively an ith emotional arousal real value and an ith emotional arousal predicted value; and N is a total number of input samples.

[0053] Preferably, the emotional regulation scheme library comprises an emotional evaluation column, a music regulation scheme column, a video regulation scheme column, and a game regulation scheme column.

[0054] Preferably, after the emotional regulation is completed, a user emotional regulation effect score table is recorded, and when the number of the emotional regulation effect score table reaches a feedback threshold, the emotional determination model and the emotional regulation scheme library are optimized by using the emotional regulation effect score table, comprising:

[0055] data cleaning is performed on the training database of the emotional determination model, and data samples with an effect evaluation lower than an effective threshold in the emotional regulation effect score table are removed to obtain an updated database;

[0056] the emotional determination model is secondarily trained by using the updated database after the regularization coefficient and the model parameter iteration step length are replaced;

[0057] social research and questionnaire statistics are performed according to the emotional regulation effect score table to obtain research results and statistical distribution data, and the emotional regulation scheme library is deleted and supplemented according to the research results and the statistical distribution data.

[0058] Preferably, a VR device-based emotion regulation system comprises the VR headset, the motion capture device, the EEG device, the ECG device, the short-focus camera, the high-definition camera and the microphone.

[0059] The VR headset is fixed on the head of the user; the motion sensors of the motion capture device are attached to the back of the user's hand and the foot position respectively; the EEG device is fixed on the scalp of the user; the ECG device is fixed at the wrist of the user; the short-focus camera and the microphone are arranged on the inner side and the side of the VR headset respectively; and the high-definition camera is arranged on the front of the user.

[0060] The present application discloses the following technical effects:

[0061] The present application provides a VR device-based emotion regulation method and system, which collects various user emotion information, solves the problem of poor emotion analysis effect of the existing model, and realizes the strengthening of the prediction effect of the model; collects emotion regulation methods and switching commands, solves the defect of poor user experience of the existing emotion regulation method, realizes the timely adjustment of the scheme according to the user demand; collects the emotion regulation effect score table and optimizes the model and the strategy table, solves the defect that the conventional model does not have the improvement according to the user feedback, and realizes the improvement of the emotion regulation effect. BRIEF DESCRIPTION OF DRAWINGS

[0062] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments. Obviously, the drawings in the following description are only some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained from these drawings without creative labor.

[0063] Figure 1 The present application provides a VR device-based emotion regulation flowchart;

[0064] Figure 2 The present application provides a data acquisition flowchart;

[0065] Figure 3 The present application provides a VR device-based emotion regulation flowchart system. DETAILED DESCRIPTION

[0066] With reference to the accompanying drawings, the technical solutions in the embodiments of the present application will be clearly and completely described below, obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work belong to the protection scope of the present application.

[0067] The present application aims to provide a VR device-based emotion regulation method and system to improve the effect of user emotion regulation.

[0068] In order to make the above-mentioned purposes, features and advantages of the present application more obvious and easy to understand, the present application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0069] Figure 1 The VR device-based emotion regulation flowchart provided by the embodiments of the present application is shown in FIG. 1, the present application provides a VR device-based emotion regulation method, which comprises: Figure 1

[0070] Step 100: guiding the user to wear a VR emotion regulation device according to a preset flow and recording the emotion regulation mode selected by the user; the emotion regulation mode comprises music regulation, video regulation and game regulation;

[0071] Step 200: guiding the user to explain personal conditions and collecting the emotion signal dataset of the user in a VR virtual environment; the emotion signal dataset comprises electroencephalogram signal dataset, electrocardiogram signal dataset, body node motion signal dataset, eye image dataset, facial expression image dataset, tone signal dataset, volume signal dataset and keyword detection signal dataset;

[0072] Step 300: performing feature extraction and feature fusion on the emotion signal dataset to obtain multi-dimensional fusion emotion features;

[0073] Step 400: inputting the multi-dimensional fusion emotion features into a pre-trained emotion determination model to perform emotion analysis and obtaining emotion efficiency value and emotion activation degree;

[0074] Step 500: matching the emotion efficiency value and the emotion activation degree with a pre-arranged emotion regulation scheme library based on the emotion regulation mode to obtain a temporary emotion regulation scheme;

[0075] Step 600: updating the temporary emotion regulation scheme to a pre-worn VR headset to regulate the emotion of the user and collecting the switching command of the user in the emotion regulation process;

[0076] ​Step 700: After receiving the switching command, the temporary mood regulation scheme is updated by re-matching the mood value and the mood activation value and the pre-processed mood regulation scheme library, and returning to the step of updating the temporary mood regulation scheme to the pre-worn VR headset to regulate the mood of the user, and collecting the switching command of the user during the mood regulation process;

[0077] Step 800: After the mood regulation is completed, the mood regulation effect score table of the user is recorded, and when the number of the mood regulation effect score table reaches the feedback threshold, the mood regulation effect score table is used to optimize the mood determination model and the mood regulation scheme library.

[0078] Reference Figure 2 In the VR virtual environment, the user is guided to explain personal conditions and the emotional signal data set of the user is collected, including:

[0079] Step 201: Detect the working state of the VR headset, motion capture device, EEG device, ECG device, short-focus camera, high-definition camera, and microphone worn by the user, and if the working state is not normal, remind the user to check the device wearing or report the device exception to the staff;

[0080] Step 202: Collect the head EEG signal of the user through the electrode set on the scalp of the user by the EEG device, and obtain the EEG signal data set;

[0081] Step 203: Collect the ECG signal of the user through the ECG device worn on the wrist of the user, and obtain the ECG signal data set;

[0082] Step 204: Collect the motion signal of the user through the motion sensor set on the back of the hand and the foot of the user by the motion capture device, and obtain the body node motion signal data set;

[0083] Step 205: Collect the image of the eyes of the user through the short-focus camera set inside the VR headset, and obtain the eye image data set;

[0084] Step 206: Collect the facial image of the user through the high-definition camera placed opposite to the head of the user, and obtain the facial expression image data set;

[0085] Step 207: Collect the voice information of the user through the microphone set on the side of the VR headset, and obtain the tone signal data set and the volume signal data set;

[0086] Step 208: Obtain the keyword detection signal data set through the matching of the voice information and the pre-set keyword database.

[0087] Specifically, feature extraction and feature fusion are performed on the emotion signal dataset to obtain multi-dimensional fused emotion features, including:

[0088] Statistical features, activity features, mobility features, complexity features, differential entropy features, power spectral density features, and wavelet transform features are extracted from the electroencephalogram signal dataset to obtain an electroencephalogram signal feature set;

[0089] RR interval features, heart rate features, QRS duration features, and power spectral density features are extracted from the electrocardiogram signal dataset to obtain an electrocardiogram signal feature set;

[0090] Fundamental frequency features, pitch features, audio energy features, short-term zero-crossing rate features, tone amplitude features, and tone change rate features are extracted from the tone signal dataset to obtain a tone signal feature set;

[0091] Linear acceleration, angular velocity, displacement, and velocity are extracted from the body node motion signal dataset to obtain a body node motion feature set.

[0092] Further, feature extraction and feature fusion are performed on the emotion signal dataset to obtain multi-dimensional fused emotion features, further including:

[0093] The eye image dataset is input into a preset feature extraction unit to obtain a feature map; the calculation formula of the feature map is:

[0094]

[0095] wherein,

[0096]

[0097] I t is the feature map of the current cycle; I t (i,j) is the value of the feature map at (i,j); H, W, and C are the dimensions of the convolution kernel; is the weight matrix of (i,j) generated randomly by the feature extraction unit for the to-be-identified image data; t (i,j) generated randomly by the feature extraction unit for the to-be-identified image data; is the reference coefficient of (i,j) generated randomly by the feature extraction unit for the to-be-identified image data; t (i,j) generated randomly by the feature extraction unit for the to-be-identified image data;

[0098] The feature map is input into a preset nonlinear activation unit for nonlinear transformation to obtain a transformation result; the calculation formula of the transformation result is:

[0099]

[0100] wherein Z t is the transform result of the current cycle; Z t (i,j) is the value of the transform result at (i,j); ReLU(·) is a nonlinear activation function;

[0101] inputting the transform result into a preset feature down-sampling unit to perform dimension reduction operation, to obtain a feature down-sampling result; the calculation formula of the feature down-sampling result is:

[0102]

[0103] is the feature down-sampling result; is the value of the feature down-sampling result at (i,j); and are the sizes of the down-sampling window.

[0104] performing flattening on the feature down-sampling result, to obtain the eye image feature set.

[0105] Preferably, the emotion judgment model comprises:

[0106] an input layer, configured to perform standardization processing on the multi-dimensional fusion emotion feature;

[0107] an image feature extraction layer, configured to perform feature extraction on an image feature part in the multi-dimensional fusion emotion feature through 3 convolution modules with a convolution kernel size of 3*3, 3 maximum pooling modules with a convolution kernel size of 2*2, and 1 flattening module;

[0108] a data feature extraction layer, configured to perform feature extraction on a time series data feature part in the multi-dimensional fusion emotion feature through 2 GRU modules each containing 64 units, and 1 fully connected layer containing 64 units;

[0109] a feature fusion layer, configured to perform normalization processing on the outputs of the image feature extraction layer and the data feature extraction layer;

[0110] a fully connected layer, configured to perform nonlinear transformation on the output of the feature fusion layer through a first fully connected module containing 128 units, a Dropout module, and a second fully connected module containing 64 units;

[0111] an output layer, configured to perform probability analysis on the output of the fully connected layer through 2 classification modules, to obtain the predicted emotion valence and emotion arousal;

[0112] the loss function in the emotion judgment model training process is: L total = αL valence + βLarousal ; wherein,

[0113]

[0114] L total is a model training loss value; a and β are respectively an emotional valence loss weight value and an emotional activation loss weight value; L valence , L arousal are respectively an emotional valence loss value and an emotional activation loss value; y valence,i , are respectively an i-th emotional valence real value and an i-th emotional valence predicted value; y arousal,i , are respectively an i-th emotional activation real value and an i-th emotional activation predicted value; N is a total number of input samples.

[0115] Optionally, the emotion regulation scheme library comprises: an emotion evaluation column, a music regulation scheme column, a video regulation scheme column, and a game regulation scheme column.

[0116] Preferably, after the emotion regulation is completed, a user emotion regulation effect score table is recorded, and when the number of the emotion regulation effect score table reaches a feedback threshold, the emotion determination model and the emotion regulation scheme library are optimized by using the emotion regulation effect score table, comprising:

[0117] Data cleaning is performed on a training database of the emotion determination model, and data samples with an effect evaluation lower than an effective threshold in the emotion regulation effect score table are removed to obtain an updated database;

[0118] After the regular coefficient and the model parameter iteration step length are replaced, the emotion determination model is trained again by using the updated database;

[0119] Social research and questionnaire statistics are performed according to the emotion regulation effect score table to obtain research results and statistical distribution data, and the emotion regulation scheme library is deleted and supplemented according to the research results and the statistical distribution data.

[0120] Reference Figure 3 , an emotion regulation system based on a VR device, comprising: the VR head-mounted device, the motion capture device, the EEG device, the ECG device, the short-focus camera, the high-definition camera, and the microphone;

[0121] The VR headset is fixed on the user's head; the motion sensors of the motion capture device are attached to the user's back of the hand and foot positions respectively; the EEG device is fixed on the user's scalp position; the ECG device is fixed on the user's wrist; the short-focus camera and the microphone are respectively arranged on the inner side and side edge of the VR headset; and the high-definition camera is arranged on the front position of the user.

[0122] Optionally, the embodiment provides some keywords related to emotions, including: anxiety, stress, sadness, loneliness, guilt, confusion, self-blame, worry, fear, fatigue, unease, excitement, helplessness, sadness, surprise, self-confidence, loss, excitement, boredom, expectation, anger, frustration, relaxation, satisfaction, disappointment, pride, comfort, despair, happiness, melancholy. When the user explains the personal situation, the user's voice is converted into text in real time, and matched with the keyword database. Each keyword corresponds to a code, and the codes corresponding to all detected keywords are combined, with special characters & used to distinguish between different codes. Each code consists of a keyword address and a frequency of occurrence.

[0123] Preferably, Affective Valence is an important concept in psychology and affective research, referring to the individual's emotional response to a specific emotion or stimulus, either positive or negative. In simple terms, it measures the "good" or "bad" feeling brought by a certain emotion or experience. Affective Valence is usually represented as a continuous dimension, from highly negative emotions (such as sadness, anger) to highly positive emotions (such as happiness, excitement). The combination of Affective Valence and Emotional Activation can basically determine the user's emotional state, and emotions can be expressed in the following combinations in these two dimensions:

[0124] Positive Affective Valence + High Emotional Activation: for example, excitement, joy, excitement, etc.

[0125] Positive Affective Valence + Low Emotional Activation: for example, satisfaction, tranquility, peace, etc.

[0126] Negative Affective Valence + High Emotional Activation: for example, anger, anxiety, fear, etc.

[0127] Negative Affective Valence + Low Emotional Activation: for example, sadness, frustration, boredom, etc.

[0128] Referring to Table 1, the present embodiment provides an emotion regulation scheme library as a reference. The first two columns in the table are as emotion evaluation columns, and the rear are regulation schemes. The scheme library is only for reference. In actual use, the emotion evaluation can be divided into several grades, and the emotion value and the emotion activation are uniformly divided into several grades and combined with each other. The specific regulation scheme is formulated according to the combined emotion evaluation grade. Since it is only for reference, the present embodiment does not provide too many videos and game types. Before the scheme is formulated, music, video and game types can be fully collected, and statistical research and effect analysis are performed, and a more detailed scheme database is formulated.

[0129] Table 1

[0130]

[0131]

[0132] The beneficial effects of the present application are as follows:

[0133] The present application increases the analysis dimension of the model and strengthens the prediction effect of the model by collecting the emotion information of multiple users. The use experience of the user is improved, and the acceptance of the user is improved by collecting the emotion regulation mode and the switching command. The generalization and reliability of the model are improved by collecting the emotion regulation effect score table and optimizing the model and the strategy table. The emotion regulation effect is improved by continuously receiving feedback information.

[0134] The embodiments in the specification are described in a progressive manner, and each embodiment focuses on the difference from other embodiments. The same or similar parts of each embodiment can be referred to each other. For the system disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple, and the related parts can be referred to the method part.

[0135] The principles and implementation modes of the present application are described by applying specific examples in the present application. The above embodiment description is only used to help understand the method of the present application and its core idea. For those skilled in the art, according to the idea of the present application, the specific implementation mode and application range will be changed. In summary, the content of the specification should not be understood as a limitation of the present application.

Claims

1. A VR device-based emotion regulation method, characterized in that, The method comprises the steps of: According to the preset process, guide the user to wear the VR emotion regulation device, and record the user's selected emotion regulation method; The emotion regulation method includes music regulation, video regulation, and game regulation; In the VR virtual environment, guide the user to explain personal situation and collect the user's emotion signal dataset; The emotion signal dataset includes: electroencephalogram signal dataset, electrocardiogram signal dataset, body node motion signal dataset, eye image dataset, facial expression image dataset, tone signal dataset, volume signal dataset, and keyword detection signal dataset; Feature extraction and feature fusion are performed on the emotion signal dataset to obtain multi-dimensional fusion emotion features; Input the multi-dimensional fusion emotion features into the pre-trained emotion judgment model for emotion analysis to obtain emotion efficiency value and emotion activation degree; Based on the emotion regulation method, the emotion efficiency value, and the emotion activation degree, and the pre-arranged emotion regulation scheme library are matched to obtain a temporary emotion regulation scheme; Update the temporary emotion regulation scheme to the pre-worn VR headset to regulate the user's emotions, and collect the user's switching command during the emotion regulation process; When the switching command is received, re-match the emotion efficiency value and the emotion activation degree with the pre-arranged emotion regulation scheme library, update the temporary emotion regulation scheme, and return to the step of "updating the temporary emotion regulation scheme to the pre-worn VR headset to regulate the user's emotions, and collecting the user's switching command during the emotion regulation process"; After the emotion regulation is completed, record the user's emotion regulation effect score table, when the number of emotion regulation effect score tables reaches the feedback threshold, use the emotion regulation effect score table to optimize the emotion judgment model and the emotion regulation scheme library; Feature extraction and feature fusion are performed on the emotion signal dataset to obtain multi-dimensional fusion emotion features, which further comprise: Input the eye image dataset into the preset feature extraction unit to obtain a feature map; The calculation formula of the feature map is: ； Wherein, ； The feature map for the current loop; For the feature map in The value at; , and All of these are the dimensions of the convolution kernel; The image data to be identified is randomly generated by the feature extraction unit. The weight matrix; The image data to be identified is randomly generated by the feature extraction unit. The benchmark coefficient; Input the feature map into the preset nonlinear activation unit for nonlinear transformation to obtain a transformation result; The calculation formula of the transformation result is: ； wherein, is the transform result for the current cycle; is the value of the transform result at is the value of the transform result at is a nonlinear activation function; Input the transformation result into the preset feature downsampling unit for dimension reduction operation to obtain a feature downsampling result; The calculation formula of the feature downsampling result is: ； wherein ; is the downsampled result for the feature; is the value of the downsampled result at ; and are the dimensions of the downsample window. Flatten the feature downsampling result to obtain the eye image feature set.

2. The emotion regulation method based on a VR device according to claim 1, wherein, In the VR virtual environment, guide the user to explain personal situation and collect the user's emotion signal dataset, which comprises: Detect the working state of the VR headset, motion capture device, EEG device, ECG device, short-focus camera, high-definition camera, and microphone worn by the user, and if the working state is not normal, remind the user to check the device wearing or report the device exception to the staff; Collect the user's head electroencephalogram signal through the electrode set on the user's scalp by the EEG device to obtain the electroencephalogram signal dataset; The ECG device worn on the wrist of the user collects the ECG signal of the user to obtain the ECG signal dataset; The motion sensor arranged on the back of the hand and the foot of the user collects the motion signal of the user to obtain the body node motion signal dataset; The short-focus camera arranged inside the VR head-mounted device collects the images of the eyes of the user to obtain the eye image dataset; The high-definition camera placed opposite to the head of the user collects the facial image of the user to obtain the facial expression image dataset; The microphone arranged on the side of the VR head-mounted device collects the voice information of the user to obtain the tone signal dataset and the volume signal dataset; The keyword detection signal dataset is obtained through matching of the voice information and a preset keyword database. 3.The VR device-based emotion regulation method of claim 1, wherein, Feature extraction and feature fusion are performed on the emotion signal dataset to obtain multi-dimensional fused emotion features, including: Statistical features, activity features, mobility features, complexity features, differential entropy features, power spectral density features, and wavelet transform features are extracted from the EEG signal dataset to obtain an EEG signal feature set; RR interval features, heart rate features, QRS duration features, and power spectral density features are extracted from the ECG signal dataset to obtain an ECG signal feature set; Fundamental frequency features, pitch features, audio energy features, short-time zero-crossing rate features, tone amplitude features, and tone change rate features are extracted from the tone signal dataset to obtain a tone signal feature set; Linear acceleration, angular velocity, displacement, and velocity are extracted from the body node motion signal dataset to obtain a body node motion feature set.

4. The emotion regulation method based on a VR device according to claim 1, wherein, The emotion determination model includes: An input layer for standardizing the multi-dimensional fused emotion features; An image feature extraction layer for extracting image features from the multi-dimensional fused emotion features through three convolution modules with a kernel size of 3*3, three maximum pooling modules with a kernel size of 2*2, and one flattening module; A data feature extraction layer for extracting time series data features from the multi-dimensional fused emotion features through two GRU modules each containing 64 units and one fully connected layer containing 64 units; A feature fusion layer for normalizing the outputs of the image feature extraction layer and the data feature extraction layer; A fully connected layer for performing non-linear transformation on the output of the feature fusion layer through a first fully connected module containing 128 units, a Dropout module, and a second fully connected module containing 64 units; An output layer for performing probability analysis on the output of the fully connected layer through two classification modules to obtain predicted emotion valence and emotion arousal; The loss function in the emotion determination model training process is: ; wherein, ; ； is a model training loss value; , are respectively an emotional valence loss weight and an emotional activation loss loss weight; , are respectively an emotional valence loss value and an emotional activation loss loss value; , are respectively an ith emotional valence real value and an ith emotional valence predicted value; , are respectively an ith emotional activation real value and an ith emotional activation predicted value; N is a total number of input samples.

5. The emotion regulation method based on a VR device according to claim 1, wherein, The emotion regulation scheme library includes an emotion evaluation column, a music regulation scheme column, a video regulation scheme column, and a game regulation scheme column.

6. The emotion regulation method based on a VR device according to claim 1, wherein, After the emotion regulation is completed, a user emotion regulation effect score table is recorded, when the number of the emotion regulation effect score table reaches a feedback threshold, the emotion regulation effect score table is used to optimize the emotion determination model and the emotion regulation scheme library, comprising: Data cleaning is performed on the training database of the emotion determination model, and data samples with an effect evaluation lower than an effective threshold in the emotion regulation effect score table are removed to obtain an updated database; The emotion determination model is secondarily trained using the updated database after the regular coefficient and the model parameter iteration step length are replaced; Social research and questionnaire statistics are performed according to the emotion regulation effect score table to obtain research results and statistical distribution data, and the emotion regulation scheme library is pruned and supplemented according to the research results and the statistical distribution data.

7. A VR device-based emotion regulation system, comprising: The system is applied to the emotion regulation method based on the VR device provided in claim 2, and the system comprises the VR head-mounted device, the motion capture device, the EEG device, the ECG device, the short-focus camera, the high-definition camera and the microphone. The VR head-mounted device is fixed on the head of the user, the motion sensors of the motion capture device are respectively attached to the back of the hand and the foot of the user, the EEG device is fixed on the scalp of the user, the ECG device is fixed on the wrist of the user, the short-focus camera and the microphone are respectively arranged on the inner side and the side of the VR head-mounted device, and the high-definition camera is arranged on the front of the user.

Images