Emotion regulation method and system based on VR equipment

Abstract

The invention belongs to the technical field of VR, and provides an emotion regulation method and system based on VR equipment. The method comprises the steps of equipment wearing, emotional signal data set collection, feature extraction and feature fusion, emotional value and emotional activation degree calculation, temporary emotional regulation scheme matching, switching command collection, temporary emotional regulation scheme updating, emotional regulation effect score table collection and model and strategy table optimization. According to the method, by collecting emotion information of various users, the analysis dimension of the model is increased, and the prediction effect of the model is enhanced; by collecting the emotion adjusting mode and the switching command, the use experience of the user is improved, and the acceptability of the user is improved; through collection of the emotion regulation effect score table and optimization of the model and strategy table, generalization and reliability of the model are improved, and improvement of the emotion regulation effect is achieved by continuously receiving feedback information.

Description

Technical Field

[0001] The present invention relates to the technical field of VR, and in particular, to an emotion regulation method and system based on a VR device. Background Art

[0002] In daily life and work, people will encounter various pressures. With the rapid development of modern society, the pressures faced by people are increasing. People often need an effective way to relieve stress and regulate emotions. Ordinary emotion regulation methods include: shouting loudly, crying, exercising, etc. In recent years, virtual reality technology has gradually penetrated into people's lives. VR devices can generate a virtual three-dimensional environment to give wearers an immersive experience, which has a great effect on regulating users' emotions.

[0003] Traditional emotion regulation methods are limited by people's own personalities, physiques, etc., and the lack of choices, resulting in short-lived or poor effects. Some methods and devices for emotion regulation through VR have also been proposed in the prior art, but these methods all have problems such as inaccurate emotion analysis, slow or no user feedback regulation, and single regulation means, resulting in insignificant emotion regulation effects. Summary of the Invention

[0004] In order to overcome the deficiencies of the prior art, the purpose of the present invention is to provide an emotion regulation method and system based on a VR device to improve the effect of user emotion regulation.

[0005] To achieve the above purpose, the present invention provides the following solutions:

[0006] An emotion regulation method based on a VR device, comprising:

[0007] Guiding a user to wear a VR emotion regulation device according to a preset process, and recording the emotion regulation method selected by the user; the emotion regulation method includes: music regulation, video regulation, and game regulation;

[0008] Guiding the user to make a personal situation statement in the VR virtual environment and collecting the user's emotion signal dataset; the emotion signal dataset includes: electroencephalogram signal dataset, electrocardiogram signal dataset, body node motion signal dataset, eye image dataset, facial expression image dataset, intonation signal dataset, volume signal dataset, and keyword detection signal dataset;

[0009] Performing feature extraction and feature fusion on the emotion signal dataset to obtain multi-dimensional fused emotion features;

[0010] Inputting the multi-dimensional fused emotion features into a pre-trained emotion determination model for emotion analysis to obtain an emotion valence and an emotion activation degree;

[0011] Match the emotional regulation method with the emotional valence, emotional activation degree, and pre - sorted emotional regulation plan library to obtain a temporary emotional regulation plan;

[0012] Update the temporary emotional regulation plan to the pre - worn VR headset to regulate the user's emotions, and collect the user's switching commands during the emotional regulation process;

[0013] After receiving the switching command, re - match through the emotional valence, emotional activation degree, and pre - sorted emotional regulation plan library, update the temporary emotional regulation plan, and return to the step "Update the temporary emotional regulation plan to the pre - worn VR headset to regulate the user's emotions, and collect the user's switching commands during the emotional regulation process";

[0014] After the emotional regulation is completed, record the user's emotional regulation effect score table. When the number of the emotional regulation effect score tables reaches the feedback threshold, use the emotional regulation effect score table to optimize the emotional judgment model and the emotional regulation plan library.

[0015] Preferably, guide the user to make a personal situation description and collect the user's emotional signal dataset in the VR virtual environment, including:

[0016] Detect the working status of the VR headset, motion capture device, EEG device, ECG device, short - focus camera, high - definition camera, and microphone worn by the user. If the working status is abnormal, remind the user to check the device wearing or report the device abnormality to the staff;

[0017] Collect the user's head EEG signals through the electrodes of the EEG device set on the user's scalp to obtain the EEG signal dataset;

[0018] Collect the user's ECG signals through the ECG device worn on the user's wrist side to obtain the ECG signal dataset;

[0019] Collect the user's motion signals through the motion sensors of the motion capture device set on the user's back of the hand and feet to obtain the body node motion signal dataset;

[0020] Collect the images of the user's two eyes through the short - focus camera set on the inner side of the VR headset to obtain the eye image dataset;

[0021] Collect the user's facial images through the high - definition camera placed facing the user's head to obtain the facial expression image dataset;

[0022] Collect the voice information of the user through the microphone set on the side of the VR headset to obtain the intonation signal dataset and the volume signal dataset;

[0023] Obtain the keyword detection signal dataset through the matching of the voice information and the preset keyword database.

[0024] Preferably, perform feature extraction and feature fusion on the emotion signal dataset to obtain multi-dimensional fused emotion features, including:

[0025] Perform statistical feature, activity feature, mobility feature, complexity feature, differential entropy feature, power spectral density feature, and wavelet transform feature extraction on the electroencephalogram signal dataset to obtain the electroencephalogram signal feature set;

[0026] Perform RR interval feature, heart rate feature, QRS duration feature, and power spectral density feature extraction on the electrocardiogram signal dataset to obtain the electrocardiogram signal feature set;

[0027] Perform fundamental frequency feature, pitch feature, audio energy feature, short-time zero-crossing rate feature, intonation amplitude feature, and intonation change rate feature extraction on the intonation signal dataset to obtain the intonation signal feature set;

[0028] Perform linear acceleration, angular velocity, displacement, and velocity extraction on the body node motion signal dataset to obtain the body node motion feature set.

[0029] Preferably, perform feature extraction and feature fusion on the emotion signal dataset to obtain multi-dimensional fused emotion features, and also include:

[0030] Input the eye image dataset into a preset feature extraction unit to obtain a feature map; the calculation formula of the feature map is:

[0031]

[0032] Among them,

[0033]

[0034] I t is the feature map of the current loop; I t (i,j) is the value of the feature map at (i,j); H, W, and C are all the sizes of the convolution kernels; is the weight matrix of I t (i,j) randomly generated by the feature extraction unit for the image data to be recognized; is the I t (i,j) randomly generated by the feature extraction unit for the image data to be recognized;

[0035] Input the feature map into a preset non - linear activation unit for non - linear transformation to obtain a transformation result; the calculation formula for the transformation result is:

[0036]

[0037] where Z t is the transformation result of the current cycle; Z t (i, j) is the value of the transformation result at (i, j); ReLU(·) is the non - linear activation function;

[0038] Input the transformation result into a preset feature down - sampling unit for dimensionality reduction operation to obtain a feature down - sampling result; the calculation formula for the feature down - sampling result is:

[0039]

[0040] where

[0041] is the feature down - sampling result; is the value of the feature down - sampling result at (i, j); and are both the sizes of the down - sampling window.

[0042] Flatten the feature down - sampling result to obtain the eye image feature set.

[0043] Preferably, the emotion determination model includes:

[0044] An input layer for normalizing the multi - dimensional fusion emotion features;

[0045] An image feature extraction layer for extracting image features in the multi - dimensional fusion emotion features through 3 convolution modules with a convolution kernel size of 3*3, 3 max - pooling modules with a convolution kernel size of 2*2, and 1 flattening module;

[0046] A data feature extraction layer for extracting temporal data features in the multi - dimensional fusion emotion features through 2 GRU modules containing 64 units and 1 fully - connected layer containing 64 units;

[0047] A feature fusion layer for normalizing the outputs of the image feature extraction layer and the data feature extraction layer;

[0048] A fully - connected layer for performing non - linear transformation on the output of the feature fusion layer through a first fully - connected module containing 128 units, a Dropout module, and a second fully - connected module containing 64 units;

[0049] An output layer, configured to perform probability analysis on the output of the fully connected layer through two classification modules, so as to obtain the predicted emotional valence and emotional activation degree;

[0050] The loss function in the training process of the emotion determination model is: L total = αL valence + βL arousal ; where

[0051]

[0052] L total is the loss value of model training; α and β are the emotional valence loss weight and emotional activation loss weight respectively; L valence and L arousal are the emotional valence loss value and emotional activation loss value respectively; y valence,i , are the i-th true emotional valence value and the i-th predicted emotional valence value respectively; y arousal,i , are the i-th true emotional activation value and the i-th predicted emotional activation value respectively; N is the total number of input samples.

[0053] Preferably, the emotion regulation scheme library includes: an emotion evaluation column, a music regulation scheme column, a video regulation scheme column, and a game regulation scheme column.

[0054] Preferably, after the emotion regulation is completed, a user's emotion regulation effect score table is recorded. When the number of the emotion regulation effect score tables reaches a feedback threshold, the emotion determination model and the emotion regulation scheme library are optimized by using the emotion regulation effect score table, including:

[0055] Performing data cleaning on the training database of the emotion determination model, and removing data samples with an effect evaluation lower than an effective threshold in the emotion regulation effect score table to obtain an updated database;

[0056] After changing the regularization coefficient and the model parameter iteration step size, the emotion determination model is retrained by using the updated database;

[0057] Performing social research and questionnaire statistics according to the emotion regulation effect score table to obtain research results and statistical distribution data, and deleting and supplementing the emotion regulation scheme library according to the research results and the statistical distribution data.

[0058] Preferably, an emotion regulation system based on a VR device includes: the VR head-mounted device, the motion capture device, the EEG device, the ECG device, the short-focus camera, the high-definition camera, and the microphone;

[0059] The VR head-mounted device is fixed on the user's head; the motion sensors of the motion capture device are respectively attached to the back of the user's hand and the foot position; the EEG device is fixed on the user's scalp position; the ECG device is fixed on the user's wrist; the short-focus camera and the microphone are respectively arranged at the inner side and the side position of the VR head-mounted device; the high-definition camera is arranged at the front position of the user.

[0060] The present invention discloses the following technical effects:

[0061] The present invention provides an emotion regulation method and system based on a VR device. By collecting various users' emotion information, the problem of poor emotion analysis effect of the existing model is solved, and the prediction effect of the model is strengthened; by collecting emotion regulation methods and switching commands, the defect of poor user experience of the existing emotion regulation method is solved, and the timely adjustment of the scheme according to the user's needs is realized; by collecting the emotion regulation effect score table and optimizing the model and strategy table, the defect that the conventional model does not have the ability to be improved according to the user's feedback is solved, and the improvement of the emotion regulation effect is realized. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required to be used in the embodiments. Obviously, the following drawings are only some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0063] Figure 1 It is a schematic diagram of the emotion regulation process based on a VR device provided by an embodiment of the present invention;

[0064] Figure 2 It is a schematic diagram of the data collection process provided by an embodiment of the present invention;

[0065] Figure 3 It is a schematic diagram of the emotion regulation process system based on a VR device provided by an embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0066] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0067] The purpose of the present invention is to provide an emotion regulation method and system based on a VR device to improve the effect of user emotion regulation.

[0068] To make the above objects, features, and advantages of the present invention more obvious and understandable, the present invention will be further described in detail below in conjunction with the accompanying drawings and specific embodiments.

[0069] Figure 1 It is a schematic diagram of the emotion regulation process based on a VR device provided for an embodiment of the present invention. As Figure 1 shown, the present invention provides an emotion regulation method based on a VR device, including:

[0070] Step 100: Guide the user to wear the VR emotion regulation device according to a preset process and record the selected emotion regulation method by the user; the emotion regulation method includes: music regulation, video regulation, and game regulation;

[0071] Step 200: Guide the user to make a personal situation description in the VR virtual environment and collect the user's emotion signal dataset; the emotion signal dataset includes: electroencephalogram signal dataset, electrocardiogram signal dataset, body node movement signal dataset, eye image dataset, facial expression image dataset, intonation signal dataset, volume signal dataset, and keyword detection signal dataset;

[0072] Step 300: Perform feature extraction and feature fusion on the emotion signal dataset to obtain multi-dimensional fused emotion features;

[0073] Step 400: Input the multi-dimensional fused emotion features into a pre-trained emotion determination model for emotion analysis to obtain an emotion valence value and an emotion activation degree;

[0074] Step 500: Based on the emotion regulation method, match the emotion valence value, the emotion activation degree, and a pre-organized emotion regulation plan library to obtain a temporary emotion regulation plan;

[0075] Step 600: Update the temporary emotion regulation plan to the pre-worn VR head-mounted device to regulate the user's emotion and collect the user's switching commands during the emotion regulation process;

[0076] Step 700: After receiving the switching command, re-match through the emotional valence value, the emotional activation degree, and the pre-organized emotional regulation solution library to update the temporary emotional regulation solution, and return to the step of "updating the temporary emotional regulation solution to the pre-worn VR head-mounted device to regulate the user's emotions and collecting the switching command of the user during the emotion regulation process";

[0077] Step 800: After the emotion regulation is completed, record the emotion regulation effect score table of the user. When the number of the emotion regulation effect score tables reaches the feedback threshold, optimize the emotion judgment model and the emotion regulation solution library by using the emotion regulation effect score table.

[0078] Reference Figure 2 , guide the user to make a personal situation statement in the VR virtual environment and collect the user's emotional signal dataset, including:

[0079] Step 201: Detect the working status of the VR head-mounted device, motion capture device, EEG device, ECG device, short-focus camera, high-definition camera, and microphone worn by the user. If the working status is abnormal, remind the user to check the device wearing or report the device abnormality to the staff;

[0080] Step 202: Collect the user's head electroencephalogram signal through the electrodes set on the user's scalp by the EEG device to obtain the electroencephalogram signal dataset;

[0081] Step 203: Collect the user's electrocardiogram signal through the ECG device worn on the user's wrist side to obtain the electrocardiogram signal dataset;

[0082] Step 204: Collect the user's motion signal through the motion sensors set on the user's back of the hand and feet by the motion capture device to obtain the body node motion signal dataset;

[0083] Step 205: Collect the images of the user's both eyes through the short-focus camera set inside the VR head-mounted device to obtain the eye image dataset;

[0084] Step 206: Collect the user's facial image through the high-definition camera placed directly in front of the user's head to obtain the facial expression image dataset;

[0085] Step 207: Collect the user's voice information through the microphone set on the side of the VR head-mounted device to obtain the intonation signal dataset and the volume signal dataset;

[0086] Step 208: Obtain the keyword detection signal dataset through the matching of the voice information and the preset keyword database.

[0087] Specifically, feature extraction and feature fusion are performed on the emotional signal dataset to obtain multi-dimensional fused emotional features, including:

[0088] Statistical features, activity features, mobility features, complexity features, differential entropy features, power spectral density features, and wavelet transform features are extracted from the electroencephalogram signal dataset to obtain an electroencephalogram signal feature set;

[0089] RR interval features, heart rate features, QRS duration features, and power spectral density features are extracted from the electrocardiogram signal dataset to obtain an electrocardiogram signal feature set;

[0090] Fundamental frequency features, pitch features, audio energy features, short-time zero-crossing rate features, intonation amplitude features, and intonation change rate features are extracted from the intonation signal dataset to obtain an intonation signal feature set;

[0091] Linear acceleration, angular velocity, displacement, and velocity are extracted from the body node motion signal dataset to obtain a body node motion feature set.

[0092] Furthermore, feature extraction and feature fusion are performed on the emotional signal dataset to obtain multi-dimensional fused emotional features, further including:

[0093] The eye image dataset is input into a preset feature extraction unit to obtain a feature map; the calculation formula of the feature map is:

[0094]

[0095] Among them,

[0096]

[0097] I t is the feature map of the current loop; I t (i,j) is the value of the feature map at (i,j); H, W, and C are all the sizes of the convolution kernels; is the weight matrix of I t (i,j) randomly generated by the feature extraction unit for the image data to be recognized; is the reference coefficient of I t (i,j) randomly generated by the feature extraction unit for the image data to be recognized;

[0098] The feature map is input into a preset non-linear activation unit for non-linear transformation to obtain a transformation result; the calculation formula of the transformation result is:

[0099]

[0100] Among them, Z t is the transformation result of the current loop; Z t (i,j) is the value of the transformation result at (i,j); ReLU(·) is a non-linear activation function;

[0101] Input the transformation result into a preset feature downsampling unit for dimensionality reduction operation to obtain a feature downsampling result; the calculation formula of the feature downsampling result is:

[0102]

[0103] is the feature downsampling result; is the value of the feature downsampling result at (i,j); and are both the sizes of the downsampling window.

[0104] Flatten the feature downsampling result to obtain the eye image feature set.

[0105] Preferably, the emotion determination model includes:

[0106] An input layer for normalizing the multi-dimensional fused emotion features;

[0107] An image feature extraction layer for extracting image features in the multi-dimensional fused emotion features through 3 convolutional modules with a convolutional kernel size of 3*3, 3 max-pooling modules with a convolutional kernel size of 2*2, and 1 flattening module;

[0108] A data feature extraction layer for extracting time-series data features in the multi-dimensional fused emotion features through 2 GRU modules containing 64 units and 1 fully-connected layer containing 64 units;

[0109] A feature fusion layer for normalizing the outputs of the image feature extraction layer and the data feature extraction layer;

[0110] A fully-connected layer for performing non-linear transformation on the output of the feature fusion layer through a first fully-connected module containing 128 units, a Dropout module, and a second fully-connected module containing 64 units;

[0111] An output layer for performing probability analysis on the output of the fully-connected layer through 2 classification modules to obtain the predicted emotion valence and emotion activation degree;

[0112] The loss function in the training process of the emotion determination model is: L total =αL valence +βLarousal ;in,

[0113]

[0114] L total is the model training loss value; α and β are the emotion valence loss weight and emotion activation loss weight respectively; L valence , L arousal are the emotional valence loss value and emotional activation loss value respectively; y valence,i , are the true value of the i-th emotional valence and the predicted value of the i-th emotional valence respectively; y arousal,i , are the true value of the i-th emotion activation and the predicted value of the i-th emotion activation respectively; N is the total number of input samples.

[0115] Optionally, the emotion regulation solution library includes: an emotion evaluation column, a music regulation solution column, a video regulation solution column, and a game regulation solution column.

[0116] Preferably, after the emotion regulation is completed, the emotion regulation effect score sheet of the user is recorded, and when the number of the emotion regulation effect score sheets reaches a feedback threshold, the emotion determination model and the emotion regulation solution library are optimized using the emotion regulation effect score sheets, including:

[0117] Performing data cleaning on the training database of the emotion determination model, removing data samples whose effect evaluation in the emotion regulation effect scoring table is lower than the effective threshold, and obtaining an updated database;

[0118] After changing the regularization coefficient and the model parameter iteration step, the emotion determination model is trained again using the updated database;

[0119] Social surveys and questionnaire statistics are conducted according to the emotion regulation effect scoring table to obtain survey results and statistical distribution data, and the emotion regulation program library is deleted and supplemented according to the survey results and the statistical distribution data.

[0120] refer to Figure 3 , an emotion regulation system based on a VR device, comprising: the VR head-mounted device, the motion capture device, the EEG device, the ECG device, the short-focus camera, the high-definition camera and the microphone;

[0121] The VR head-mounted device is fixed on the user's head; the motion sensors of the motion capture device are respectively attached to the user's back of the hand and foot positions; the EEG device is fixed on the user's scalp position; the ECG device is fixed on the user's wrist; the short-focus camera and the microphone are respectively arranged at the inner side and the side position of the VR head-mounted device; the high-definition camera is arranged in the front position of the user.

[0122] Optionally, this embodiment provides some keywords related to emotions, including: anxiety, stress, sadness, loneliness, guilt, confusion, self-blame, upset, fear, exhaustion, uneasiness, excitement, helplessness, sadness, surprise, confidence, loss, agitation, boredom, expectation, anger, frustration, relaxation, satisfaction, disappointment, pride, comfort, despair, happiness, melancholy. When the user makes a personal statement, the user's voice is converted into text in real time and matched with the keyword database. Each keyword corresponds to a code. The codes corresponding to all detected keywords are combined, and special symbol & is used to distinguish between different codes. Each code consists of a keyword address and an occurrence frequency.

[0123] Preferably, Affective Valence is an important concept in psychology and emotion research, referring to the positive or negative degree of an individual's emotional response to a specific emotion or stimulus. Simply put, it measures the "good" or "bad" feeling brought by a certain emotion or experience. Affective Valence is usually represented as a continuous dimension, ranging from highly negative emotions (such as sadness, anger) to highly positive emotions (such as happiness, excitement). The combination of Affective Valence value and emotional arousal can basically determine the user's emotional state, and emotions can be manifested as the following combinations in these two dimensions:

[0124] Positive valence + high arousal: such as excitement, joy, agitation, etc.;

[0125] Positive valence + low arousal: such as satisfaction, tranquility, peace, etc.;

[0126] Negative valence + high arousal: such as anger, anxiety, fear, etc.;

[0127] Negative valence + low arousal: such as sadness, frustration, boredom, etc.

[0128] Referring to Table 1, this embodiment provides an emotional regulation solution library for reference. The first two columns in the table are used as the emotional evaluation columns, and the following are the regulation solutions. This solution library is only for reference. In the actual use process, the emotional evaluation can adopt a grading system, evenly dividing the emotional valence and emotional arousal into several grades and then combining them with each other. Specifically, the mediation solution is formulated according to the combined emotional evaluation grades. Since it is only for reference, this embodiment does not provide too many video and game types. Before formulating the solution, music, video, and game types can be fully collected, and research statistics and effect analysis can be carried out, and then a more detailed solution database can be formulated.

[0129] Table 1

[0130]

[0131]

[0132] The beneficial effects of the present invention are as follows:

[0133] By collecting the emotional information of multiple users, the present invention increases the analysis dimension of the model and strengthens the prediction effect of the model; by collecting the emotional regulation methods and switching commands, it improves the user experience and increases the user acceptance; by collecting the emotional regulation effect score table and optimizing the model and strategy table, it improves the generalization and reliability of the model, and by continuously receiving feedback information, it realizes the improvement of the emotional regulation effect.

[0134] In this specification, each embodiment is described in a progressive manner. The key point of each embodiment is to illustrate the differences from other embodiments. The same or similar parts among the embodiments can be referred to each other. For the system disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the description in the method part.

[0135] Specific examples are used in this article to elaborate on the principle and implementation manner of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea; at the same time, for those of ordinary skill in the art, according to the idea of the present invention, there will be changes in the specific implementation manner and application scope. In summary, the content of this specification should not be construed as a limitation to the present invention.

Claims

1. An emotion regulation method based on VR equipment, characterized in that: include: Instruct users to wear VR emotion regulation equipment according to the preset process, and record the emotion regulation method selected by the user; The emotion regulation methods include: music regulation, video regulation and game regulation; Instructing the user to explain his / her personal situation in a VR virtual environment and collecting the user's emotional signal data set; the emotional signal data set includes: an electroencephalogram signal data set, an electrocardiogram signal data set, a body node motion signal data set, an eye image data set, a facial expression image data set, a tone signal data set, a volume signal data set, and a keyword detection signal data set; Performing feature extraction and feature fusion on the emotion signal data set to obtain multi-dimensional fusion emotion features; Inputting the multi-dimensional fusion emotion features into a pre-trained emotion determination model to perform emotion analysis to obtain emotion effectiveness value and emotion activation degree; Based on the emotion regulation method, the emotion efficacy value and the emotion activation degree are matched with a pre-sorted emotion regulation solution library to obtain a temporary emotion regulation solution; Updating the temporary emotion regulation scheme to the pre-worn VR head mounted device to regulate the user's emotions, and collecting the user's switching commands during the emotion regulation process; After receiving the switching command, the temporary emotion regulation scheme is updated by re-matching the emotion efficacy value and the emotion activation degree with the pre-sorted emotion regulation scheme library, and the process returns to the step of "updating the temporary emotion regulation scheme to the pre-worn VR head mounted device to regulate the user's emotions, and collecting the user's switching command during the emotion regulation process"; After the emotion regulation is completed, the emotion regulation effect score sheet of the user is recorded, and when the number of the emotion regulation effect score sheets reaches a feedback threshold, the emotion determination model and the emotion regulation solution library are optimized using the emotion regulation effect score sheets.

2. The emotion regulation method based on VR equipment according to claim 1, characterized in that: Instruct users to explain their personal situation in a VR virtual environment and collect the user's emotional signal data set, including: Detect the working status of the VR head-mounted device, motion capture device, EEG device, ECG device, short-focus camera, high-definition camera and microphone worn by the user. If the working status is not working properly, remind the user to check the device wearing or report the device abnormality to the staff; Collecting the user's head electroencephalogram (EEG) signal through electrodes arranged on the user's scalp by the EEG device to obtain the EEG signal data set; Collecting the user's electrocardiogram (ECG) signal by the ECG device worn on the user's wrist to obtain the ECG signal data set; The motion sensors of the motion capture device disposed on the back of the user's hands and feet collect the user's motion signals to obtain the body node motion signal data set; The eye image data set is obtained by collecting images of both eyes of the user through the short-focus camera arranged inside the VR head mounted device; The facial expression image dataset is obtained by collecting the user's facial image through the high-definition camera placed directly opposite the user's head; The microphone disposed on the side of the VR head mounted device collects the user's voice information to obtain the tone signal dataset and the volume signal dataset; The keyword detection signal data set is obtained by matching the voice information with a preset keyword database.

3. The emotion regulation method based on VR equipment according to claim 1, characterized in that: Performing feature extraction and feature fusion on the emotion signal data set to obtain multi-dimensional fusion emotion features, including: Extracting statistical features, activity features, mobility features, complexity features, differential entropy features, power spectrum density features, and wavelet transform features from the EEG signal data set to obtain an EEG signal feature set; Extracting RR interval features, heart rate features, QRS duration features, and power spectrum density features from the ECG signal data set to obtain an ECG signal feature set; Extracting fundamental frequency features, pitch features, audio energy features, short-time zero-crossing rate features, intonation amplitude features, and intonation change rate features from the intonation signal data set to obtain an intonation signal feature set; Linear acceleration, angular velocity, displacement and speed are extracted from the body node motion signal data set to obtain a body node motion feature set.

4. The emotion regulation method based on VR equipment according to claim 1, characterized in that: Extracting and fusing features of the emotion signal data set to obtain multi-dimensional fusion emotion features also includes: The eye image data set is input into a preset feature extraction unit to obtain a feature map; the calculation formula of the feature map is: in, I t is the characteristic graph of the current cycle; t (i, j) is the value of the feature map at (i, j); H, W and C are the sizes of the convolution kernel; I is the image data to be identified that is randomly generated by the feature extraction unit t The weight matrix of (i,j); I is the image data to be identified that is randomly generated by the feature extraction unit t The base coefficient of (i,j); The feature map is input into a preset nonlinear activation unit for nonlinear transformation to obtain a transformation result; the calculation formula of the transformation result is: Among them, Z t is the transformation result of the current cycle; Z t (i, j) is the value of the transformation result at (i, j); ReLU(·) is a nonlinear activation function; The transformation result is input into a preset feature downsampling unit for dimensionality reduction operation to obtain a feature downsampling result; the calculation formula of the feature downsampling result is: in, is the feature downsampling result; is the value of the feature downsampling result at (i, j); and are the sizes of the downsampling windows. The feature downsampling result is flattened to obtain the eye image feature set.

5. The emotion regulation method based on VR equipment according to claim 1, characterized in that: The emotion determination model includes: An input layer, used for normalizing the multi-dimensional fusion emotion features; An image feature extraction layer, used for extracting the image feature part of the multi-dimensional fusion emotion feature through three convolution modules with a convolution kernel size of 3*3, three maximum pooling modules with a convolution kernel size of 2*2, and one flattening module; A data feature extraction layer, used for extracting features of the time series data feature part in the multi-dimensional fusion emotion feature through two GRU modules containing 64 units and one fully connected layer containing 64 units; A feature fusion layer, used for normalizing the outputs of the image feature extraction layer and the data feature extraction layer; A fully connected layer, used for performing a nonlinear transformation on the output of the feature fusion layer through a first fully connected module including 128 units, a Dropout module, and a second fully connected module including 64 units; An output layer, used for performing probability analysis on the output of the fully connected layer through two classification modules to obtain the predicted emotion efficacy value and the emotion activation degree; The loss function in the emotion determination model training process is: L total =αL valence +βL arousal ;in, L total is the model training loss value; α and β are the emotion valence loss weight and emotion activation loss weight respectively; L valence , L arousal are the emotional valence loss value and emotional activation loss value respectively; y valence,i , are the true value of the i-th emotional valence and the predicted value of the i-th emotional valence respectively; y arousal,i , are the true value of the i-th emotion activation and the predicted value of the i-th emotion activation respectively; N is the total number of input samples.

6. The emotion regulation method based on VR equipment according to claim 1, characterized in that: The emotion regulation solution library includes: an emotion evaluation column, a music regulation solution column, a video regulation solution column, and a game regulation solution column.

7. The emotion regulation method based on VR equipment according to claim 1, characterized in that: After the emotion regulation is completed, the emotion regulation effect score sheet of the user is recorded. When the number of the emotion regulation effect score sheets reaches the feedback threshold, the emotion determination model and the emotion regulation solution library are optimized using the emotion regulation effect score sheets, including: Performing data cleaning on the training database of the emotion determination model, removing data samples whose effect evaluation in the emotion regulation effect scoring table is lower than the effective threshold, and obtaining an updated database; After replacing the regularization coefficient and the model parameter iteration step, the emotion determination model is trained again using the updated database; Social surveys and questionnaire statistics are conducted according to the emotion regulation effect scoring table to obtain survey results and statistical distribution data, and the emotion regulation program library is deleted and supplemented according to the survey results and the statistical distribution data.

8. An emotion regulation system based on VR equipment, characterized in that: An emotion regulation method based on a VR device as provided in claim 2, the system comprising: the VR head mounted device, the motion capture device, the EEG device, the ECG device, the short focus camera, the high definition camera and the microphone; The VR head-mounted device is fixed on the user's head; the motion sensors of the motion capture device are respectively attached to the back of the user's hands and feet; the EEG device is fixed on the user's scalp; the ECG device is fixed on the user's wrist; the short-focus camera and the microphone are respectively arranged on the inner side and the side of the VR head-mounted device; and the high-definition camera is arranged in front of the user.

Images