Mental health evaluation method and system based on virtual reality equipment and storage medium

Abstract

The invention discloses a psychological health assessment method and system based on virtual reality equipment, and a medium, the method comprises: acquiring multi-modal data of a virtual assessment scene, the multi-modal data comprising facial expression data, eye movement data, behavior data and voice data; the virtual evaluation scene is constructed by the virtual reality equipment; and analyzing the multi-modal data through a multi-modal fusion algorithm, and generating a mental health assessment result. According to the invention, the multi-modal data of the virtual scene is collected and analyzed, and the psychological state of the student is more accurately captured.

Description

Technical Field

[0001] The present invention relates to the field of smart teaching technology, and in particular to a mental health assessment method, system and medium based on virtual reality equipment. Background Art

[0002] In the traditional primary and secondary education system, psychological health assessment in primary and secondary schools is extremely important and can effectively assess the psychological health of primary and secondary school students. Traditional psychological assessment mainly relies on paper questionnaires, psychological interviews and other forms, which have the following problems:

[0003] 1. The assessment context is single and it is difficult to reflect students’ true psychological state.

[0004] 2. The data collection method is single and lacks dynamic interaction and real-time performance.

[0005] 3. There is a delay in the discovery and intervention of students’ mental health problems, which may result in missing the best time for intervention.

[0006] 4. Static questionnaire methods cannot capture behavioral and physiological responses in real time. Summary of the invention

[0007] The present invention provides a psychological health assessment method, system and medium based on virtual reality equipment to analyze multimodal data of virtual scenes, more accurately capture the psychological state of students, and break through the limitations of existing technologies.

[0008] In order to solve the above technical problems, the first aspect of the present invention discloses a method for evaluating mental health based on a virtual reality device, the method comprising:

[0009] Acquire multimodal data of a virtual assessment scene, wherein the multimodal data includes facial expression data, eye movement data, behavior data, and voice data; the virtual assessment scene is constructed by the virtual reality device;

[0010] The multimodal data is analyzed by a multimodal fusion algorithm to generate a mental health assessment result.

[0011] In some embodiments, analyzing the multimodal data by a multimodal fusion algorithm includes:

[0012] Extracting and concatenating feature vectors from the multimodal data, the feature vectors including facial expression features, eye movement features, speech features, and behavioral features;

[0013] Input the feature vector into a Transformer-based multimodal fusion model to output a multidimensional evaluation result; the multidimensional evaluation result includes expression score, anxiety index, attention concentration, and emotion regulation score;

[0014] The mental health assessment results are determined based on the multidimensional assessment results.

[0015] In some implementations, extracting facial expression features includes:

[0016] Extracting facial key points, wherein the facial key points include at least eyebrows, eyes, nose, mouth and facial contour;

[0017] Recognize facial expressions based on expression recognition algorithms; perform time series analysis on the facial expressions to extract emotion fluctuation curves;

[0018] Calculate the frequency of expression changes and emotional distribution, and generate an expression feature matrix based on time series;

[0019] Extract eye movement features, including:

[0020] Obtain the gaze point position, pupil size, gaze duration and gaze path, generate a gaze point distribution heat map, and calculate the gaze point concentration and eye movement frequency;

[0021] The gaze point distribution is analyzed by density clustering method, the gaze features of key areas are identified, and eye movement features are generated.

[0022] In some implementations, extracting speech features includes:

[0023] Extracting speech rate, volume fluctuation and fundamental frequency of the speech data;

[0024] The speech data is converted into text data, semantic information in the text data is extracted, the integrity and emotional word frequency of the speech data are evaluated, and speech features are generated.

[0025] In some embodiments, the expression score is calculated by the following formula:

[0026] for:

[0027] Among them, F i (t) is the expression feature matrix, w i is the weight of this expression;

[0028] The anxiety index is calculated by the following formula:

[0029] Anxiety index = expression score × first weight + eye movement frequency score × second weight + voice emotion fluctuation score × third weight + behavior score × fourth weight;

[0030] The concentration is calculated by the following formula:

[0031]

[0032] The emotion regulation score is calculated by the following formula:

[0033]

[0034] In some embodiments, the multimodal fusion model is trained using a total loss function, where the total loss function includes a classification task loss function and a regression task loss function; the total loss function is:

[0035]

[0036] Among them, α and β are the classification task weight and regression task weight respectively. is the classification task loss function, y i，c is the true label of the i-th sample in category c, is the predicted probability of the i-th sample in category c;

[0037] is the regression task loss function, y i is the true label of the i-th sample, is the predicted label of the i-th sample.

[0038] In some embodiments, the feature vector is input into a Transformer-based multimodal fusion model, and a multidimensional evaluation result is output, including:

[0039] Mapping the feature vector to a high-dimensional space, using sine and cosine functions to encode the positional relationship of the time series to obtain input features;

[0040] Modeling the time dependency of the input features based on a multi-head attention mechanism;

[0041] The classification or regression task is input through the fully connected layer and the average pooling layer, and the multi-dimensional evaluation results are output.

[0042] In some embodiments, the virtual assessment scenarios include classroom questioning scenarios, exam stress scenarios, and social conflict scenarios;

[0043] Also includes:

[0044] Generate a personalized emotion regulation training program based on the results of the mental health assessment.

[0045] According to a second aspect of the present invention, a mental health assessment system based on a virtual reality device is disclosed, comprising:

[0046] Virtual reality equipment, constructing a virtual assessment scene and collecting multimodal data of users in the virtual assessment scene; the multimodal data includes facial expression data, eye movement data, behavior data and voice data;

[0047] The psychological assessment module analyzes the multimodal data through a multimodal fusion algorithm to generate a psychological health assessment result.

[0048] In a third aspect, a computer storage medium is disclosed, characterized in that a computer program is stored thereon, and when the computer program is executed by a processor, the steps of any one of the above methods for psychological health assessment based on virtual reality equipment are implemented.

[0049] Compared with the prior art, the present invention has the following beneficial effects:

[0050] The present invention provides a psychological health assessment method, system, and medium based on virtual reality equipment. The immersive assessment scene constructed based on virtual reality equipment can simulate a real stress environment, so that the collected multimodal data is closer to the individual's psychological reaction in the actual situation, comprehensively analyzes the psychological state from multiple dimensions, and generates more objective and accurate psychological health assessment results. It can capture the individual's emotional fluctuations and behavioral characteristics in real time, provide a scientific basis for timely identification of psychological problems and optimization of intervention measures, and achieve a comprehensive, dynamic and accurate assessment of the individual's psychological health status. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1 A flowchart of a method for evaluating mental health based on virtual reality equipment provided by the present invention;

[0052] Figure 2 A schematic diagram of the process of step S2 provided by the present invention;

[0053] Figure 3 This is a schematic diagram of the structure of the mental health assessment system based on virtual reality equipment provided by the present invention. DETAILED DESCRIPTION

[0054] For better understanding and implementation, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0055] The terms "including" and "having" and any variations thereof in the embodiments of the present invention are intended to cover non-exclusive inclusions. For example, a process, method, system, product or apparatus comprising a series of steps or modules is not necessarily limited to those steps or modules explicitly listed, but may include other steps or modules not explicitly listed or inherent to these processes, methods, products or apparatus.

[0056] The embodiment of the present invention discloses a psychological health assessment method based on virtual reality equipment, which analyzes multimodal data of virtual scenes and captures students' psychological states more accurately.

[0057] Specifically, Figure 1 As shown, the method comprises the following steps:

[0058] Step S1, obtaining multimodal data of a virtual evaluation scene, wherein the multimodal data includes facial expression data, eye movement data, behavior data and voice data; the virtual evaluation scene is constructed by the virtual reality device;

[0059] The virtual assessment scene is constructed through virtual reality equipment. Virtual reality equipment is VR equipment, which supports facial expression capture and eye tracking. Users can log in to the VR device with real name or anonymously to enter the virtual assessment scene. Virtual assessment scenes may include but are not limited to classroom questioning scenes, exam pressure scenes and social conflict scenes. In the classroom questioning scene, the teacher asks questions randomly and observes the students' facial tension, the completeness of the answers and the eye movement trajectory. The exam pressure scene simulates the exam environment and records the students' facial expression changes, gaze distribution and answering performance under time-limited tasks. The social conflict scene simulates peer conflict and analyzes students' language emotions, eye movement transfer frequency and expression changes. The immersive assessment scene built based on virtual reality equipment can simulate a real pressure environment, so that the collected multimodal data is closer to the individual's psychological response in the actual situation, comprehensively analyzes the psychological state from multiple dimensions, and generates more objective and accurate mental health assessment results.

[0060] Multimodal data is collected through VR equipment, and the multimodal data includes facial expression data, eye movement data, behavior data and voice data. Facial expression data can be captured by a camera device. By taking images or videos, the user's micro-expression changes, such as raised eyebrows and depressed corners of the mouth, are collected, and the expression reflects emotional fluctuations. Through eye tracking technology, the eye movement data of students in the virtual assessment scene, including gaze point, gaze duration and eye movement trajectory, etc., are recorded to analyze attention distribution and emotional response. The student's voice data is extracted through speech recognition technology. The voice data includes the answer content and the user's personal voice during the virtual assessment scene. The voice data includes intonation, volume and speaking speed. Combined with language emotion analysis tools, the student's expression integrity and emotional tendency are evaluated, and the user's emotional state is further accurately analyzed.

[0061] Step S2: Analyze the multimodal data using a multimodal fusion algorithm to generate a mental health assessment result. The multimodal data fusion algorithm is used to integrate and analyze the facial expression, eye movement and voice data collected above with the behavioral data to generate a mental health assessment result.

[0062] Specifically, Figure 2As shown, step S2 includes:

[0063] Step S21, extracting and concatenating feature vectors in the multimodal data, wherein the feature vectors include expression features, eye movement features, speech features, and behavior features.

[0064] Among them, extracting facial expression features includes:

[0065] Extracting facial key points, wherein the facial key points include at least eyebrows, eyes, nose, mouth and facial contour;

[0066] Recognize facial expressions based on expression recognition algorithms; perform time series analysis on the facial expressions to extract emotion fluctuation curves;

[0067] Calculate the frequency of expression changes and emotional distribution, and generate an expression feature matrix based on time series.

[0068] The facial key points include eyebrows, eyes, nose, mouth, facial contour, etc., totaling 133 points. The expression recognition algorithm is used to identify seven basic emotions: joy, anger, worry, fear, disgust, surprise and neutral. The expression recognition algorithm can be an expression classification model based on a convolutional neural network or other learning model to recognize facial expressions. By classifying each detected facial image, an emotion label is obtained. In this application, the emotion labels are divided into 7 types, including joy, anger, worry, fear, disgust, surprise and neutral.

[0069] After each expression recognition, record the current expression and its corresponding timestamp. The timestamp can be marked by the current time or the serial number of each frame. The emotion fluctuation curve refers to the emotion intensity curve that changes over time. The emotion fluctuation curve is obtained by plotting the relationship between the emotion label at each time point and the time axis. Count the number of times the facial expression changes within a period of time. The frequency of expression changes can be determined by comparing the emotion labels of two consecutive frames of images. Calculate the emotion distribution by counting the frequency of occurrence of each emotion within the time period, and use a histogram or pie chart to show the proportion of different emotions.

[0070] The facial expressions, change frequency, and emotional distribution at each time point are extracted to form a matrix. The rows represent the time points, and the columns represent different facial features, resulting in an expression feature matrix. The expression feature matrix is ​​used to analyze the user's emotional change trend, so as to better evaluate the students' mental health in the virtual scene.

[0071] Extract eye movement features from multimodal data, including:

[0072] Obtain the gaze point position, pupil size, gaze duration and gaze path, generate a gaze point distribution heat map, and calculate the gaze point concentration and eye movement frequency;

[0073] The gaze point distribution is analyzed by density clustering method, the gaze features of key areas are identified, and eye movement features are generated.

[0074] Through eye tracking technology, students' gaze points, gaze duration and eye movement trajectory are recorded to analyze their attention distribution and emotional response. Virtual reality devices are equipped with eye trackers to record the gaze point position and pupil size by tracking eye movements. The gaze point position is usually provided by the sensor of the eye tracking device, usually expressed in image coordinates (x, y), indicating the position on the screen where the eyes are looking. Pupil size is used to reflect eye focus and emotional changes. Gaze duration refers to the time the eyes stay on a certain gaze point, which is calculated by detecting continuous gaze points, that is, points with the same x, y coordinates in the image coordinates. The gaze path is the trajectory of the user's eyes moving in the process of observing an object in the virtual assessment scene. It is usually formed by the continuous gaze point coordinates provided by the eye tracking device. The movement trajectory of the eyeball is displayed by drawing the connection path of the continuous gaze point coordinates.

[0075] By calculating the number of times or the duration of the eyes staying at each position during the evaluation process, the area where the eyes are most frequently fixed on is determined. The fixation point concentration measures the concentration of the eye movement distribution, that is, whether the fixation points are concentrated in certain areas. It is generally evaluated by standard deviation or entropy value. A smaller standard deviation or entropy value indicates concentrated eye movement, and a larger standard deviation or entropy value indicates dispersed eye movement. The eye movement frequency indicates the frequency of changes in the eye fixation point within a certain period of time. It is obtained by calculating the event frequency in the eye movement data, that is, the number of times the fixation point changes.

[0076] The distribution of gaze points is determined by density clustering method. A distance threshold and a minimum number of samples are set to find the concentrated areas of gaze points. The dense gaze point areas are grouped by measuring the density between gaze points, which are the areas that users pay the most attention to. The dense areas after clustering can be regarded as key areas. The above calculated or identified data are combined to generate eye movement features. The eye movement data can be fully analyzed to reveal the rules of users' eye gaze behavior in the virtual assessment scene and the degree of attention to key areas.

[0077] Extract speech features from multimodal data, including:

[0078] Extracting speech rate, volume fluctuation and fundamental frequency of the speech data;

[0079] The speech data is converted into text data, semantic information in the text data is extracted, the integrity and emotional word frequency of the speech data are evaluated, and speech features are generated.

[0080] The speaking rate is obtained by calculating the number of syllables or words per unit time. Specifically, the voice signal is framed, the number of syllables / words in each frame is identified, and the average number of syllables or words per unit time is calculated. The volume fluctuation is calculated by calculating the loudness of the voice data, and the audio intensity of each time frame can be calculated by short-time Fourier transform. The fundamental frequency is the main frequency representing the voice data, which is used to describe the pitch of the voice. The fundamental frequency is extracted by the autocorrelation method or Mel Frequency Cepstral Coefficient (MFCC).

[0081] Use deep learning models to convert voice data into text data, and use natural language processing (NLP) technology to extract semantic information from text data, including keyword extraction, sentiment analysis, and syntactic analysis. Evaluate the rationality of grammar and sentences through syntactic tree or dependency analysis. At the same time, extract emotion-related words from text data based on sentiment analysis models or sentiment dictionaries, calculate the frequency of emotion-related words in unit time, and obtain the emotion word frequency. Combine the above data to form voice features to better identify the mental health of users in virtual assessment scenarios.

[0082] In the virtual behavior scene, the behavior data includes the time it takes for the user to complete the question or the specified behavior, the error rate of the answer, and the frequency of behavior switching, which are combined with the time series to form the behavior characteristics. The facial expression characteristics, eye movement characteristics, voice characteristics and behavior characteristics are spliced ​​together to obtain the feature vector.

[0083] Step S22, inputting the feature vector into a Transformer-based multimodal fusion model, and outputting a multidimensional evaluation result; the multidimensional evaluation result includes expression score, anxiety index, attention concentration, and emotion regulation score;

[0084] By collecting and processing the participants' voice, facial expression, behavior and other modal data, feature vectors are obtained. The feature vectors will be input into the Transformer-based multimodal fusion model for comprehensive analysis and evaluation, and finally output multi-dimensional evaluation results. Based on the participants' voice performance, facial expression, emotional response and other dimensional data, their psychological state, attention level, emotional regulation ability and other psychological indicators are evaluated.

[0085] The multimodal fusion model uses the self-attention mechanism to perform weighted fusion and feature interaction on data from different modalities. It can effectively capture the correlation between different modalities and model time series data, such as emotional changes in speech or the evolution of facial expressions, to achieve more sophisticated and multi-dimensional analysis. Through a multi-layer neural network structure, it gradually extracts information such as the overall emotional state and psychological reactions of the participants, and generates evaluation results in multiple dimensions on this basis.

[0086] Specifically, they include:

[0087] Mapping the feature vector to a high-dimensional space, using sine and cosine functions to encode the positional relationship of the time series to obtain input features;

[0088] Modeling the time dependency of the input features based on a multi-head attention mechanism;

[0089] The classification or regression task is input through the fully connected layer and the average pooling layer, and the multi-dimensional evaluation results are output.

[0090] The feature vectors are mapped into a high-dimensional space to ensure that the features of each modality can be represented in a unified, high-dimensional space, thereby facilitating subsequent fusion and modeling.

[0091]

[0092] in, is a learnable parameter, X is a feature vector, H 0 is the feature representation of the feature vector.

[0093] Since the input feature vector usually has the nature of a time series, the time information needs to be encoded. The sine and cosine functions encode the positional relationship of the time series, and the positional information of the time step is explicitly added to the feature so that the model can perceive the order of the input features.

[0094] PE(t,2i)=sin(t / 10000 2i / d )

[0095] PE(t,2i+1)=cos(t / 10000 2i / d )

[0096] Among them, t is the position in the time series, i is the dimension index of the encoding vector, d is the total dimension of the encoding vector, PE is the position encoding, and the resulting input vector is:

[0097] Z 0 =H 0 +PE

[0098] The position encoding is added to the modal features of each time step to form a feature representation with temporal position information. This enables the model to understand the positional relationship in the time series and handle dependencies in time series data.

[0099] The multimodal fusion model will model features through a multi-head attention mechanism. The multi-head attention mechanism allows the model to perform attention calculations on input features in different subspaces in parallel. Specifically, the model will learn how to pay attention to the relationship between different modalities and time steps in multiple "heads", thereby capturing richer cross-modal information and temporal dependencies.

[0100] The multi-head attention mechanism can capture the dependencies between each time step based on the multimodal features of the input. For example, emotional changes in speech and emotional fluctuations in facial expressions can all be captured in the model through the self-attention mechanism and fused according to the time order.

[0101] The calculation formula for each head is:

[0102]

[0103] Where Q = Z l W Q is the query matrix, K = Z l W K is the bond matrix, V = Z l W V is the value matrix, W Q , W K , W V are weight matrices respectively, and Z is the multi-head attention output.

[0104] The fused multimodal features are processed by the fully connected layer and the average pooling layer. The fully connected layer further enhances the expressive power of the features by mapping the multidimensional features to a higher-dimensional space, so that the model can capture more complex nonlinear relationships. Two fully connected layers are used in this application, and the formula is:

[0105]

[0106] in, is the weight matrix; b1 and b2 are bias terms.

[0107] In the average pooling layer, the features on the time series are pooled. Average pooling calculates the average value of the features in each time period, thereby reducing the dimension of the features and focusing on the overall trend over a long period of time rather than the fluctuations at a single time point. Through the pooling operation, the model can capture the global characteristics of dynamic processes such as emotions and attention in a long time series.

[0108] The features processed by the fully connected layer and the pooling layer will be input into the final classification task or regression task. Multi-dimensional evaluation results are output according to the requirements of different tasks. In the virtual assessment scenario, the evaluation results usually include indicators such as expression score, anxiety index, concentration, and emotion regulation score. The expression score is generated through a comprehensive evaluation of the emotional concentration and emotional fluctuations of facial expressions, reflecting the intensity of the participant's emotional response.

[0109] The expression score is calculated by the following formula:

[0110] for:

[0111] Among them, F i (t) is the expression feature matrix, w i is the weight of this expression.

[0112] The anxiety index is based on the fundamental frequency changes and speech speed in the voice, as well as the anxiety characteristics of facial expressions, to assess the user's anxiety level. It is calculated using the following formula:

[0113] Anxiety index = expression score × first weight + eye movement frequency score × second weight + voice emotion fluctuation score × third weight + behavior score × fourth weight;

[0114] Among them, the eye movement frequency score is calculated by the number of eye movement switches and the total time. The number of eye movement switches is the total number of times the eyes switch from one fixation point to another in the total time. The higher the eye movement frequency score, the less focused the user is. The voice emotion fluctuation score can be calculated based on the fundamental frequency fluctuation, speech speed change, and emotion intensity fluctuation and the corresponding weights. The behavior score is determined by the error rate or task completion time.

[0115] The weight can be determined by adjusting the optimization parameters according to the training model. The anxiety index can be divided into: normal, mild anxiety, moderate anxiety, and severe anxiety.

[0116] For example, expression score: 0.78 (high), eye movement frequency score: 65% (medium), speech rate (200 words per minute, higher than normal)

[0117] Its anxiety index is 0.78×0.4+0.6×0.3+0.2×0.55+0.1×0.5=0.655

[0118] The first weight is 0.4, the second weight is 0.3, the third weight is 0.2, and the fourth weight is 0.1. 0.655 indicates moderate anxiety, and emotion regulation training is recommended.

[0119] Attention concentration takes into account both speech and eye movement data, and gives the degree of concentration during task execution by analyzing the distribution and concentration of participants' attention. It is calculated using the following formula:

[0120]

[0121] The emotion regulation score measures the participant's ability to regulate their emotions when facing external stimuli, and is usually evaluated based on characteristics such as the stability of emotional fluctuations and emotional recovery ability. It is calculated using the following formula:

[0122]

[0123] The duration of positive emotions and the total time after the occurrence of negative emotions can be determined through facial data. After the expression label is recognized, the duration of the expression is determined. Among them, joy is a positive emotion, anger, fear, sadness, and disgust are negative emotions, and surprise is a neutral emotion.

[0124] These multi-dimensional evaluation results provide detailed and comprehensive psychological analysis data for the virtual assessment system, which can be used to dynamically monitor the participants' emotional state, psychological stress, cognitive load, etc. It can not only be used for mental health monitoring, but also help develop more accurate interactive experiences in virtual environments, such as practical applications in scenarios such as psychotherapy, education and training, and user experience optimization.

[0125] The multimodal fusion model is trained by using a mental health dataset and a total loss function. A mental health dataset of 1,000 students is used, and a multimodal feature time series (100 time steps) is collected from each student. The mental health dataset is divided into 80% training set and 20% validation set. During the training process, the encoder is 4 layers, the model dimension is 128, and the input data is mapped to a 128-dimensional space for processing. The intermediate layer dimension of the feed-forward network (FFN) is 256, and the number of attention heads of the multi-head attention mechanism is 8. The multi-head attention mechanism divides the input features into multiple subspaces, and each head independently calculates the attention score in different subspaces, and finally splices the results. The Adam optimizer is used with a learning rate of 1e-4. Each epoch jointly optimizes the classification and regression tasks. During the training process, the multimodal fusion model is obtained until the validation set accuracy reaches 90% and the mean square error is 0.05.

[0126] The total loss function includes the classification task loss function and the regression task loss function; the total loss function is:

[0127]

[0128] Among them, α and β are the classification task weight and regression task weight respectively. is the classification task loss function, y i，c is the true label of the i-th sample in category c, is the predicted probability of the i-th sample in category c;

[0129] is the regression task loss function, y i is the true label of the i-th sample, is the predicted label of the i-th sample.

[0130] Step S23: Determine the mental health assessment result according to the multi-dimensional assessment result.

[0131] The multidimensional evaluation results calculated by the multimodal data fusion model will be further comprehensively analyzed, and weights will be assigned according to the importance of each evaluation indicator. The weighted sum will be taken to obtain the final mental health score.

[0132] In some embodiments, the mental health assessment result = w1×expression score + w2×anxiety index + w3×attention concentration + w4×emotion regulation score. w1, w2, w3, w4 are the weights of the assessment results of each dimension. The weights are set according to the specific mental health assessment scenario. For example, when assessing the anxiety level, the weights of the anxiety index and expression score may be higher; when assessing the concentration, the weight of the attention score may be higher.

[0133] In some embodiments, the expression score weight is 0.3, the anxiety index weight is 0.4, the attention concentration weight is 0.2, and the emotion regulation score weight is 0.1.

[0134] According to the results of mental health assessment, mental health status is divided into different levels, such as normal, mild problems, moderate problems, and severe problems. Predefined intervals are used for classification, for example: 0.8-1.0: good mental health; 0.6-0.8: mild mental problems; 0.4-0.6: moderate mental problems; 0.0-0.4: severe mental problems.

[0135] Step S23: generating a personalized emotion regulation training program according to the determined mental health assessment results.

[0136] The results of the mental health assessment include scores of multiple dimensions such as expression score, anxiety index, concentration, emotion regulation score, etc., and the comprehensive score is combined to divide the mental health status level. The problem points of each dimension will serve as the core of the personalized emotion regulation training program. For example, when the anxiety index is high, it helps users relieve emotional tension and reduce psychological pressure. When the concentration is low, it helps users improve their concentration and reduce distracting behaviors. When the ability to regulate emotions is insufficient, it enhances the user's ability to calm down under stress or negative emotions.

[0137] Personalized emotion regulation training programs may include breathing, meditation or eye movement attention tracking, short-term focus training, etc. In some embodiments, multiple trainings are combined into a feasible plan and provided to the user intuitively. For example, 5 minutes of breathing training and 10 minutes of eye movement focus training are completed within a certain time. Help users improve their mental state and gradually achieve healthy emotion management ability and psychological resilience.

[0138] What's more, the results of mental health assessment will be used to generate a detailed mental health report, including: facial expression curve (tension-relaxation change trend), gaze point heat map and eye movement trajectory map, voice emotion fluctuation curve and answer content analysis summary, generated "personalized emotion regulation training program", and provide a standardized mental health rating scale for comparison with the GAD-7 anxiety scale.

[0139] Based on the same inventive idea, Figure 3 As shown, the present application provides a mental health assessment system based on virtual reality equipment, including:

[0140] Virtual reality equipment, constructing a virtual assessment scene and collecting multimodal data of users in the virtual assessment scene; the multimodal data includes facial expression data, eye movement data, behavior data and voice data;

[0141] The psychological assessment module analyzes the multimodal data through a multimodal fusion algorithm to generate a psychological health assessment result.

[0142] The processing method of this system can refer to the description of the above method and will not be repeated here.

[0143] The present invention provides a method, system and medium for psychological health assessment based on virtual reality equipment. The immersive assessment scene constructed based on virtual reality equipment can simulate a real stress environment, so that the collected multimodal data is closer to the individual's psychological reaction in the actual situation, comprehensively analyzes the psychological state from multiple dimensions, and generates more objective and accurate psychological health assessment results. It can capture the individual's emotional fluctuations and behavioral characteristics in real time, provide a scientific basis for timely identification of psychological problems and optimization of intervention measures, and achieve a comprehensive, dynamic and accurate assessment of the individual's psychological health status.

[0144] The present invention also provides a device, which may include: a memory storing executable program code;

[0145] a processor coupled to the memory;

[0146] A transceiver used to communicate with other devices or communication networks and to receive or send network messages;

[0147] A bus used to connect memory, processors, and transceivers for internal communication.

[0148] The transceiver receives messages transmitted over the network and passes them to the processor via the bus. The processor calls the executable program code stored in the memory through the bus for processing, and passes the processing results to the transceiver via the bus for sending, thereby realizing the method provided in the embodiment of the present application.

[0149] An embodiment of the present application also provides a non-transitory machine-readable storage medium, on which an executable program is stored. When the executable program is executed by a processor, the processor executes the method provided in the above embodiment.

[0150] An embodiment of the present invention discloses a computer-readable storage medium storing a computer program for electronic data exchange, wherein the computer program enables a computer to execute the described method.

[0151] An embodiment of the present invention discloses a computer program product, which includes a non-transitory computer-readable storage medium storing a computer program, and the computer program is operable to enable a computer to execute the described method.

[0152] The embodiments described above are only illustrative, wherein the modules described as separate components may or may not be physically separated, and the components displayed as modules may or may not be physical modules, i.e., they may be located in one place, or they may be distributed on multiple network modules. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Those of ordinary skill in the art may understand and implement it without paying creative labor.

[0153] Through the specific description of the above embodiments, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on such an understanding, the above technical solution can be essentially or partly contributed to the prior art in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, and the storage medium includes a read-only memory (ROM), a random access memory (RAM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), a one-time programmable read-only memory (OTPROM), an electronically erasable rewritable read-only memory (EEPROM), a compact disc (CD-ROM) or other optical disc storage, magnetic disk storage, magnetic tape storage, or any other computer-readable medium that can be used to carry or store data.

[0154] Finally, it should be noted that the embodiments disclosed in the present invention are only preferred embodiments of the present invention, which are only used to illustrate the technical solutions of the present invention rather than to limit them. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A psychological health assessment method based on virtual reality equipment, characterized in that: include: Acquire multimodal data of a virtual assessment scene, wherein the multimodal data includes facial expression data, eye movement data, behavior data, and voice data; The virtual evaluation scene is constructed by the virtual reality device; The multimodal data is analyzed by a multimodal fusion algorithm to generate a mental health assessment result.

2. The method for evaluating mental health based on virtual reality equipment according to claim 1, characterized in that: Analyzing the multimodal data by a multimodal fusion algorithm includes: Extracting and concatenating feature vectors from the multimodal data, the feature vectors including facial expression features, eye movement features, speech features, and behavioral features; Input the feature vector into a Transformer-based multimodal fusion model to output a multidimensional evaluation result; the multidimensional evaluation result includes expression score, anxiety index, attention concentration, and emotion regulation score; The mental health assessment results are determined based on the multidimensional assessment results.

3. The method for evaluating mental health based on virtual reality equipment according to claim 2, characterized in that: Extract facial expression features, including: Extracting facial key points, wherein the facial key points include at least eyebrows, eyes, nose, mouth and facial contour; Recognize facial expressions based on expression recognition algorithms; perform time series analysis on the facial expressions to extract emotion fluctuation curves; Calculate the frequency of expression changes and emotional distribution, and generate an expression feature matrix based on time series; Extract eye movement features, including: Obtain the gaze point position, pupil size, gaze duration and gaze path, generate a gaze point distribution heat map, and calculate the gaze point concentration and eye movement frequency; The gaze point distribution is analyzed by density clustering method, the gaze features of key areas are identified, and eye movement features are generated.

4. The method for evaluating mental health based on virtual reality equipment according to claim 2, characterized in that: Extract speech features, including: Extracting speech rate, volume fluctuation and fundamental frequency of the speech data; The speech data is converted into text data, semantic information in the text data is extracted, the integrity and emotional word frequency of the speech data are evaluated, and speech features are generated.

5. The method for evaluating mental health based on virtual reality equipment according to claim 3, characterized in that: The expression score is calculated by the following formula: for: Among them, F i (t) is the expression feature matrix, w i is the expression weight; The anxiety index is calculated by the following formula: Anxiety index = expression score × first weight + eye movement frequency score × second weight + voice emotion fluctuation score × third weight + behavior score × fourth weight; The concentration is calculated by the following formula: The emotion regulation score is calculated by the following formula:

6. The method for evaluating mental health based on virtual reality equipment according to claim 2, characterized in that: The multimodal fusion model is trained using a total loss function, where the total loss function includes a classification task loss function and a regression task loss function; the total loss function is: Among them, α and β are the classification task weight and regression task weight respectively. is the classification task loss function, y i，c is the true label of the i-th sample in category c, is the predicted probability of the i-th sample in category c; is the regression task loss function, y i is the true label of the i-th sample, is the predicted label of the i-th sample.

7. The method for evaluating mental health based on virtual reality equipment according to claim 2, characterized in that: The feature vector is input into a Transformer-based multimodal fusion model to output a multi-dimensional evaluation result, including: Mapping the feature vector to a high-dimensional space, using sine and cosine functions to encode the positional relationship of the time series to obtain input features; Modeling the time dependency of the input features based on a multi-head attention mechanism; The classification or regression task is input through the fully connected layer and the average pooling layer, and the multi-dimensional evaluation results are output.

8. The method for evaluating mental health based on virtual reality equipment according to claim 1, characterized in that: The virtual assessment scenarios include classroom questioning scenarios, exam stress scenarios, and social conflict scenarios; Also includes: Generate a personalized emotion regulation training program based on the results of the mental health assessment.

9. A mental health assessment system based on virtual reality equipment, characterized in that: include: Virtual reality equipment, constructing a virtual assessment scene and collecting multimodal data of users in the virtual assessment scene; the multimodal data includes facial expression data, eye movement data, behavior data and voice data; The psychological assessment module analyzes the multimodal data through a multimodal fusion algorithm to generate a psychological health assessment result.

10. A computer storage medium, characterized in that: A computer program is stored thereon, and when the computer program is executed by a processor, the steps of the mental health assessment method based on virtual reality equipment as claimed in any one of claims 1 to 8 are implemented.

Images