A method of monitoring mental state based on virtual reality

Abstract

The present invention belongs to the field of psychological monitoring, and discloses a psychological state monitoring method based on virtual reality, including: S1, obtaining the heart rate of the treated person through a physiological monitoring device worn on the treated person; S2, based on the heart rate, using an adaptive update interval to update the number of images taken per second; S3, based on the number of images taken per second, obtaining images containing the treatment person's movements; S4, transmitting the obtained images to a target device. The present invention controls the change in the number of images taken per second through the change in the heart rate of the treated person, and updates the number of images taken per second once after each adaptive update interval, effectively reducing the number of images obtained when the movement changes slightly, and can timely obtain a sufficient number of images when the movement changes significantly, thereby facilitating the improvement of the efficiency of subsequent psychological analysis based on images.

Description

Technical Field

[0001] The present invention relates to the field of psychological monitoring, and in particular to a psychological state monitoring method based on virtual reality. Background Art

[0002] Virtual reality exposure therapy (VRET) is a psychotherapy method that combines specific stress scenarios in virtual reality with exposure therapy. It is a transformation form of traditional behavioral therapy and an alternative treatment form to classic real-life exposure therapy. Virtual reality integrates real-time computer graphics, body sensory sensing, and visual imaging technology to provide visitors with a virtual environment that is close to reality, immersive, and interactive.

[0003] In the process of using virtual reality exposure therapy, it is usually necessary to combine multiple types of sensors to record various data that can reflect the psychological state of the person being treated, including data on heart rate, movement, etc. When obtaining movement data, it is generally necessary to place a camera or other shooting device next to the person being treated, and use the shooting device to obtain images containing the body movements of the person being treated during the treatment process. In the prior art, the number of images taken per second by the shooting device is usually fixed, which results in too many images being obtained when the person being treated has little movement change, thereby affecting the efficiency of subsequent psychological state analysis based on images. Summary of the invention

[0004] The purpose of the present invention is to disclose a psychological state monitoring method based on virtual reality, and to solve the technical problem of how to more reasonably determine the number of images taken per second in the process of implementing exposure therapy using virtual reality technology, so as to achieve effective retention of action information while reducing the total number of images finally obtained after the treatment process is completed.

[0005] In order to achieve the above object, the present invention provides the following technical solutions:

[0006] The present invention provides a psychological state monitoring method based on virtual reality, comprising:

[0007] S1, obtaining the heart rate of the treated person through a physiological monitoring device worn on the treated person;

[0008] S2, based on the heart rate, updates the number of images taken per second using an adaptive update interval;

[0009] S3, acquiring an image including the action of the treatment personnel based on the number of images captured per second;

[0010] S4, transmitting the acquired image to the target device;

[0011] Among them, the number of images taken per second is updated using an adaptive update interval, including:

[0012] After the current update interval ends, calculate the duration of the next update interval and the number of images captured per second corresponding to the next update interval;

[0013] The number of images captured per second is calculated using the following formula:

[0014]

[0015] represents the number of images taken per second corresponding to the sth update interval, Indicated in the image and The total number of pixels that have changed, represents the total number of pixels in the image containing the therapist's actions, and They represent the images closest to kK and k in the shooting time among all the images obtained in the s-th update interval, k represents the end time of the s-1-th update interval, and K represents the time length of the s-1-th update interval; Indicates the time period The variance of all heart rates obtained, Indicates the time period The maximum value of all heart rates obtained, Indicates the number of settings, and represent image weight and heart rate weight respectively.

[0016] Preferably, the calculation process of the adaptive update interval includes:

[0017] Use the following formula to calculate the duration of the sth update interval:

[0018]

[0019] represents the length of the sth update interval, Indicates the time period The variance of the effective data coefficients of all images obtained, Indicates the time period The maximum value of the effective data coefficients of all images obtained, represents the weighted ratio, Indicates the set time length.

[0020] Preferably, the target device includes a host computer.

[0021] Preferably, The calculation process is as follows:

[0022] The first step is to calculate Each pixel in The grayscale difference of pixels with the same coordinates in ;

[0023] Respectively and The median coordinate is The gray value of the pixel is expressed as and , then the grayscale difference is ;

[0024] The second step is to judge Is the grayscale difference corresponding to each pixel in greater than the adaptive grayscale difference threshold? If so, the grayscale difference corresponding to The pixels in the set are stored in ;

[0025] The third step is to The total number of pixels in The value of .

[0026] Preferably, the process of obtaining the adaptive grayscale difference threshold includes:

[0027] for The pixel v in The center is v, and the radius is The pixels within the range are stored in the collection ;

[0028] Use the following formula to calculate the adaptive grayscale difference threshold:

[0029]

[0030] Represents the adaptive grayscale difference threshold corresponding to the pixel v, and denote the first threshold and the second threshold respectively, express The total number of pixels in represents the gray value of pixel i, express The average gray value of the pixels in express The maximum gray value of the pixel in .

[0031] Preferably, The calculation formula is as follows:

[0032]

[0033] Indicates the radius of the setting. Indicates that the pixel v is the center of the window. The set of pixels in the window, represents the image gradient value of pixel j, express The maximum value of the image gradient in .

[0034] Preferably, and The median coordinate is The process of obtaining the gray value of a pixel includes:

[0035] Respectively and Grayscale it to get and ;

[0036] Respectively and Filter and get and ;

[0037] Respectively and The median coordinate is The gray value of the pixel is expressed as and .

[0038] Preferably, before S1, the method further includes:

[0039] Wear the head-mounted virtual reality device on the head of the person being treated, and control the head-mounted virtual reality device to play the treatment video.

[0040] Preferably, after S4, the method further includes:

[0041] The image is recognized by the recognition model stored in the target device to obtain the type of action in the image.

[0042] Preferably, the recognition model comprises a pre-trained CNN model.

[0043] Beneficial effects:

[0044] Compared with the prior art, in the process of acquiring images of the body movements of the treated person through the shooting device, the present invention controls the change in the number of images shot per second by the change in the heart rate of the treated person, and updates the number of images shot per second once after each adaptive update interval, so that the greater the change in the movement of the treated person and the greater the change in the heart rate, the more images the shooting device shoots per second. The number of images obtained when the movement changes slightly is effectively reduced, and a sufficient number of images can be obtained in time when the movement changes significantly, which is conducive to improving the efficiency of subsequent psychological analysis based on images. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for describing the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0046] Figure 1 It is a schematic diagram of the psychological state monitoring method based on virtual reality of the present invention.

[0047] Figure 2 Schematic diagram of the wavelet denoising process. DETAILED DESCRIPTION

[0048] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0049] The present invention provides a psychological state monitoring method based on virtual reality, comprising:

[0050] S1, obtaining the heart rate of the treated person through a physiological monitoring device worn on the treated person;

[0051] S2, based on the heart rate, updates the number of images taken per second using an adaptive update interval;

[0052] S3, acquiring an image including the action of the treatment personnel based on the number of images captured per second;

[0053] S4, transmitting the acquired image to the target device;

[0054] Among them, the number of images taken per second is updated using an adaptive update interval, including:

[0055] After the current update interval ends, calculate the duration of the next update interval and the number of images captured per second corresponding to the next update interval;

[0056] After calculating the length of the next update interval, the next update interval begins;

[0057] The number of images captured per second is calculated using the following formula:

[0058]

[0059] represents the number of images taken per second corresponding to the sth update interval, Indicated in the image and The total number of pixels that have changed, represents the total number of pixels in the image containing the therapist's actions, and They represent the images closest to kK and k in the shooting time among all the images obtained in the s-th update interval, k represents the end time of the s-1-th update interval, and K represents the time length of the s-1-th update interval; Indicates the time period The variance of all heart rates obtained, Indicates the time period The maximum value of all heart rates obtained, Indicates the number of settings, and represent image weight and heart rate weight respectively.

[0060] In the process of obtaining images of the body movements of the treated person through the shooting device, the change in the number of images shot per second is controlled by the change in the heart rate of the treated person, and the number of images shot per second is updated once after each adaptive update interval, so that the greater the change in the movement of the treated person and the greater the change in the heart rate, the more images the shooting device shoots per second. The number of images obtained when the movement changes slightly is effectively reduced, and a sufficient number of images can be obtained in time when the movement changes significantly, which is conducive to improving the efficiency of subsequent psychological analysis based on images.

[0061] In addition, the present invention calculates the number of images taken per second based on different types of data, which can effectively improve the validity of the calculation results. Because if it is based on a single type of data, when a single type of data has a large error, then the calculated result will also have a large error. If it is based on two different types of data, even if a single type of data has an error, because another type of data is also considered in the calculation, therefore, the calculated result will be compared with a single type of data, the error is smaller.

[0062] Specifically, steps S2 to S4 can be implemented by a camera, a video camera or other photographing device.

[0063] Data transmission between the photographing device and the physiological monitoring device can be achieved by wired communication or wireless communication.

[0064] The wireless communication method includes Bluetooth communication.

[0065] The camera can be placed in front of the person being treated.

[0066] Specifically, the number is set to 60.

[0067] Specifically, the image weight and heart rate weight are and .

[0068] Preferably, the calculation process of the adaptive update interval includes:

[0069] Use the following formula to calculate the duration of the sth update interval:

[0070]

[0071] represents the length of the sth update interval, Indicates the time period The variance of the effective data coefficients of all images obtained, Indicates the time period The maximum value of the effective data coefficients of all images obtained, represents the weighted ratio, Indicates the set time length.

[0072] In the process of performing adaptive update interval, the present invention is obtained by comprehensive consideration of the effective data coefficient of the image and the heart rate. When the fluctuation degree of the effective data coefficient is greater and the fluctuation degree of the heart rate is greater, the update interval is smaller, so that more images containing actions can be obtained when the treated person makes effective actions; and when the fluctuation degree of the effective data coefficient is smaller and the fluctuation degree of the heart rate is smaller, the update interval is larger, which can avoid obtaining too many images when the action changes of the treated person are small.

[0073] Specifically, the value range of the weighted ratio is .

[0074] Preferably, the value of the weighted ratio is .

[0075] Specifically, the time length is set to 20 seconds.

[0076] Specifically, the effective data coefficient is calculated using the following formula:

[0077]

[0078] represents the effective data coefficients of image b, Represents the total number of pixels whose grayscale value is equal to k in the grayscale image corresponding to image b.

[0079] Specifically, the effective data coefficient is calculated by counting the number of pixels with different gray values, so that when the number of pixels with different gray values ​​is greater, that is, when the effective data of the image is greater, the number of pixels calculated is greater, so that the effective data contained in the image can be effectively represented.

[0080] Preferably, the target device includes a host computer.

[0081] Specifically, the host computer can be the equipment used by the doctor responsible for psychotherapy, including computers, tablets, mobile phones, etc.

[0082] By transmitting the image to a host computer, further calculations can be performed by the host computer.

[0083] Preferably, The calculation process is as follows:

[0084] The first step is to calculate Each pixel in The grayscale difference of pixels with the same coordinates in ;

[0085] Respectively and The median coordinate is The gray value of the pixel is expressed as and , then the grayscale difference is ;

[0086] The second step is to judge Is the grayscale difference corresponding to each pixel in greater than the adaptive grayscale difference threshold? If so, the grayscale difference corresponding to The pixels in the set are stored in ;

[0087] The third step is to The total number of pixels in The value of .

[0088] The present invention does not use a fixed threshold when obtaining the number of changed pixels, because under different lighting conditions, the fixed threshold has too low adaptability and cannot well identify the changed pixels. Therefore, the present invention uses an adaptive grayscale difference threshold to determine whether a pixel has changed, so that the total number of changed pixels in an image taken later than in an image taken earlier can be more accurately identified.

[0089] Preferably, the process of obtaining the adaptive grayscale difference threshold includes:

[0090] for The pixel v in The center is v, and the radius is The pixels within the range are stored in the collection ;

[0091] Use the following formula to calculate the adaptive grayscale difference threshold:

[0092]

[0093] Represents the adaptive grayscale difference threshold corresponding to the pixel v, and denote the first threshold and the second threshold respectively, express The total number of pixels in represents the gray value of pixel i, express The average gray value of the pixels in express The maximum gray value of the pixel in .

[0094] When calculating the adaptive grayscale difference threshold, the present invention uses two different thresholds for calculation, thereby realizing the adaptive change of the threshold. The larger the standard deviation of the grayscale values ​​of the pixels in the image, the larger the adaptive grayscale difference threshold. That is, the worse the uniformity of the illumination, the larger the threshold is used to determine whether a change has occurred, which is conducive to reducing the influence of noise and obtaining more accurate judgment results.

[0095] Specifically, the first threshold is 10 and the second threshold is 5.

[0096] Preferably, The calculation formula is as follows:

[0097]

[0098] Indicates the radius of the setting. Indicates that the pixel v is the center of the window. The set of pixels in the window, represents the image gradient value of pixel j, express The maximum value of the image gradient in .

[0099] The radius of the present invention is not fixed, because the uniformity of illumination of images taken under different conditions is different. Therefore, the larger the numerical difference between the image gradient values ​​of different pixel points, the larger the radius of the present invention is, so that the adaptive grayscale difference threshold can be calculated with reference to the pixel points in a larger range. On the one hand, it can avoid directly using a fixed threshold, and on the other hand, it can avoid calculating the radius with reference to too small a range, because the smaller the range, the greater the variance of the grayscale values ​​of the pixel points in the range, which can easily lead to the numerical process of the calculated radius, thereby resulting in the use of too many pixel points to calculate the adaptive grayscale difference threshold, affecting the efficiency of the calculation.

[0100] Preferably, and The median coordinate is The process of obtaining the gray value of a pixel includes:

[0101] Respectively and Grayscale it to get and ;

[0102] Respectively and Filter and get and ;

[0103] Respectively and The median coordinate is The gray value of the pixel is expressed as and .

[0104] Specifically, the image can be grayed out by using a weighted average method, a maximum value method, or the like.

[0105] Preferably, and Filter and get and ,include:

[0106] Wavelet denoising algorithm was used to and Filter and get and .

[0107] like Figure 2 As shown, the basic steps of the wavelet denoising algorithm include:

[0108] Decompose the image into high-frequency coefficients and low-frequency coefficients;

[0109] Perform threshold processing on high-frequency coefficients to remove the noise carried in the high-frequency coefficients;

[0110] The low-frequency coefficients and the high-frequency coefficients obtained after threshold processing are reconstructed by wavelet to obtain a denoised image.

[0111] Specifically, algorithms such as Gaussian filtering algorithm may be used for filtering.

[0112] Preferably, before S1, the method further includes:

[0113] Wear the head-mounted virtual reality device on the head of the person being treated, and control the head-mounted virtual reality device to play the treatment video.

[0114] Specifically, the treatment video corresponds to the condition of the mental illness of the person being treated, and the treatment video is a VR video.

[0115] For example, when the person being treated has a fear of heights:

[0116] Scenarios involving heights should be played, but with gradual increases in height and challenge.

[0117] Scenarios can include:

[0118] The view looking down from the lower floors.

[0119] Stand on the balcony and look out over the streets below.

[0120] Walk across a suspended bridge or mountain trail.

[0121] Sitting on the edge of a tall building, or even riding a Ferris wheel, etc.

[0122] The intensity of the fear-of-height stimulation in the video should be gradually increased according to the patient's adaptability, and the patient should avoid entering a high-stimulation environment at the beginning to prevent excessive stress response.

[0123] When the person being treated has a fear of flying:

[0124] Scenes from the therapy videos that may be shown include:

[0125] Security check and boarding gate before boarding.

[0126] The process of an airplane taking off and landing.

[0127] A scene of an airplane flying at high altitude, which may be accompanied by slight turbulence or the process of handling an emergency situation.

[0128] Preferably, after S4, the method further includes:

[0129] The image is recognized by the recognition model stored in the target device to obtain the type of action in the image.

[0130] Specifically, the types of movements include crossing arms, clenching hands, leaning forward, leaning back, etc.

[0131] Crossing your arms indicates a defensive, resistant, uneasy, distrustful, or rejecting state of mind.

[0132] Leaning forward indicates a mental state of interest, concentration, or active engagement.

[0133] Specifically, after the type of action is output, the doctor can comprehensively judge the psychological state of the person being treated during the treatment process based on the type of action and data including heart rate, blood pressure, etc. obtained by the physiological monitoring equipment.

[0134] Preferably, the recognition model comprises a pre-trained CNN model.

[0135] Specifically, the training process includes:

[0136] 1. Data preparation

[0137] The input data for action recognition is usually images containing various types of actions.

[0138] First, you need to prepare the data into a format suitable for training the model:

[0139] Data collection: Collect a large number of labeled video or image frame datasets that cover all target action types. Common datasets include UCF101, Kinetics, HMDB51, etc.

[0140] Data preprocessing:

[0141] Normalization: Normalize the image, such as adjusting the size (for example, 224x224 pixels) and grayscale normalization (such as adjusting the pixel value to the [0, 1] or [-1, 1] interval).

[0142] Data augmentation: Apply data augmentation techniques to increase the diversity and robustness of the data, such as rotation, cropping, flipping, adjusting brightness and contrast, etc.

[0143] Label encoding: Convert action type labels into a format suitable for model training (such as one-hot encoding).

[0144] 2. Model design

[0145] To design a suitable CNN model for action recognition, common choices include 2D or 3D convolution-based models.

[0146] 2D-CNN Model:

[0147] Use traditional 2D convolutional networks (such as ResNet, VGGNet, etc.) to process single-frame images or short time series clips.

[0148] Feature extraction: Extract features for each frame or clip, and then integrate these features through a recurrent neural network (such as LSTM) or a fully connected layer to perform action classification.

[0149] 3D-CNN Model:

[0150] Using 3D convolutional networks (such as C3D, I3D) to perform convolution in spatial and temporal dimensions can capture both spatial and temporal features.

[0151] When designing, it is necessary to consider the length of the input video clip, the size and stride of the convolution kernel, etc.

[0152] 3. Model training

[0153] The training process involves steps such as model initialization, forward propagation, loss calculation, backpropagation, and weight update:

[0154] Loss function selection:

[0155] Choose an appropriate loss function for training, such as cross entropy loss for multi-classification tasks.

[0156] Optimizer selection:

[0157] Select an optimizer to update the model's weight parameters. Common optimizers include SGD (with or without momentum), Adam, RMSprop, etc.

[0158] Forward propagation:

[0159] Input a training sample, perform calculations through each layer of the model, and obtain output predictions.

[0160] Loss calculation:

[0161] Calculate the loss between the output prediction and the true label.

[0162] Backward Propagation:

[0163] Based on the loss value, the gradient of the loss with respect to the model parameters is calculated using the chain derivative method.

[0164] Weight update:

[0165] Based on the calculated gradients, the optimizer is used to adjust the model parameters to minimize the loss function.

[0166] 4. Model evaluation and validation

[0167] During model training, the model needs to be evaluated regularly to check its performance and avoid overfitting:

[0168] Validation set:

[0169] The dataset is divided into a training set and a validation set. During the training process, the validation set is regularly evaluated.

[0170] Performance indicators:

[0171] Model performance was evaluated using metrics such as accuracy, F1-score, and confusion matrix.

[0172] Early Stopping:

[0173] When the loss of the validation set no longer decreases, stop training early to prevent overfitting.

[0174] 5. Model Optimization

[0175] Based on the evaluation results, perform model optimization and hyperparameter adjustment:

[0176] Hyperparameter Tuning:

[0177] Adjust the model’s hyperparameters (such as learning rate, batch size, convolution kernel size, number of layers, etc.) to improve the model’s performance.

[0178] Regularization:

[0179] Regularization methods (such as L2 regularization, Dropout) are used to reduce overfitting.

[0180] Learning rate scheduling:

[0181] Use a learning rate scheduler (such as learning rate decay) to dynamically adjust the learning rate to help the model converge faster.

[0182] The preferred embodiments of the present invention disclosed above are only used to help illustrate the present invention. The preferred embodiments do not describe all the details in detail, nor do they limit the invention to the specific implementation methods described. Obviously, many modifications and changes can be made according to the content of this specification. This specification selects and specifically describes these embodiments in order to better explain the principles and practical applications of the present invention, so that those skilled in the art can understand and use the present invention well. The present invention is limited only by the claims and their full scope and equivalents.

Claims

1. A method for monitoring mental state based on virtual reality, characterized in that: include: S1, obtaining the heart rate of the treated person through a physiological monitoring device worn on the treated person; S2, based on the heart rate, updates the number of images taken per second using an adaptive update interval; S3, acquiring an image including the action of the treatment personnel based on the number of images captured per second; S4, transmitting the acquired image to the target device; Among them, the number of images taken per second is updated using an adaptive update interval, including: After the current update interval ends, calculate the duration of the next update interval and the number of images captured per second corresponding to the next update interval; The number of images captured per second is calculated using the following formula: represents the number of images taken per second corresponding to the sth update interval, Indicated in the image and The total number of pixels that have changed, represents the total number of pixels in the image containing the therapist's actions, and They represent the images closest to kK and k in the shooting time among all the images obtained in the s-th update interval, k represents the end time of the s-1-th update interval, and K represents the time length of the s-1-th update interval; Indicates the time period The variance of all heart rates obtained, Indicates the time period The maximum value of all heart rates obtained, Indicates the number of settings, and represent image weight and heart rate weight respectively; The calculation process is as follows: The first step is to calculate Each pixel in The grayscale difference of pixels with the same coordinates in ; Respectively and The median coordinate is The gray value of the pixel is expressed as and , then the grayscale difference is ; The second step is to judge Is the grayscale difference corresponding to each pixel in greater than the adaptive grayscale difference threshold? If so, the grayscale difference corresponding to The pixels in the set are stored in ; The third step is to The total number of pixels in The value of The process of obtaining the adaptive grayscale difference threshold includes: for The pixel v in The center is v, and the radius is The pixels within the range are stored in the collection ; Use the following formula to calculate the adaptive grayscale difference threshold: Represents the adaptive grayscale difference threshold corresponding to the pixel v, and denote the first threshold and the second threshold respectively, express The total number of pixels in represents the gray value of pixel i, express The average gray value of the pixels in express The maximum gray value of the pixel in .

2. The method for monitoring mental state based on virtual reality according to claim 1, characterized in that: The calculation process of the adaptive update interval includes: Use the following formula to calculate the duration of the sth update interval: represents the length of the sth update interval, Indicates the time period The variance of the effective data coefficients of all images obtained, Indicates the time period The maximum value of the effective data coefficients of all images obtained, represents the weighted ratio, Indicates the set time length.

3. The method for monitoring mental state based on virtual reality according to claim 1, characterized in that: The target device includes the host computer.

4. The method for monitoring mental state based on virtual reality according to claim 1, characterized in that: The calculation formula is as follows: Indicates the radius of the setting. Indicates that the pixel v is the center of the window. The set of pixels in the window, represents the image gradient value of pixel j, express The maximum value of the image gradient in .

5. The method for monitoring mental state based on virtual reality according to claim 1, characterized in that: and The median coordinate is The process of obtaining the gray value of a pixel includes: Respectively and Grayscale it to get and ; Respectively and Filter and get and ; Respectively and The median coordinate is The gray value of the pixel is expressed as and .

6. The method for monitoring mental state based on virtual reality according to claim 1, characterized in that: Prior to S1, this also included: Wear the head-mounted virtual reality device on the head of the person being treated, and control the head-mounted virtual reality device to play the treatment video.

7. The method for monitoring mental state based on virtual reality according to claim 1, characterized in that: After S4, it also includes: The image is recognized by the recognition model stored in the target device to obtain the type of action in the image.

8. The method for monitoring mental state based on virtual reality according to claim 7, characterized in that: The recognition model includes a pre-trained CNN model.

Images