User mental health assessment method and system based on machine accompanying chat service, medium and processor

Abstract

The invention belongs to the technical field of psychological health detection, and discloses a user psychological health assessment method and system based on machine accompanying chat service, a medium and a processor. The method comprises the steps of obtaining accompanying chat data in a chat conversation between a user and an accompanying chat robot; processing the accompanying chat data, and extracting feature data in accompanying chat of the user, including dynamic facial features, facial brightness image features, voice intonation features and text features; inputting the feature data into an emotion calculation model obtained by training based on an improved MFRM model, and outputting an emotion intensity value; inputting the feature data into a personality calculation model obtained based on CNN-CAM model training, and outputting a personality influence value; calculating an emotion intensity value according to the emotion intensity value and the personality influence value; and generating a mental health assessment result corresponding to the user according to the emotion intensity value. The psychological problem of the user is found in time through the machine accompanying chat, a basis is provided for providing targeted service for the health-care hotel, and the health-care service quality of the user is improved.

Description

Technical Field

[0001] The present invention relates to the field of mental health detection technology, and in particular to a user mental health assessment method, system, medium and processor based on a machine chat service. Background Art

[0002] In the fast-paced modern society, people are facing various pressures and psychological problems are gradually increasing. Not only are the elderly prone to psychological problems such as loneliness and depression, but young people are also facing psychological problems such as depression and anxiety. The demand for mental health services is increasing.

[0003] Health hotels are a type of hotel that integrates health elements into hotel services to meet consumers' dual needs for health and wellness and leisure vacations. As an important means to improve the service quality and added value of health hotels, chat robots have great application potential. Chat robots can not only provide emotional companionship for users, but also evaluate users' mental health through analysis of chat content and facial expressions, timely discover potential psychological problems, and provide hotels with more targeted health service suggestions.

[0004] In view of this, a user mental health assessment method, system, medium and processor based on machine chat service are needed. Summary of the invention

[0005] In view of the problem in the prior art of using a chat robot to evaluate the user's mental health and provide more targeted health care service suggestions for hotels, the present invention provides a user mental health evaluation method, system, medium and processor based on a machine chat service, which can. The specific technical solution is as follows:

[0006] A user mental health assessment method based on a machine chat service, comprising:

[0007] Obtain the chat data in the chat dialogue between the user and the chat robot;

[0008] Process the chat data and extract the feature data of the user during the chat, including dynamic facial features, facial brightness image features, voice intonation features and text features;

[0009] Input the feature data into the emotion calculation model trained based on the improved MFRM model, and output the emotion intensity value;

[0010] The feature data is input into the personality calculation model trained based on the CNN-CAM model, and the personality influence value is output;

[0011] The emotional intensity value is calculated based on the emotional intensity value and the personality influence value;

[0012] Generate the user's corresponding mental health assessment result based on the emotion intensity value.

[0013] Preferably, the acquiring of the chat data in the chat dialogue between the user and the chat robot comprises:

[0014] Obtain the video data recorded by the camera device during the chat between the user and the chat robot, as well as the voice data recorded by the microphone device during the chat.

[0015] Preferably, the processing of the chat companion data to extract characteristic data of the user's chat companion includes:

[0016] Based on the facial area video of the companion chat data, the key facial feature points are detected and located through the dlib model, and the position changes of the key points between adjacent frames are analyzed, and the distance and angle information between the key points are calculated to obtain dynamic facial features;

[0017] According to the facial area video of the companion chat data, the facial brightness image features are obtained by converting the frames into the facial area grayscale image;

[0018] Calculate the average grayscale value of all pixels in the grayscale image of the facial area to obtain the facial brightness value;

[0019] According to the voice of the chat data, the voice is filtered to extract the fundamental frequency, volume and speaking speed to obtain the voice intonation features;

[0020] According to the text data of the companion chat data, the text data is processed through the constructed emotional vocabulary dictionary to extract text features related to emotions;

[0021] Based on the processed dynamic facial features, facial brightness values, voice intonation features and text features, feature data of the user during the chat is obtained.

[0022] Preferably, the step of inputting the feature data into the emotion calculation model trained based on the improved MFRM model and outputting the emotion intensity value comprises:

[0023] Integrate the dynamic facial features, facial brightness values, voice intonation features and text features of the feature data in a preset order to form a feature vector of multimodal information;

[0024] The feature vector of multimodal information is taken as input and input into the sentiment calculation model trained based on the improved MFRM model. The model performs sentiment calculation based on the fused feature vector and outputs the sentiment intensity value.

[0025] Preferably, the emotion computing model trained based on the improved MFRM model includes multiple input channels, corresponding to facial brightness values, dynamic facial features, speech intonation features and text features respectively; the emotion computing model trained based on the improved MFRM model includes CNN for processing video and image features, LSTM and GRU for processing speech and text sequence features, and CVL for cross-modal learning.

[0026] Preferably, the step of inputting the feature data into a personality calculation model trained based on the CNN-CAM model and outputting the personality influence value comprises:

[0027] The dynamic facial features and facial brightness image features are fused to form a feature vector of bimodal information;

[0028] The feature vector of bimodal information is used as input and input into the personality calculation model trained based on the CNN-CAM model. The model predicts personality traits based on the fused feature vector and outputs the personality influence value.

[0029] Preferably, the step of calculating the emotion intensity value according to the emotion intensity value and the personality influence value comprises:

[0030] The emotion intensity attenuation value and emotion change impact value are calculated based on the emotion intensity value. The specific calculation is as follows:

[0031] α(O i,t )=O i,t-1 ×e -λ

[0032] β(O i,t )=O i,t -O i,t-1

[0033] In the formula, α(O i,t ) is the emotion intensity attenuation value; β(O i,t ) is the impact value of emotional change; O i,t is the emotional intensity value of the i-th emotion at time t; e -λ is an exponential function with a decay law;

[0034] The emotional intensity value is calculated based on the emotional intensity attenuation value, emotional change impact value and personality impact value. The specific calculation is as follows:

[0035]

[0036] In the formula, I is the emotion intensity value; It is the personality influence value.

[0037] A user mental health assessment system based on a machine chat service, applied to the aforementioned user mental health assessment method based on a machine chat service, comprising:

[0038] A data acquisition unit, used to acquire the chat data in the chat dialogue between the user and the chat robot;

[0039] A data processing unit, used to process the chat data and extract feature data of the user during the chat, including dynamic facial features, facial brightness image features, voice intonation features and text features;

[0040] An emotion calculation unit, used for inputting feature data into an emotion calculation model trained based on the improved MFRM model, and outputting an emotion intensity value;

[0041] A personality calculation unit, used for inputting feature data into a personality calculation model obtained by training based on a CNN-CAM model, and outputting a personality influence value;

[0042] An emotion calculation unit, used for calculating the emotion intensity value according to the emotion intensity value and the personality influence value;

[0043] The mental health assessment unit is used to generate a mental health assessment result corresponding to the user according to the emotion intensity value.

[0044] A computer-readable storage medium, the computer-readable storage medium including a stored program, wherein when the program is run, the device where the computer-readable storage medium is located is controlled to execute the aforementioned user mental health assessment method based on a machine chat service.

[0045] A processor is used to run a program, wherein the program, when running, executes the aforementioned user mental health assessment method based on a machine chat service.

[0046] Compared with the prior art, the present invention has the following beneficial effects:

[0047] The present invention provides a user mental health assessment method based on a machine chat service, which obtains chat data, processes the data, extracts multi-dimensional features including dynamic face, facial brightness, voice intonation and text features, and considers that the reliability of facial expression (dynamic face) recognition is reduced in a dimly lit environment. At this time, voice data should be given a higher weight, and the user status is fully captured by adding facial brightness features. The improved MFRM model is used to calculate the emotion intensity value, which can comprehensively consider the influence of multiple factors on the quantified emotion. At the same time, the personality influence value is obtained based on the CNN-CAM model, taking into account the effect of personality on emotion. Then, the emotion intensity value is calculated based on the emotion and personality influence value, and the mental health assessment result is obtained by comparing it with the preset threshold. Through the above method, the user's potential psychological problems can be discovered in time, providing a basis for health care hotels and other places so that they can provide targeted services, which helps to improve user experience and health care service quality. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] In order to more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the following is a brief introduction to the drawings required for the specific embodiments or the description of the prior art. In all the drawings, similar elements or parts are generally identified by similar reference numerals. In the drawings, the elements or parts are not necessarily drawn according to the actual scale.

[0049] Figure 1 The present invention is a flow chart of the method.

[0050] Figure 2 It is a system principle diagram of the present invention. DETAILED DESCRIPTION

[0051] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0052] It should be understood that when used in this specification and the appended claims, the terms "include" and "comprises" indicate the presence of described features, integers, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or combinations thereof.

[0053] It should also be understood that the terms used in the present specification are only for the purpose of describing specific embodiments and are not intended to limit the present invention. As used in the present specification and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include plural forms.

[0054] It should be further understood that the term "and / or" used in the present description and the appended claims refers to and includes any and all possible combinations of one or more of the associated listed items.

[0055] See the following examples Figure 1 and Figure 2 .

[0056] The present application embodiment provides a user mental health assessment method based on a machine chat service, including:

[0057] Step S1, obtaining the chat data in the chat dialogue between the user and the chat robot;

[0058] Specifically, obtaining the chat data in the chat dialogue between the user and the chat robot includes:

[0059] Obtain the video data recorded by the camera device during the chat between the user and the chat robot, the voice data recorded by the microphone device during the chat, and the text recording system used to record the chat content.

[0060] Through camera equipment, microphone equipment and text recording system, all kinds of data of users in the process of chatting are collected in an all-round way. Video data records non-verbal information such as users' facial expressions and body language; voice data contains acoustic characteristics such as the tone, intonation and speed of the user's speech; text data directly reflects the content and thinking logic of the user's expression. This multi-source data collection method can comprehensively present the status of users in the process of chatting from multiple dimensions.

[0061] Step S2, processing the chat data, extracting the characteristic data of the user in the chat, including dynamic facial features, facial brightness image features, voice and tone features, and text features;

[0062] By extracting different types of features from video, voice and text data, feature extraction of multimodal data is achieved. Dynamic facial features can capture subtle changes in users' facial expressions, such as smiling, frowning, and shifting eyes, which are often intuitive manifestations of emotions; facial brightness image features are used to assist dynamic facial features. In a dimly lit environment, the reliability of facial expression recognition is reduced; voice and intonation features include acoustic features such as voice fundamental frequency, volume, and speech speed, which are closely related to people's emotions and psychological states; text features analyze chat content to mine users' emotional tendencies, concerns, and thinking patterns. By extracting multimodal features, we can have a more comprehensive understanding of users' psychological states.

[0063] Step S3, inputting the feature data into the emotion calculation model obtained by training based on the improved MFRM model, and outputting the emotion intensity value;

[0064] The improved MFRM model can fully integrate multimodal information such as dynamic facial features, facial brightness image features, voice intonation features, and text features, taking into account the interaction and weight of different features in emotional expression. Through comprehensive analysis of these features, the model can quantify the user's emotional state into a specific value, namely the emotional intensity value. This value can intuitively reflect the intensity of the user's current emotions and provide a quantifiable indicator for subsequent emotional analysis and evaluation.

[0065] Step S4, inputting the feature data into a personality calculation model trained based on the CNN-CAM model, and outputting a personality influence value;

[0066] The CNN-CAM model can mine the influence of the user's personality factors on their psychological state by learning and analyzing multimodal feature data. Personality is a relatively stable psychological characteristic of an individual, and different personality traits can affect people's cognition, emotions, and behavioral responses to things. By outputting the personality influence value, we can understand the role of the user's personality traits in his or her emotions and psychological state.

[0067] Everyone's personality is unique, and by calculating the personality impact value, we can achieve personalized analysis of the user's psychological state. Compared with the general psychological assessment method, this analysis based on individual personality characteristics can more accurately grasp the psychological characteristics of each user.

[0068] Step S5, calculating the emotion intensity value according to the emotion intensity value and the personality influence value;

[0069] Specifically, the calculation of the emotion intensity value according to the emotion intensity value and the personality influence value includes:

[0070] The emotion intensity attenuation value and emotion change impact value are calculated based on the emotion intensity value. The specific calculation is as follows:

[0071] α(O i,t )=O i,t-1 ×e -λ

[0072] β(O i,t )=O i,t -O i,t-1

[0073] In the formula, α(O i,t ) is the emotion intensity attenuation value; β(O i,t ) is the impact value of emotional change; O i,t is the emotion intensity value representing the i-th emotion at time t, which is the output result of step S3; e -λ is an exponential function with a decay law;

[0074] The emotional intensity value is calculated based on the emotional intensity attenuation value, emotional change impact value and personality impact value. The specific calculation is as follows:

[0075]

[0076] In the formula, I is the emotion intensity value; is the personality influence value, which is the output result of step S4.

[0077] Transfer equation between personality traits and emotional intensity:

[0078] H=P×K T

[0079] Where K is the mapping matrix between personality traits and emotions; H is the six-dimensional emotion space matrix, including happiness, sadness, surprise, fear, anger and disgust; P is the personality space matrix;

[0080] Transfer equation between personality traits and emotion intensity:

[0081] L=H×F

[0082] Among them, L is the emotional space matrix; F is the transfer matrix between personality traits and emotional intensity;

[0083] The influence function of different personality types on emotional intensity was obtained:

[0084] Φ(P)=[H p,1 ,H p,2 ,H p,3 ,H p,4 ,H p,5 ,H p,6 ]

[0085] In the formula, P refers to personality trait type; H p,1 ,H p,2,H p,3 ,H p,4 ,H p,5 ,H p,6 They correspond to the six emotions of happiness, sadness, surprise, fear, anger and disgust.

[0086] The calculation of the emotion intensity value combines the emotion intensity value and the personality influence value, taking into full account the impact of two important factors, emotion and personality, on the user's psychological state. The emotion intensity value reflects the intensity of the user's current emotional experience, while the personality influence value reflects the potential impact of the user's personality traits on his or her emotions. By combining these two factors, the obtained emotion intensity value can more comprehensively and accurately reflect the user's true emotional state.

[0087] Step S6: Generate a psychological health assessment result corresponding to the user according to the emotion intensity value.

[0088] Each user's psychological state and needs are different. Based on the different levels of emotional intensity values, health hotels can accurately grasp the user's mental health status. For users whose emotional intensity values ​​show mild anxiety, the hotel can arrange specific meditation courses or relaxing yoga activities for them to help them relieve anxiety; for users with low moods, warm social activities may be provided to encourage them to interact with other guests to improve their mood. This personalized service can better meet the specific needs of users and enhance their experience and comfort in the hotel.

[0089] For those users whose emotional intensity values ​​exceed the normal range and who have certain psychological problems, the psychological counseling and psychotherapy services provided by health hotels can provide them with timely support and help, helping users to better understand and manage their emotions, relieve psychological pressure, and gradually restore mental health.

[0090] By providing personalized services based on the results of mental health assessment, health hotels can allocate resources more reasonably. In the process of providing personalized services, health hotels can collect a large amount of users' mental health data and feedback information. These data are of great value for hotels to further optimize services and improve assessment methods.

[0091] In step S1 of this embodiment, a high-definition camera, a high-sensitivity microphone and a reliable text recording system are equipped on the chat robot. The camera is used to capture the user's facial image, the microphone is used to record the user's voice, and the text recording system is used to record the chat conversation content. When the user chats with the chat robot, the camera continuously captures the user's facial image at a certain frame rate (for example, 30 frames per second) to form video stream data. The microphone records the user's voice at a suitable sampling frequency (for example, 44.1kHz) and sampling accuracy (for example, 16 bits) to obtain audio data. At the same time, the text recording system accurately records the content of each chat conversation, including the text entered by the user and the chat robot's reply.

[0092] The chat robot collects the user's facial images, records the user's voice and the content of the chat conversation, and organizes the collected data in a certain format, such as saving the facial images as video files (such as MP4 format), recording the voice as audio files (such as WAV format), and organizing the chat conversation into text files (such as TXT format). And establish a network communication connection between the chat robot and the health hotel server. The chat robot encapsulates the sorted data into a request package according to the interface specified by the server. Then, the request package is sent to the server through the network. After receiving the request package, the server parses it, extracts the data, and stores it in the specified database or file system.

[0093] In step S2 of this embodiment, the processing of the chat data to extract characteristic data of the user's chat includes:

[0094] Processing of dynamic facial features:

[0095] Based on the facial area video of the chat data, the key facial feature points are detected and located through the dlib model, and the position changes of the key points between adjacent frames are analyzed. The distance and angle information between the key points are calculated to obtain dynamic facial features; among them, the facial dynamic features mainly capture the user's facial expressions.

[0096] Accurately separate the video stream containing the user's face from the chat data, import the video stream of the user's face and initialize the facial key point detection model in the dlib library. For each frame of the image, input it into the dlib model. Based on its internal algorithm and training data, the dlib model analyzes the input image, accurately detects and locates the key feature points of the face, which cover important parts such as eyes, eyebrows, nose, mouth, etc., a total of 68 (taking the common 68 key point model as an example). Each feature point is represented in the form of coordinates in the image, and these coordinates accurately define the location of the facial features. Process two adjacent frames of images in sequence to obtain the coordinates of the facial key points in each frame. Calculate the position changes of the corresponding key points between adjacent frames, and determine the moving direction and distance of the key points by calculating the coordinate difference. Based on the position information of these key points, calculate the distance and angle between the key points. For example, calculate the distance between the two eyes, the distance between the corner of the mouth and the tip of the nose, the tilt angle of the eyebrows, the opening and closing angle of the mouth, etc. The changes in these distances and angles can intuitively reflect the dynamic changes of the user's facial expressions. The calculated distance and angle information between key points of adjacent frames are sorted to form ordered feature vectors. These feature vectors constitute dynamic facial features for subsequent analysis and understanding of user facial expressions.

[0097] Facial brightness image feature processing process:

[0098] According to the facial area video of the companion chat data, the facial brightness image features are obtained by converting the frames into the facial area grayscale image;

[0099] For the facial area video in the companion chat data, each color image is also read frame by frame. Convert the color image to a grayscale image. For example, according to the sensitivity of the human eye to different colors, convert it into a grayscale value by weighted averaging. Perform necessary preprocessing on the converted grayscale image, such as noise reduction. Determine the boundary range of the facial area based on the facial key points previously detected by the dlib model. Using these key points as a reference, define a rectangular area or a more precise irregular area that contains the complete face. Intercept the facial area from the grayscale image to obtain a grayscale image containing only the face, so as to calculate the brightness value later. Traverse all the pixels in the intercepted grayscale image of the facial area. Accumulate the grayscale value of each pixel. Divide the accumulated sum by the total number of pixels in the facial area to obtain the average grayscale value of all pixels in the grayscale image of the facial area, which is the facial brightness value.

[0100] In a dimly lit environment, the reliability of facial expression recognition decreases. Using facial brightness value as a feature for model learning can reduce the impact of ambient brightness on the analysis of user emotions. Based on model learning, it can be expected that in a dimly lit environment, due to the reduced reliability of facial expression recognition, voice data will be given a higher weight. Such feature input can make the model's emotional calculation more accurate.

[0101] Speech and intonation feature processing process:

[0102] According to the voice of the chat data, the voice is filtered to extract the fundamental frequency, volume and speaking speed to obtain the voice intonation features;

[0103] Adaptive filtering is used to process speech data to remove background noise and other interfering signals. The base frequency of the filtered speech signal is extracted using algorithms such as the autocorrelation method or the cepstrum method. The speech signal is processed by framing, and the length of each frame is set according to the specific application scenario and algorithm requirements, for example, 20-40 milliseconds. For each frame of speech signal, its short-time energy (volume) is calculated. Endpoint detection is performed on the speech signal to determine the start and end positions of the speech, thereby segmenting each speech segment. The number of syllables or words in each speech segment is counted, and the speaking rate is calculated based on the length of the speech segment and the number of syllables or words obtained by counting.

[0104] The extracted fundamental frequency, volume, and speech rate are integrated to form a vector containing speech intonation features. This vector comprehensively reflects the pitch, volume, and speech rate of the user's voice.

[0105] Text feature processing process:

[0106] According to the text data of the companion chat data, the text data is processed through the constructed emotional vocabulary dictionary to extract text features related to emotions;

[0107] Clean and normalize the text data in the chat data. Remove punctuation marks, special characters, and extra spaces in the text. Select the appropriate word segmentation tool according to the language type of the text. You can use the word segmenter in NLTK (Natural Language Toolkit); for Chinese text, you can use tools such as Jieba word segmentation. Segment the preprocessed text into independent words or phrases.

[0108] At the same time, load the constructed emotional vocabulary dictionary, which contains a series of words related to different emotions, such as positive emotional words (such as "happy", "joyful", "satisfied") and negative emotional words (such as "sad", "angry", "disappointed"). Traverse the text after word segmentation and match each word or phrase with the vocabulary in the emotional vocabulary dictionary. Count the number of positive emotional words and negative emotional words that appear in the text. Calculate the proportion of positive emotional words and negative emotional words in the text as an important feature related to emotions.

[0109] Based on the processed dynamic facial features, facial brightness values, voice intonation features and text features, feature data of the user during the chat is obtained.

[0110] The present invention provides a user mental health assessment method based on a machine chat service, which obtains chat data, processes the data, extracts multi-dimensional features including dynamic face, facial brightness, voice intonation and text features, and considers that the reliability of facial expression (dynamic face) recognition is reduced in a dimly lit environment. At this time, voice data should be given a higher weight, and the user status is fully captured by adding facial brightness features. The improved MFRM model is used to calculate the emotion intensity value, which can comprehensively consider the influence of multiple factors on the quantified emotion. At the same time, the personality influence value is obtained based on the CNN-CAM model, taking into account the effect of personality on emotion. Then, the emotion intensity value is calculated based on the emotion and personality influence value, and the mental health assessment result is obtained by comparing it with the preset threshold. Through the above method, the user's potential psychological problems can be discovered in time, providing a basis for health care hotels and other places so that they can provide targeted services, which helps to improve user experience and health care service quality.

[0111] Specifically, in a preferred implementation of the present application, the inputting of feature data into the emotion calculation model obtained by training based on the improved MFRM model and outputting the emotion intensity value comprises:

[0112] Integrate the dynamic facial features, facial brightness values, voice intonation features and text features of the feature data in a preset order to form a feature vector of multimodal information;

[0113] The feature vector of multimodal information is taken as input and input into the sentiment calculation model trained based on the improved MFRM model. The model performs sentiment calculation based on the fused feature vector and outputs the sentiment intensity value.

[0114] In this embodiment, the improved MFRM model can be trained using the AMIGOS dataset by acquiring facial brightness values, dynamic facial features, voice intonation features, and text features in the AMIGOS dataset.

[0115] Construct an emotional computing model based on the improved MFRM model, and set multiple input channels corresponding to facial brightness values, dynamic facial features, voice intonation features, and text features. The CNN module is used to process video and image features, such as dynamic facial features and facial brightness values. CNN automatically extracts local features and abstract features in the image through convolutional layers and pooling layers, and learns the relationship between facial expressions and brightness information and emotions. The LSTM and GRU modules are used to process speech and text sequence features. LSTM and GRU can effectively capture long-term dependencies in sequence data, and learn the connection between voice intonation changes and text semantic information and emotions. The CVL module is responsible for cross-modal learning, fusing features from different modalities (video, voice, text), mining complementary information between multimodal features, and improving the accuracy of emotional computing. Determine the hyperparameters and loss function during the MFRM model training process;

[0116] According to the above, the training process of the improved MFRM model based on the AMIGOS dataset is as follows:

[0117] The first step is to integrate the preprocessed data into feature vectors of multimodal information in a preset order and input them into the corresponding input channels respectively;

[0118] In the second step, the data enters the model through various input channels, and is processed by modules such as CNN, LSTM / GRU and CVL, and finally outputs a sentiment intensity value prediction result.

[0119] Specifically, in a preferred embodiment of the present application, the inputting of feature data into a personality calculation model obtained by training based on the CNN-CAM model and outputting the personality influence value comprises:

[0120] The dynamic facial features and facial brightness image features are fused to form a feature vector of bimodal information;

[0121] The feature vector of bimodal information is used as input and input into the personality calculation model trained based on the CNN-CAM model. The model predicts personality traits based on the fused feature vector and outputs the personality influence value.

[0122] In this embodiment, the FirstImpressions data set is selected to train the CNN-CAM model.

[0123] First, build the model layers: create an input layer, stack multiple convolutional layers and pooling layers, and build the output layer;

[0124] The input layer receives the fused bimodal information feature vector, which is a vector composed of dynamic facial features and facial brightness image features.

[0125] The convolution layer is used to extract local features in the image. By setting different convolution kernel sizes and step sizes, different levels of features of the facial image can be captured. The pooling layer is used to reduce the data dimension and the amount of calculation while retaining important feature information. After the convolution layer and the pooling layer, a fully connected layer is added to convert the feature maps output by the convolution layer and the pooling layer into a vector form; the CAM module is introduced, and the class activation map (CAM) module is introduced at the appropriate position of the CNN part; the role of the CAM module is to generate a class activation map, and by analyzing the model's attention to different features, the key areas in the image related to personality traits are located.

[0126] Construct an output layer to output the personality trait prediction results. Since the personality influence value is predicted, the output layer can be a fully connected layer, which outputs a numerical value to represent the personality influence value.

[0127] Initialize the parameters of the constructed CNN-CAM model; select the loss function and optimizer:

[0128] The steps of training the CNN-CAM model to obtain the personality calculation model are as follows:

[0129] The first step is to obtain data containing user facial images and related information from the FirstImpressions dataset;

[0130] Step 2: Extract dynamic facial features and facial brightness image features based on the user's facial image and related information:

[0131] The third step is to divide the processed data set into training set, validation set and test set according to a certain ratio (such as 70% training set, 15% validation set, and 15% test set);

[0132] The fourth step is to input the bimodal information feature vector in the training set into the model. The data is forward propagated through the CNN layer, CAM module and output layer. The model predicts personality traits based on the input feature vector and outputs the predicted personality influence value.

[0133] The embodiment of the present application also provides a user mental health assessment system based on a machine chat service, which is applied to the aforementioned user mental health assessment method based on a machine chat service, including:

[0134] A data acquisition unit, used to acquire the chat data in the chat dialogue between the user and the chat robot;

[0135] A data processing unit, used to process the chat data and extract feature data of the user during the chat, including dynamic facial features, facial brightness image features, voice intonation features and text features;

[0136] An emotion calculation unit, used for inputting feature data into an emotion calculation model trained based on the improved MFRM model, and outputting an emotion intensity value;

[0137] A personality calculation unit, used for inputting feature data into a personality calculation model obtained by training based on a CNN-CAM model, and outputting a personality influence value;

[0138] An emotion calculation unit, used for calculating the emotion intensity value according to the emotion intensity value and the personality influence value;

[0139] The mental health assessment unit is used to generate a mental health assessment result corresponding to the user according to the emotion intensity value.

[0140] The technical effect of this embodiment is the same as the technical effect of a user psychological health assessment method based on a machine chatting service in an embodiment, and the functional explanations of the corresponding units are also the same and will not be repeated here.

[0141] An embodiment of the present application also provides a computer-readable storage medium, which includes a stored program, wherein when the program is running, the device where the computer-readable storage medium is located is controlled to execute the aforementioned user mental health assessment method based on machine chat service.

[0142] The technical effect of this embodiment is the same as the technical effect of a user psychological health assessment method based on a machine chat service in an embodiment, and will not be repeated here.

[0143] The present invention can be used in many general or special computer system environments or configurations, such as personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronic devices, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, and the like.

[0144] An embodiment of the present application also provides a processor, which is used to run a program, wherein the program, when running, executes the aforementioned user mental health assessment method based on a machine chat service.

[0145] The technical effect of this embodiment is the same as the technical effect of the user psychological health assessment method based on machine chatting service in embodiment 1, and will not be repeated here.

[0146] The processor in this embodiment may be a central processing unit (CPU), a controller, a microcontroller, or other data processing chips.

[0147] Those of ordinary skill in the art will appreciate that the units of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition of each example has been generally described in terms of function in the above description. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0148] In the embodiments provided by the present invention, it should be understood that the division of units is only a logical function division, and there may be other division methods in actual implementation, for example, multiple units can be combined into one unit, one unit can be split into multiple units, or some features can be ignored, etc.

[0149] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0150] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-0nlyMemory), random access memory (RAM, RandomAccessMemory), mobile hard disk, magnetic disk or optical disk, etc., which can store program code.

[0151] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein by equivalents. These modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present invention, and they should all be included in the scope of the claims and specification of the present invention.

Claims

1. A user psychological health assessment method based on machine chat service, characterized in that: include: Obtain the chat data in the chat dialogue between the user and the chat robot; Process the chat data and extract the feature data of the user during the chat, including dynamic facial features, facial brightness image features, voice intonation features and text features; Input the feature data into the emotion calculation model trained based on the improved MFRM model, and output the emotion intensity value; The feature data is input into the personality calculation model trained based on the CNN-CAM model, and the personality influence value is output; The emotional intensity value is calculated based on the emotional intensity value and the personality influence value; Generate the user's corresponding mental health assessment result based on the emotion intensity value.

2. According to claim 1, a user psychological health assessment method based on machine chat service is characterized in that: The acquisition of the chat data in the chat dialogue between the user and the chat robot includes: Obtain the video data recorded by the camera device during the chat between the user and the chat robot, as well as the voice data recorded by the microphone device during the chat.

3. According to claim 2, a user psychological health assessment method based on machine chat service is characterized in that: The processing of the chat companion data to extract characteristic data of the user's chat companion includes: Based on the facial area video of the companion chat data, the key facial feature points are detected and located through the dlib model, and the position changes of the key points between adjacent frames are analyzed, and the distance and angle information between the key points are calculated to obtain dynamic facial features; According to the facial area video of the companion chat data, the facial brightness image features are obtained by converting the frames into the facial area grayscale image; Calculate the average grayscale value of all pixels in the grayscale image of the facial area to obtain the facial brightness value; According to the voice of the chat data, the voice is filtered to extract the fundamental frequency, volume and speaking speed to obtain the voice intonation features; According to the text data of the companion chat data, the text data is processed through the constructed emotional vocabulary dictionary to extract text features related to emotions; Based on the processed dynamic facial features, facial brightness values, voice intonation features and text features, feature data of the user during the chat is obtained.

4. According to claim 1, a user psychological health assessment method based on machine chat service is characterized in that: The step of inputting the feature data into the emotion calculation model obtained by training the improved MFRM model and outputting the emotion intensity value comprises: integrating the dynamic facial features, facial brightness values, voice intonation features and text features of the feature data in a preset order to form a feature vector of multimodal information; The feature vector of multimodal information is taken as input and input into the sentiment calculation model trained based on the improved MFRM model. The model performs sentiment calculation based on the fused feature vector and outputs the sentiment intensity value.

5. According to claim 4, a user psychological health assessment method based on machine chat service is characterized in that: The emotion calculation model trained based on the improved MFRM model includes multiple input channels corresponding to facial brightness value, dynamic facial features, voice intonation features and text features respectively; The emotion computing model trained based on the improved MFRM model includes CNN for processing video and image features, LSTM and GRU for processing speech and text sequence features, and CVL for cross-modal learning.

6. According to claim 4, a user psychological health assessment method based on machine chat service is characterized in that: The inputting of the feature data into the personality calculation model obtained by training based on the CNN-CAM model and outputting the personality influence value comprises: fusing the dynamic facial features and the facial brightness image features to form a feature vector of bimodal information; The feature vector of bimodal information is used as input and input into the personality calculation model trained based on the CNN-CAM model. The model predicts personality traits based on the fused feature vector and outputs the personality influence value.

7. A user psychological health assessment method based on machine chat service according to claim 6, characterized in that: The calculation of the emotion intensity value according to the emotion intensity value and the personality influence value comprises: The emotion intensity attenuation value and emotion change impact value are calculated based on the emotion intensity value. The specific calculation is as follows: a(O i,t )=O i,t-1 ×e -λ β(The i,t )=The i,t -THE i,t-1 In the formula, α(O i,t ) is the emotion intensity attenuation value; β(O i,t ) is the impact value of emotional change; O i,t is the emotional intensity value of the i-th emotion at time t; e -λ is an exponential function with a decay law; The emotional intensity value is calculated based on the emotional intensity attenuation value, emotional change impact value and personality impact value. The specific calculation is as follows: In the formula, I is the emotion intensity value; It is the personality influence value.

8. A user mental health assessment system based on machine chat service, characterized in that: A user mental health assessment method based on a machine chat service as described in any one of claims 1 to 7, comprising: A data acquisition unit, used to acquire the chat data in the chat dialogue between the user and the chat robot; A data processing unit, used to process the chat data and extract feature data of the user during the chat, including dynamic facial features, facial brightness image features, voice intonation features and text features; An emotion calculation unit, used for inputting feature data into an emotion calculation model trained based on the improved MFRM model, and outputting an emotion intensity value; A personality calculation unit, used for inputting feature data into a personality calculation model obtained by training based on a CNN-CAM model, and outputting a personality influence value; An emotion calculation unit, used for calculating the emotion intensity value according to the emotion intensity value and the personality influence value; The mental health assessment unit is used to generate a mental health assessment result corresponding to the user according to the emotion intensity value.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored program, wherein when the program is running, the device where the computer-readable storage medium is located is controlled to execute a user mental health assessment method based on a machine chat service as described in any one of claims 1 to 7.

10. A processor, characterized in that: The processor is used to run a program, wherein when the program is run, a user mental health assessment method based on a machine chat service as described in any one of claims 1 to 7 is executed.

Images