Psychological state detection method and apparatus, readable storage medium, and terminal device

Abstract

A psychological state detection method and apparatus, a computer readable storage medium, and a terminal device (6). The psychological state detection method comprises: acquiring multi-modal information of a user during a dialogue (S101); and processing the multi-modal information by using a preset psychological state detection model to obtain a psychological state detection result of the user (S102), wherein the psychological state detection model is a deep learning model obtained by training a preset training sample set, the training sample set comprises a preset number of training samples, and each training sample comprises a multi-modal information sample and a corresponding expected detection result. By means of the psychological state detection method, the multi-modal information of the user can be acquired when having a dialogue with the user, and the multi-modal information is processed by using the psychological state detection model to obtain the psychological state detection result of the user, so that the accuracy of the psychological state detection result is improved.

Description

A method, device, readable storage medium and terminal device for detecting mental state

[0001] This application claims priority to the Chinese patent application filed with the China Patent Office on June 30, 2023, with application number 202310803855.7 and invention name “A method, device, readable storage medium and terminal device for detecting a mental state”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The present application belongs to the field of artificial intelligence technology, and in particular relates to a psychological state detection method, apparatus, computer-readable storage medium, and terminal device. Background Art

[0003] In the existing technology, psychological state detection is mainly carried out through questionnaires, but questionnaires cannot accurately determine the psychological state of the subject because the content that can be obtained through a single text information such as a questionnaire is small and not accurate enough, which leads to a low accuracy rate of the psychological state detection results. Technical issues

[0004] In view of this, embodiments of the present application provide a method, apparatus, computer-readable storage medium, and terminal device for detecting a mental state to solve the problem of low accuracy in existing mental state detection. Technical Solutions

[0005] A first aspect of the embodiments of the present application provides a method for detecting a mental state, which may include:

[0006] Obtain multimodal information of users during the conversation;

[0007] Processing the multimodal information using a preset mental state detection model to obtain a mental state detection result of the user;

[0008] The mental state detection model is a deep learning model obtained by training with a preset training sample set, wherein the training sample set includes a preset number of training samples, and each training sample includes a multimodal information sample and a corresponding expected detection result.

[0009] In a specific implementation of the first aspect, obtaining multimodal information of the user during the conversation includes:

[0010] Generate a conversation text with the user using a preset conversation model;

[0011] Performing voice conversion on the conversation text to obtain a conversation voice with the user;

[0012] Performing talking head synthesis and body posture synthesis based on the conversational speech to obtain a multimodal digital human video;

[0013] A conversation is conducted with the user through the multimodal digital human video, and multimodal information of the user during the conversation is obtained.

[0014] In a specific implementation of the first aspect, the training process of the dialogue model includes:

[0015] Obtain a pre-trained general language processing model and a preset psychological counseling dialogue dataset;

[0016] The general language processing model is trained using the psychological counseling dialogue dataset to obtain the dialogue model.

[0017] In a specific implementation of the first aspect, the training process of the mental state detection model includes:

[0018] Obtaining the training sample set;

[0019] The initial deep learning model is trained with the multimodal information sample of each training sample in the training sample set as input and the expected detection result of each training sample in the training sample set as expected output to obtain the mental state detection model.

[0020] In a specific implementation of the first aspect, after processing the multimodal information using a preset mental state detection model to obtain a mental state detection result of the user, the method further includes:

[0021] Generate multimodal feedback material corresponding to the mental state detection result, and display the multimodal feedback material to the user.

[0022] In a specific implementation of the first aspect, generating multimodal feedback material corresponding to the mental state detection result includes:

[0023] Processing the mental state detection result using a preset material generation model to obtain music feedback material corresponding to the mental state detection result; wherein the material generation model is a probabilistic generation model trained with a preset material training sample set, the material training sample set including a preset number of training samples, each training sample including a mental state detection result sample and corresponding expected music feedback material;

[0024] The digital human dance movements are synthesized according to the music feedback material to obtain the multimodal feedback material.

[0025] In a specific implementation of the first aspect, the multimodal information includes at least two of the following: visual information, voice information, text information, and physiological signal information.

[0026] A second aspect of the embodiments of the present application provides a mental state detection device, which may include:

[0027] Information acquisition module, used to obtain multimodal information of users during the conversation;

[0028] An information processing module, configured to process the multimodal information using a preset mental state detection model to obtain a mental state detection result of the user;

[0029] The mental state detection model is a deep learning model obtained by training with a preset training sample set, wherein the training sample set includes a preset number of training samples, and each training sample includes a multimodal information sample and a corresponding expected detection result.

[0030] In a specific implementation of the second aspect, the acquisition module includes:

[0031] A generation submodule, configured to generate a conversation text with the user using a preset conversation model;

[0032] A conversion submodule, configured to convert the conversation text into speech to obtain a speech of the conversation with the user;

[0033] a synthesis submodule, configured to perform talking head synthesis and body posture synthesis based on the conversational speech to obtain a multimodal digital human video;

[0034] The dialogue submodule is used to conduct a dialogue with the user through the multimodal digital human video and obtain multimodal information of the user during the dialogue.

[0035] In a specific implementation of the second aspect, the mental state detection device further includes:

[0036] The acquisition module is used to obtain the pre-trained general language processing model and the preset psychological counseling dialogue dataset;

[0037] A training module is used to train the general language processing model using the psychological counseling dialogue data set to obtain the dialogue model.

[0038] In a specific implementation of the second aspect, the processing module includes:

[0039] A sample set acquisition submodule, configured to acquire the training sample set;

[0040] The model training submodule is used to train the initial deep learning model with the multimodal information sample of each training sample in the training sample set as input and the expected detection result of each training sample in the training sample set as expected output to obtain the mental state detection model.

[0041] In a specific implementation of the second aspect, the mental state detection device further includes:

[0042] The material generation module is used to generate multimodal feedback material corresponding to the psychological state detection result and display the multimodal feedback material to the user.

[0043] In a specific implementation of the second aspect, the material generation module includes:

[0044] a music material generation submodule, configured to process the mental state detection result using a preset material generation model to obtain music feedback material corresponding to the mental state detection result; wherein the material generation model is a probabilistic generation model trained using a preset material training sample set, wherein the material training sample set includes a preset number of training samples, each training sample including a mental state detection result sample and corresponding expected music feedback material;

[0045] The synthesis submodule is used to synthesize the dance movements of the digital human according to the music feedback material to obtain the multimodal feedback material.

[0046] In a specific implementation of the second aspect, the multimodal information includes at least two of the following: visual information, voice information, text information, and physiological signal information.

[0047] A third aspect of an embodiment of the present application provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the steps of any of the above-mentioned mental state detection methods are implemented.

[0048] The fourth aspect of an embodiment of the present application provides a terminal device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of any one of the above-mentioned mental state detection methods when executing the computer program.

[0049] A fifth aspect of an embodiment of the present application provides a computer program product, which, when executed on a terminal device, enables the terminal device to execute the steps of any one of the above-mentioned mental state detection methods. Beneficial effects

[0050] Compared with the prior art, the embodiments of the present application have the following advantages: the embodiments of the present application obtain multimodal information of a user during a conversation; use a preset psychological state detection model to process the multimodal information to obtain a psychological state detection result of the user; wherein the psychological state detection model is a deep learning model trained with a preset training sample set, wherein the training sample set includes a preset number of training samples, each training sample including a multimodal information sample and a corresponding expected detection result. Through the present application, it is possible to obtain a user's multimodal information during a conversation with the user, and use the psychological state detection model to process the multimodal information to obtain a psychological state detection result of the user, thereby improving the accuracy of the psychological state detection result. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the embodiments or descriptions of the prior art. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0052] FIG1 is a flow chart of an embodiment of a method for detecting a mental state according to an embodiment of the present application;

[0053] FIG2 is a schematic flow chart of obtaining multimodal information of a user during a conversation;

[0054] FIG3 is a schematic flow chart of generating multimodal feedback materials corresponding to psychological state detection results;

[0055] Figure 4 is a complete flowchart of the mental state detection;

[0056] FIG5 is a structural diagram of an embodiment of a mental state detection device according to an embodiment of the present application;

[0057] FIG6 is a schematic block diagram of a terminal device in an embodiment of the present application. Modes for Carrying Out the Invention

[0058] In order to make the purpose, features, and advantages of the invention of this application more obvious and easy to understand, the technical solutions in the embodiments of this application will be clearly and completely described below in conjunction with the drawings in the embodiments of this application. Obviously, the embodiments described below are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0059] It will be understood that when used in this specification and the appended claims, the term "comprising" indicates the presence of described features, integers, steps, operations, elements and / or components, but does not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof.

[0060] It should also be understood that the terms used in this specification are for the purpose of describing specific embodiments only and are not intended to limit the present application. As used in this specification and the appended claims, the singular forms "a," "an," and "the" are intended to include the plural forms unless the context clearly indicates otherwise.

[0061] It should be further understood that the term "and / or" used in this specification and the appended claims refers to and includes any and all possible combinations of one or more of the associated listed items.

[0062] As used in this specification and the appended claims, the term "if" can be interpreted as "when" or "upon" or "in response to determining" or "in response to detecting," depending on the context. Similarly, the phrase "if it is determined" or "if [described condition or event] is detected" can be interpreted as meaning "upon determination" or "in response to determining" or "upon detection of [described condition or event]" or "in response to detecting [described condition or event]," depending on the context.

[0063] In addition, in the description of the present application, the terms "first", "second", "third", etc. are only used to distinguish the description and cannot be understood as indicating or implying relative importance.

[0064] In the existing technology, questionnaires that are text information are currently the most commonly used method for detecting psychological states. However, due to various circumstances, questionnaires cannot well reflect the psychological state of the subject, resulting in a low accuracy rate of the psychological state detection results obtained through questionnaires. In addition, the psychological state of the subject cannot be accurately expressed by text information alone, because in addition to being expressed through text information, a person's psychological state can also be expressed through visual information, voice information, and physiological signal information.

[0065] In an embodiment of the present application, a mental state detection system is provided. A user can register an account and log in through the system. After logging in to the system, the user can perform mental state detection on the system. During the mental state detection process, the system can generate a multimodal digital human video to communicate with the user, and extract the user's multimodal information during the conversation, and then use a preset mental state detection model to process the multimodal information to obtain the user's mental state detection result. According to the user's mental state detection result, corresponding multimodal feedback materials are generated and displayed to the user. At the same time, the user can also query previous mental state detection results and corresponding multimodal feedback materials on the system.

[0066] The embodiment of the present application also provides a method for detecting psychological state through multimodal information, which can be applied to any terminal device such as mobile phones, tablet computers, desktop computers, notebooks, and handheld computers.

[0067] Referring to FIG1 , an embodiment of a method for detecting a mental state in an embodiment of the present application may include:

[0068] Step S101: Acquire multimodal information of the user during the conversation.

[0069] The multimodal information includes at least two of the following: visual information, voice information, text information, and physiological signal information.

[0070] Before obtaining the multimodal information of the user during the conversation, it can be determined through sensors such as cameras or microphones whether the user needs to perform a mental state test. Specifically, when the camera detects that the user has entered a preset distance and the user issues a conversation request through buttons, voice or gestures, it is considered that the user needs to perform a mental state test and step S101 and subsequent steps are started; if the user is not found to have entered the preset distance, or the user has entered the preset distance but has not issued a conversation request, it is considered that the user does not need to perform a mental state test, and the sensor is continuously used to determine whether the user needs to perform a mental state test.

[0071] In a specific implementation of the embodiment of the present application, step S101 may specifically include the process shown in FIG2 :

[0072] Step S1011: Generate a conversation text with the user using a preset conversation model.

[0073] In a specific implementation of an embodiment of the present application, the training process of the dialogue model is to first obtain a pre-trained general language processing model and a preset psychological counseling dialogue data set, and then use the psychological counseling dialogue data set to train the general language processing model to obtain a trained dialogue model. Specifically, any pre-trained general language processing model in the prior art can be used, including but not limited to a generative pretrained transformer (GPT) model, etc., and the embodiment of the present application does not specifically limit this. However, the general language processing model in the prior art is not a language processing model specifically used in the field of psychological state detection, so the dialogue text generated by it cannot be directly used for psychological state detection. At this time, the psychological counseling dialogue data set can be used to perform knowledge distillation on the general language processing model, so that the general language processing model can generate dialogue text for psychological state detection. The general language processing model that completes knowledge distillation is the dialogue model in the embodiment of the present application.

[0074] Step S1012: convert the conversation text into speech to obtain the conversation speech with the user.

[0075] In a specific implementation of an embodiment of the present application, any speech conversion model in the prior art can be used for speech conversion, such as a Massively Multilingual Speech (MMS) model, etc., and the embodiment of the present application does not specifically limit this.

[0076] Step S1013: Perform talking head synthesis and body posture synthesis based on the conversational speech to obtain a multimodal digital human video.

[0077] In the embodiments of the present application, any model in the prior art can be used for talking head synthesis, including but not limited to the Face with Implicit Attribute Learning (FACIAL) model, the Voice Operated Character Animation (VOCA) model, and the Periodic Implicit Generative Adversarial Networks (Pi-GAN). When performing body posture synthesis, any model in the prior art can be selected, including but not limited to the Phase-Functioned Neural Networks (PFNN), the Local Motion Phase (LMP) method, and the Motion Diffusion Model (MDM).

[0078] After selecting the appropriate talking head synthesis model and body posture synthesis model according to needs, input the conversation speech and conversation text into the talking head synthesis model and body posture synthesis model respectively, and you can get the text-synchronized synthesized talking head and synthesized body posture. Then, fuse the synthesized talking head and synthesized body posture to obtain a multimodal digital human video.

[0079] Step S1014: Conduct a dialogue with the user through the multimodal digital human video, and obtain multimodal information of the user during the dialogue.

[0080] When communicating with a user through a multimodal digital human video, the visual information, voice information or physiological signal information generated by the user during the conversation can be collected through visual sensors, auditory sensors or physiological signal sensors respectively, and the voice information can be converted into text information based on automatic speech recognition technology (Automatic Speech Recognition, ASR).

[0081] In order to improve users' trust in multimodal digital humans, the content of the conversation can be saved in the conversation history data corresponding to the user when having a conversation with the user. When the user mentions the relevant topic again, the relevant content can be found in the conversation history data and the user can be talked to. This is conducive to the smooth progress of the conversation process and the authenticity of the conversation content, and further improves the accuracy of psychological state detection.

[0082] Step S102: Use a preset mental state detection model to process the multimodal information to obtain a mental state detection result of the user.

[0083] In a specific implementation of an embodiment of the present application, the mental state detection model is a deep learning model obtained by training with a preset training sample set. The specific type of deep learning model to be used can be selected according to actual conditions, such as attention mechanism long short-term memory network (Attention-Long Short Term Memory, Attention-LSTM), radial basis function network (Radial Basis Function, RBF), convolutional neural network (Convolutional Neural Networks, CNN), deconvolutional network (Deconvolutional Networks, DN), generative adversarial network (Generative Adversarial Networks, GAN), recurrent neural network (Recurrent Neural Networks, RNN), deep residual network (Deep Residual Networks, DRN) and deep convolutional inverse graphics network (Deep Convolutional Inverse Graphics Networks, DCIGN), etc. The embodiment of the present application does not make specific limitations on this.

[0084] In an embodiment of the present application, the training sample set used to train the mental state detection model includes a preset number of training samples, each training sample includes a multimodal information sample and a corresponding expected detection result. The multimodal information sample of each training sample in the training sample set is used as input, and the expected detection result of each training sample in the training sample set is used as the expected output. The initial deep learning model is trained, and when the preset training conditions are met, the mental state detection model can be obtained.

[0085] During the training process, for each training sample in the training sample set, the multimodal information sample in the training sample can be processed using the deep learning model to obtain the actual output detection result. The training loss value is then calculated based on the expected detection result of the expected output in the training sample and the actual output detection result. The specific calculation method of the training loss value can be set according to the actual situation. For example, the squared error between the expected detection result of the expected output and the actual output detection result can be calculated and determined as the training loss value.

[0086] After calculating the training loss value, the model parameters of the deep learning model can be adjusted according to the training loss value. In the embodiment of the present application, assuming that in the initial state, the model parameter of the deep learning model is W1, the training loss value is back-propagated to modify the model parameter W1 of the deep learning model to obtain the modified model parameter W2. After modifying the parameters, the next training process is continued. During this training process, the training loss value is recalculated and back-propagated to modify the model parameter W2 of the deep learning model, obtaining the modified model parameter W3, and so on. The above process is repeated continuously, and the model parameters can be modified in each training process until the preset training conditions are met. The training condition can be that the number of training times reaches a preset threshold. The threshold can be set according to actual conditions, for example, it can be set to thousands, tens of thousands, hundreds of thousands, or even larger values. The training condition can also be that the deep learning model converges. Because the number of training times may not reach the threshold, but the deep learning model has converged, unnecessary work may be repeated. Or the deep learning model may never converge, resulting in an infinite loop and the training process cannot be terminated. Based on the above two situations, the training condition can also be that the number of training times reaches the threshold or the deep learning model converges. When the training conditions are met, a trained mental state detection model is obtained.

[0087] After obtaining a trained mental state detection model, in addition to using the mental state detection model to process multimodal information and directly obtain the user's mental state detection result, the user's mental state detection result can also be obtained by combining the obtained mental state detection result before the combination (referred to as the first detection result) with the user's historical mental state detection results after processing the multimodal information using the mental state detection model. The user's historical mental state detection results are the mental state detection results of each time the user performs the current mental state detection. The first detection result and the historical mental state detection results are weighted averaged to obtain the user's mental state detection result. The weight of the historical mental state detection result is lower for the detection time farther from the current mental state detection, and higher for the detection time closer to the current mental state detection. The mental state detection results can be divided into level 1 mental state, level 2 mental state, level 3 mental state, level 4 mental state, and level 5 mental state from good to bad. Level 1 mental state indicates the best mental state, and level 5 mental state indicates the worst mental state. After obtaining the user's mental state detection result, step S301 and subsequent steps can be executed. It should be noted that the above content is only an example in the embodiment of the present application. The classification of the psychological state detection results and the method of determining them can be set according to actual conditions, and the embodiment of the present application does not make specific limitations on this.

[0088] If the user's mental state detection result is obtained in advance before the conversation session ends, the accuracy of the mental state detection may be affected. Therefore, to ensure that the conversation session has ended, the end of the conversation session with the user can be determined based on a conversation end statement set or the user's mental state changes over a period of time. The conversation end statement set stores several statements representing the end of the conversation (referred to as end statements). Specifically, when determining whether the conversation session has ended based on the conversation end statement set, the user's current conversation statement can be compared with the end statements in the conversation end statement set. If the user's current conversation statement has the same or similar meaning as a certain end statement, the conversation session is considered to have ended, and the user's mental state at this time is used as the mental state detection result. Step S301 and its subsequent steps are executed simultaneously. If the user's current conversation statement has the same or similar meaning as each end statement, the conversation session continues until the user's current conversation statement has the same or similar meaning as the end statement.

[0089] When determining whether a conversation session has ended based on the user's mental state changes over a period of time, the user's mental state can be determined by determining whether the conversation session has ended. If the user's mental state is at the same level three times in a row, the user's mental state is considered stable. If the user's mental state is stable, the conversation session is considered to have ended. If the user's mental state is unstable, the conversation session is considered to have not ended. For example, if the user's mental state is, in order, level 2, level 3, level 3, level 2, level 1, and level 1, the user's mental state is unstable, and the conversation session is considered to have not ended, and the conversation with the user continues. If the user's mental state is, in order, level 3, level 1, level 1, level 2, level 2, and level 2, the user's mental state detection result is considered to be level 2, the conversation session is considered to have ended, and step S301 and subsequent steps are executed.

[0090] After obtaining the user's mental state detection result, the embodiment of the present application may further generate multimodal feedback materials corresponding to the mental state detection result and display the multimodal feedback materials to the user. Generating the multimodal feedback materials corresponding to the mental state detection result may specifically include the process shown in FIG3 :

[0091] Step S301: Use a preset material generation model to process the psychological state detection result to obtain music feedback material corresponding to the psychological state detection result.

[0092] In a specific implementation of the embodiment of the present application, the material generation model is a probability generation model obtained by training with a preset material training sample set.

[0093] The material training sample set includes a preset number of training samples, each of which includes a psychological state detection result sample and corresponding expected music feedback material. An initial probability generation model is trained using the psychological state detection result sample of each training sample in the material training sample set as input and the expected music feedback material of each training sample in the material training sample set as the expected output. When preset training conditions are met, a material generation model is obtained. The specific training process can be referred to existing techniques and will not be described in detail here.

[0094] After obtaining a trained material generation model, the material generation model can be used to generate music feedback materials corresponding to the user's psychological state detection results. Compared to traditional rule-based and database retrieval methods, the data-driven probability generation model based on psychological state detection results in the embodiments of the present application can better fit continuous complex probability functions, thereby better handling various complex and changing situations in real-world scenarios.

[0095] Step S302: synthesize digital human dance movements based on the music feedback material to obtain multimodal feedback material.

[0096] In a specific implementation of an embodiment of the present application, a Vector Quantized Variational Auto-Encoder (VQ-VAE) can be used to decompose dance movements into body posture codes (Pose Codes), and the body posture codes under a standard dance posture are used as a posture code sequence. In order to make the body posture codes cover as many dance movements as possible, the dance movements can be first divided into upper and lower body parts, and then VQ-VAE is used to decompose the dance movements into body posture codes. The body posture codes of the upper and lower body are then combined into a posture code sequence. At the same time, the posture code sequence is evaluated by GPT to remove posture code sequences that do not meet the requirements of standard dance postures. Use the posture coding sequence and the corresponding music to train a neural network such as a long short-term memory network (LSTM) or a gated recurrent unit network (GRU) to obtain a choreography model, input the music feedback material into the choreography model to obtain a generated posture coding sequence, and then map the posture coding sequence to a two-dimensional image space through three-dimensional graphics, a generative adversarial network (GAN) or a diffusion model (DM), etc., to obtain the generated multimodal feedback material. It should be noted that the above content is only an example in the embodiment of the present application. The specific method used for digital human dance movement synthesis can be selected according to actual conditions, and the embodiment of the present application does not make specific limitations on this.

[0097] FIG4 shows a complete flowchart of the mental state detection process in an embodiment of the present application. First, a user's dialogue request is obtained through a sensor. If the user's dialogue request is not obtained through the sensor, this step is repeated until a user's dialogue request is obtained. If a user's dialogue request is obtained, a dialogue model is first used to generate a dialogue text with the user. This dialogue text is then converted into a dialogue speech with the user, and a multimodal digital human video is generated based on the dialogue speech. Finally, a dialogue is conducted with the user through the multimodal digital human video, and multimodal information of the user during the dialogue is obtained. The multimodal information is then processed using the mental state detection model to obtain a mental state detection result. After corresponding multimodal feedback material is generated based on the mental state detection result and displayed to the user, the mental state detection process is completed. The details of each step can be referred to the relevant description of the aforementioned embodiment and will not be repeated here.

[0098] In summary, the embodiments of the present application obtain multimodal information of a user during a conversation; use a preset psychological state detection model to process the multimodal information to obtain a psychological state detection result of the user; wherein the psychological state detection model is a deep learning model trained with a preset training sample set, wherein the training sample set includes a preset number of training samples, each training sample including a multimodal information sample and a corresponding expected detection result. Through the present application, it is possible to obtain a user's multimodal information during a conversation with the user, and use the psychological state detection model to process the multimodal information to obtain a psychological state detection result of the user, thereby improving the accuracy of the psychological state detection result.

[0099] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0100] Corresponding to the mental state detection method described in the above embodiment, FIG5 shows a structural diagram of an embodiment of a mental state detection device provided in an embodiment of the present application.

[0101] In this embodiment, a mental state detection device may include:

[0102] Information acquisition module 501, used to obtain multimodal information of the user during the conversation;

[0103] An information processing module 502 is configured to process the multimodal information using a preset mental state detection model to obtain a mental state detection result of the user;

[0104] The mental state detection model is a deep learning model obtained by training with a preset training sample set, wherein the training sample set includes a preset number of training samples, and each training sample includes a multimodal information sample and a corresponding expected detection result.

[0105] In a specific implementation of the embodiment of the present application, the acquisition module includes:

[0106] A generation submodule, configured to generate a conversation text with the user using a preset conversation model;

[0107] A conversion submodule, configured to convert the conversation text into speech to obtain a speech of the conversation with the user;

[0108] a synthesis submodule, configured to perform talking head synthesis and body posture synthesis based on the conversational speech to obtain a multimodal digital human video;

[0109] The dialogue submodule is used to conduct a dialogue with the user through the multimodal digital human video and obtain multimodal information of the user during the dialogue.

[0110] In a specific implementation of the embodiment of the present application, the mental state detection device further includes:

[0111] The acquisition module is used to obtain the pre-trained general language processing model and the preset psychological counseling dialogue dataset;

[0112] A training module is used to train the general language processing model using the psychological counseling dialogue data set to obtain the dialogue model.

[0113] In a specific implementation of the embodiment of the present application, the processing module includes:

[0114] A sample set acquisition submodule, configured to acquire the training sample set;

[0115] The model training submodule is used to train the initial deep learning model with the multimodal information sample of each training sample in the training sample set as input and the expected detection result of each training sample in the training sample set as expected output to obtain the mental state detection model.

[0116] In a specific implementation of the embodiment of the present application, the mental state detection device further includes:

[0117] The material generation module is used to generate multimodal feedback material corresponding to the psychological state detection result and display the multimodal feedback material to the user.

[0118] In a specific implementation of the embodiment of the present application, the material generation module includes:

[0119] a music material generation submodule, configured to process the mental state detection result using a preset material generation model to obtain music feedback material corresponding to the mental state detection result; wherein the material generation model is a probabilistic generation model trained using a preset material training sample set, wherein the material training sample set includes a preset number of training samples, each training sample including a mental state detection result sample and corresponding expected music feedback material;

[0120] The synthesis submodule is used to synthesize the dance movements of the digital human according to the music feedback material to obtain the multimodal feedback material.

[0121] In a specific implementation of the embodiment of the present application, the multimodal information includes at least two of the following: visual information, voice information, text information, and physiological signal information.

[0122] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described devices, modules and units can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0123] In the above embodiments, the description of each embodiment has its own focus. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant description of other embodiments.

[0124] Figure 6 shows a schematic block diagram of a terminal device provided in an embodiment of the present application. For ease of explanation, only the parts related to the embodiment of the present application are shown.

[0125] As shown in FIG6 , the terminal device 6 of this embodiment includes: a processor 60, a memory 61, and a computer program 62 stored in the memory 61 and executable on the processor 60. When the processor 60 executes the computer program 62, it implements the steps in the aforementioned mental state detection method embodiments, such as steps S101 to S102 shown in FIG1 . Alternatively, when the processor 60 executes the computer program 62, it implements the functions of the modules / units in the aforementioned apparatus embodiments, such as the functions of modules 501 to 502 shown in FIG5 .

[0126] Exemplarily, the computer program 62 may be divided into one or more modules / units, which are stored in the memory 61 and executed by the processor 60 to implement the present application. The one or more modules / units may be a series of computer program instruction segments capable of performing specific functions, and the instruction segments are used to describe the execution process of the computer program 62 in the terminal device 6.

[0127] The terminal device 6 may be a computing device such as a mobile phone, tablet computer, desktop computer, notebook computer, PDA, robot, etc. Those skilled in the art will appreciate that FIG6 is merely an example of the terminal device 6 and does not limit the terminal device 6 . The terminal device 6 may include more or fewer components than shown in the figure, or may combine certain components or different components. For example, the terminal device 6 may also include input and output devices, network access devices, buses, etc.

[0128] The processor 60 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor.

[0129] The memory 61 can be an internal storage unit of the terminal device 6, such as a hard disk or memory of the terminal device 6. The memory 61 can also be an external storage device of the terminal device 6, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the terminal device 6. Furthermore, the memory 61 can also include both an internal storage unit of the terminal device 6 and an external storage device. The memory 61 is used to store the computer program and other programs and data required by the terminal device 6. The memory 61 can also be used to temporarily store data that has been output or is about to be output.

[0130] Those skilled in the art can clearly understand that, for the convenience and brevity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated into one processing unit, or each unit can exist physically alone, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0131] In the above embodiments, the description of each embodiment has its own focus. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant description of other embodiments.

[0132] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0133] In the embodiments provided in this application, it should be understood that the disclosed devices / terminal equipment and methods can be implemented in other ways. For example, the device / terminal equipment embodiments described above are merely illustrative. For example, the division of the modules or units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0134] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0135] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0136] If the integrated module / unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the process in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and when the computer program is executed by the processor, it can implement the steps of the above-mentioned various method embodiments. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable storage medium may include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium. It should be noted that the content contained in the computer-readable storage medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable storage media do not include electric carrier signals and telecommunication signals.

[0137] The above-described embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present application, and should all be included in the scope of protection of the present application.

Claims

1. A method for detecting a mental state, characterized in that: include: Obtain multimodal information from users during the conversation; Using a preset psychological state detection model to process the multimodal information to obtain a psychological state detection result of the user; Among them, the mental state detection model is a deep learning model trained by a preset training sample set, and the training sample set includes a preset number of training samples, each training sample includes a multimodal information sample and a corresponding expected detection result.

2. The method for detecting a psychological state according to claim 1, characterized in that: The obtaining of multimodal information of the user during the conversation process includes: Generate a dialogue text with the user using a preset dialogue model; Performing voice conversion on the conversation text to obtain a conversation voice with the user; Performing talking head synthesis and body posture synthesis according to the dialogue speech to obtain a multimodal digital human video; A conversation is conducted with the user through the multimodal digital human video, and multimodal information of the user during the conversation is obtained.

3. The method for detecting a psychological state according to claim 2, characterized in that: The training process of the dialogue model includes: Obtain a pre-trained general language processing model and a preset psychological counseling dialogue dataset; The general language processing model is trained using the psychological counseling dialogue data set to obtain the dialogue model.

4. The method for detecting a psychological state according to claim 1, characterized in that: The training process of the mental state detection model includes: Obtaining the training sample set; The multimodal information sample of each training sample in the training sample set is used as input, and the expected detection result of each training sample in the training sample set is used as the expected output, and the initial deep learning model is trained to obtain the mental state detection model.

5. The method for detecting a psychological state according to claim 1, characterized in that: After the multimodal information is processed using a preset psychological state detection model to obtain the psychological state detection result of the user, the method further includes: Generate multimodal feedback material corresponding to the psychological state detection result, and display the multimodal feedback material to the user.

6. The method for detecting mental state according to claim 5, characterized in that: The generating of multimodal feedback material corresponding to the psychological state detection result includes: The mental state detection result is processed using a preset material generation model to obtain music feedback material corresponding to the mental state detection result; wherein the material generation model is a probability generation model obtained by training with a preset material training sample set, the material training sample set includes a preset number of training samples, each training sample includes a mental state detection result sample and a corresponding expected music feedback material; The digital human dance movements are synthesized according to the music feedback material to obtain the multimodal feedback material.

7. The method for detecting a mental state according to any one of claims 1 to 6, characterized in that: The multimodal information includes at least two of the following: visual information, voice information, text information and physiological signal information.

8. A psychological state detection device, characterized in that: include: The information acquisition module is used to obtain the multimodal information of the user during the conversation; An information processing module, used to process the multimodal information using a preset psychological state detection model to obtain a psychological state detection result of the user; Among them, the mental state detection model is a deep learning model trained by a preset training sample set, and the training sample set includes a preset number of training samples, each training sample includes a multimodal information sample and a corresponding expected detection result.

9. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the mental state detection method according to any one of claims 1 to 7 are implemented.

10. A terminal device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the mental state detection method according to any one of claims 1 to 7 are implemented.

Images