Psychological consultation robot based on multimodal data and vertical large model

Abstract

The present application relates to a psychological counseling robot based on multimodal data and vertical big models. The robot is used to provide psychological counseling services in scenarios such as families, schools, and enterprises. The front end relies on intelligent robot terminals, with virtual digital people as the main way of human-computer interaction, supplemented by multimodal emotion recognition to enhance the effect of psychological counseling; the back end uses intelligent agents as the control core and psychological big models as the computing platform. Utilizing the powerful semantic understanding and dialogue interaction capabilities of the big model, combined with multimodal pattern recognition of expressions and voices, psychological assessment and psychological healing functions are realized, while having the ability to supplement and verify knowledge in external knowledge bases; using structured prompts combined with multimodal emotion data enhancement strategies, with the help of retrieval enhancement generation and knowledge graph technology, the performance advantages of the big model are fully utilized while reducing the occurrence of hallucination problems, and digital human interaction methods are used to reduce interpersonal barriers and improve the quality of consultation.

Description

Technical Field

[0001] The present application relates to the technical field of psychological counseling robots, and in particular to a psychological counseling robot based on multimodal data and vertical large models. Background Art

[0002] In the real world, it is difficult for robots to complete the entire complex action at one time, so complex actions need to be broken down into multiple simple steps. In addition, since the real world is far more complex than the robot training scenario, the robot is not the only object, so it will be subject to various interferences during the task process, and the environment will continue to change, so the robot needs to have strong environmental adaptability. The emergence of large models is expected to completely subvert the software development paradigm of robots. Pre-training gives large models the ability to compress knowledge. The pre-trained large models have high generalization capabilities and can cope with low-probability task scenarios that have never been seen, thereby greatly reducing the complexity of algorithm development. When the model parameters grow to a certain level, the large model will emerge with the ability of thinking chain. After the emergence of large models, the human-computer interaction ability of robots has been greatly improved. The large model system runs in the cloud, and various multimodal interaction content is generated by cloud computing. At present, large model technology has developed from a simple large language model (LLM) to a visual language model (VLM). The data modality of large models is gradually enriched, the data scale is rapidly increasing, and the application scenarios and value of large models are also proportionally expanded.

[0003] The document with publication number CN114998440A performs a psychological scale assessment on the user, obtains his facial image for emotion recognition and orientation analysis, and executes the corresponding psychological assessment process in combination with the user's answer time, emotional information and facial orientation, thereby improving the accuracy of the online scale assessment. This solution only relies on the scale assessment method, which has a relatively simple means and is difficult to ensure that users answer honestly. The present invention is based on the latest large-model intelligent agent framework, utilizing its powerful semantic understanding and dialogue interaction capabilities, combined with multimodal pattern recognition for expression and voice, to provide psychological assessment and healing services, and has good application prospects and commercial value.

[0004] Existing automated psychological counseling systems are often based on psychological questionnaires and traditional question-answering techniques. They have monotonous interfaces, standardized interactive processes, and rigid content. They lack emotional resonance and are prone to user fatigue. The latest large model technology has shown great potential in automated mental health analysis, but there are still some shortcomings, including insufficient evaluation, lack of prompting strategies, and neglect of interpretability exploration of large models. Summary of the invention

[0005] Based on this, it is necessary to provide a psychological counseling robot based on multimodal data and vertical large models to address the above technical problems.

[0006] A psychological counseling robot based on multimodal data and vertical big models, the psychological counseling robot comprising: an intelligent robot terminal and an intelligent robot backend, wherein the intelligent robot terminal comprises a virtual digital human, a multimodal emotion recognition module, and a data fusion module; the intelligent robot backend comprises: an intelligent agent module, an external knowledge base, and a psychological big model.

[0007] Virtual digital human is used to realize multimodal intelligent interaction between humans and machines.

[0008] The multimodal emotion recognition module is used to extract features and classify and recognize the images and voice data of the user speaking collected by the voice collection device and image or video collection device carried by the psychological counseling robot itself, so as to obtain facial expression recognition results and voice emotion recognition results.

[0009] The data fusion module is used to fuse the facial expression recognition results and the speech emotion recognition results with the user dialogue text to obtain the emotion and dialogue text fusion result.

[0010] The intelligent agent module is the control core of the psychological counseling robot. It is used to realize psychological assessment, psychological therapy, health management, and result verification functions. It is also used to realize RAG functions and interactive control with the psychological big model.

[0011] External knowledge base, used to supplement and verify psychological professional knowledge.

[0012] The psychological big model is a big model that is fine-tuned based on the open source LLM architecture using the psychological counseling fine-tuning dataset. It is used to implement the computing platform of the psychological counseling robot.

[0013] In one embodiment, the multimodal emotion recognition module includes: a real-time data acquisition submodule, a dynamic expression recognition submodule, and a speech recognition submodule.

[0014] The real-time data acquisition submodule is used to synchronously capture facial images and voice signal sequences of a preset time period during the user's speech in each interview dialogue.

[0015] The dynamic expression recognition submodule is used to use a convolutional neural network encoder to perform deep feature extraction on each facial image to obtain a facial feature sequence, and then use a standard Transformer encoder to extract spatiotemporal features on the facial feature sequence to obtain a word unit encoding sequence, and finally determine the facial expression recognition result based on the word unit encoding sequence.

[0016] The speech recognition submodule is used to preprocess the speech signal sequence, and then use the lightweight audio classification model architecture to extract the Mel-frequency cepstral coefficient features of the speech signal. Finally, the LSTM neural network is used to classify the Mel-frequency cepstral coefficient features to obtain the speech emotion recognition results.

[0017] In one embodiment, determining a facial expression recognition result according to a word unit encoding sequence includes:

[0018] A phrase classifier is used to perform calculations based on the word-meta encoding sequence to obtain a facial expression recognition result; wherein the phrase classifier includes a word-meta pooling layer and a fully connected layer; the word-meta pooling layer is used to cross-divide the word-meta encoding sequence into two groups, for each word-meta in the first group, find a most similar word-meta from the second group as a merging target and record its word-meta similarity, and average-merge a preset number of words with the highest matching word-meta similarity ranking in the first group with the corresponding words in the second group. After the merger is completed, the total number of words will be reduced by a preset number, and the remaining words will be concatenated in sequence to obtain a word-meta pooling result; the fully connected layer is used to perform classification based on the word-meta pooling result to obtain a facial expression recognition result.

[0019] In one embodiment, the word-unit similarity is:

[0020]

[0021] in, For vector A , B The word similarity of A , B For two non-zero dimensional vector, Represents vectors A and B The elements.

[0022] In one embodiment, the intelligent agent module includes: a psychological assessment submodule, a psychological healing submodule, a health management submodule, a result checking submodule, a RAG function submodule and an interactive control submodule.

[0023] The psychological assessment submodule is used to assess the user's psychological state by combining preliminary screening through psychological interviews and professional evaluations with psychological scales.

[0024] The psychological healing sub-module is used to define the content of the dialogue response part based on the psychological big model and use the prompt engineering method to achieve dialogue generation.

[0025] The health management submodule is used to automatically generate user health reports based on user physiological and psychological measurement data to achieve comprehensive health management.

[0026] The result verification submodule is used to provide the verification function of the report results.

[0027] The RAG functional submodule is used to retrieve relevant information from a preset document collection or database when the psychological model needs to generate text or answer questions, and then use the retrieved information to guide text generation.

[0028] The interactive control submodule is used to realize the access request to the psychological big model according to the interactive requirements of the intelligent robot terminal.

[0029] In one embodiment, the working process of the psychological assessment submodule includes:

[0030] Based on the big psychological model, prompt engineering methods are used to define prompt information to guide the interview dialogue; prompt information includes dialogue roles, task descriptions, round limits, questioning methods, and empathy support.

[0031] The facial expression recognition results and speech emotion recognition results are inserted into the user conversation text as prefixes according to the preset format for text feature fusion to obtain a complete interview text sequence.

[0032] Taking the interview text sequence as input and guided by the prompt information of different symptoms, the psychological big model gives the user's psychological state assessment results based on the interview content.

[0033] Based on the user's psychological state assessment results, a specific scale is recommended for evaluation based on preset custom rules to obtain the scale evaluation results.

[0034] Create a comprehensive user portrait based on the user's psychological state assessment results and scale evaluation results; the user portrait includes personal information, interview analysis results and scale evaluation results.

[0035] In one embodiment, the preset format is:

[0036] [{“role”: “user”, “content”: “<emotion description>X…X”}, {“role”: “assistant”, “content”: “X…X”}, …];

[0037] Among them, role represents the dialogue role, user corresponds to the user, and assistant corresponds to the big model; content represents the dialogue content of the corresponding role, the prefix <emotion description> contains the user's facial and voice emotion data, and X represents the corresponding dialogue text.

[0038] In one embodiment, the working process of the psychological healing submodule includes: following the core principles of cognitive behavioral therapy, the dialogue response of the psychological healing submodule involves five components; among which, the first component is confirmation and empathy, the second component is thinking or belief, the third component is challenge or reflection, the fourth component is strategy or insight, and the fifth component is encouragement and prospect.

[0039] Based on the big psychological model, the prompt engineering method is used to specify the contents of the five components to achieve dialogue generation.

[0040] In one of the embodiments, the external knowledge base is generated based on graph retrieval enhancement.

[0041] In one embodiment, the process of constructing a mental model includes:

[0042] Select Llama 3 8B as the base LLM.

[0043] Collect Chinese psychological counseling open source data and psychology professional materials, and construct a psychological counseling fine-tuning dataset through manual selection.

[0044] The psychological counseling fine-tuning dataset is used to fully fine-tune the base LLM based on XTuner to obtain a large psychological model suitable for psychological counseling tasks.

[0045] The above-mentioned psychological counseling robot based on multimodal data and vertical large models is used to provide psychological counseling services in scenarios such as families, schools, and enterprises. The front end relies on intelligent robot terminals, with virtual digital people as the main way of human-computer interaction, supplemented by multimodal emotion recognition to enhance the effect of psychological counseling; the back end uses intelligent agents as the control core and psychological large models as the computing platform, using their powerful semantic understanding and dialogue interaction capabilities, combined with multimodal pattern recognition for expressions and voices, to achieve psychological assessment and psychological healing functions. At the same time, it has the ability to supplement and check knowledge in external knowledge bases, and can provide psychological assessment and healing services. The psychological counseling robot uses structured prompts combined with multimodal emotion data enhancement strategies, with the help of retrieval enhancement generation and knowledge graph technology, fully utilizing the performance advantages of large models while reducing the occurrence of hallucination problems, and adopts digital human interaction methods to reduce interpersonal barriers and improve the quality of consultation. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 The present invention is a block diagram of a psychological counseling robot based on multimodal data and vertical large model in one embodiment;

[0047] Figure 2 A schematic diagram of an interactive interface of a psychological digital human in another embodiment;

[0048] Figure 3 is a schematic diagram of a workflow of multimodal emotion recognition in another embodiment;

[0049] Figure 4 A schematic diagram of the composition of a dynamic expression recognition model in another embodiment;

[0050] Figure 5 A schematic diagram of the composition of a speech emotion recognition model in another embodiment;

[0051] Figure 6 A schematic diagram of an intelligent agent framework in another embodiment. DETAILED DESCRIPTION

[0052] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0053] Cognitive Behavioral Therapy, Cognitive Behavioral Therapy, abbreviated as CBT.

[0054] Convolutional Neural Network, referred to as CNN.

[0055] Large Language Model, Large Language Model, abbreviated as LLM.

[0056] Long Short-Term Memory, Long Short-Term Memory, abbreviated as LSTM.

[0057] Mel Frequency Cepstrum Coefficient, Mel Frequency Cepstrum Coefficient, referred to as: MFCC.

[0058] Retrieval-Augmented Generation, abbreviated as RAG.

[0059] A neural network architecture with self-attention mechanism as its main feature, referred to as Transformer.

[0060] Voice Activity Detection, Voice Activity Detection, referred to as VAD.

[0061] Vision-Language Model, Vision-Language Model, abbreviated as VLM.

[0062] In one embodiment, Figure 1As shown, a psychological counseling robot based on multimodal data and vertical big model is provided, and the psychological counseling robot includes: an intelligent robot terminal 10 and an intelligent robot backend 20, wherein the intelligent robot terminal 10 includes a virtual digital human 101, a multimodal emotion recognition module 102, and a data fusion module 103; the intelligent robot backend 20 includes: an intelligent agent module 201, an external knowledge base 202 and a psychological big model 203.

[0063] Specifically, the intelligent robot terminal is a front-end device that directly interacts with psychological counseling users. It has functional modules such as speech recognition and synthesis, virtual digital human, multimodal emotion recognition and data fusion, and interacts with the back-end intelligent agent and psychological big model. The intelligent robot terminal is the front-end, and the intelligent agent module + external knowledge base + psychological big model are the back-end services.

[0064] Virtual digital human 101 is used to realize multimodal intelligent interaction between humans and machines.

[0065] Specifically, a virtual digital human is designed and developed based on Unity software, which is packaged as an Android software development kit and then deployed to the psychological counseling robot terminal to achieve multimodal intelligent interaction between man and machine. The interactive interface of the psychological digital human is as follows: Figure 2 shown.

[0066] The multimodal emotion recognition module 102 is used to extract features and classify and recognize the image and voice data of the user speaking collected by the voice collection device and image or video collection device carried by the psychological counseling robot itself, so as to obtain facial expression recognition results and voice emotion recognition results.

[0067] Specifically, emotion recognition can determine the internal emotions and intentions conveyed by an individual, which helps to understand the user's psychological state. Doctors will pay attention to the user's expressions, movements, and tone of voice during psychological counseling. During psychological interviews, the robot's own camera and microphone can be used to collect images and voice data when the user is speaking, perform feature extraction and classification recognition, and obtain facial and voice emotion recognition results. The workflow of multimodal emotion recognition is as follows: Figure 3 shown.

[0068] The data fusion module 103 is used to fuse the facial expression recognition result and the voice emotion recognition result with the user dialogue text to obtain the emotion and dialogue text fusion result.

[0069] The intelligent agent module 201 is the control core of the psychological counseling robot, which is used to realize the functions of psychological assessment, psychological therapy, health management, and result verification. It is also used to realize the RAG function and interactive control with the psychological big model.

[0070] Specifically, the intelligent agent module 201 is the core part of the entire psychological counseling robot system. The intelligent agent module 201 is responsible for the interactive control of the psychological health-related functions and the psychological big model 203. The former mainly includes psychological assessment, psychological therapy and health management, and the latter mainly includes the psychological big model 203 access and RAG functions.

[0071] The external knowledge base 202 is used to supplement and verify psychological professional knowledge.

[0072] The psychological big model 203 is a big model that is fine-tuned based on the open source LLM architecture using the psychological counseling fine-tuning dataset, and is used to implement the computing platform of the psychological counseling robot.

[0073] Specifically, the big psychological model aims to fully understand and promote the mental health of individuals, groups and even the entire society, and usually includes the following components:

[0074] a) Cognitive factors: involve an individual’s thinking patterns, belief systems, cognitive biases, and problem-solving abilities.

[0075] b) Emotional factors: including emotion regulation, emotion expression and emotion experience.

[0076] c) Behavioral factors: involve individual behavior patterns, habits and coping strategies.

[0077] d) Social environment: includes external factors such as family, work, community and cultural background, which have direct and indirect impacts on individual mental health.

[0078] e) Physical health: Physical health can promote mental health and vice versa.

[0079] f) Psychological resilience: refers to an individual’s ability to recover and adapt when faced with adversity.

[0080] g) Assessment methods: Use scientific tools to assess the individual's psychological state and identify possible psychological problems.

[0081] h) Preventive intervention: includes strategies to prevent psychological problems and promote mental health, such as psychoeducation, psychological counseling, psychotherapy and social support systems.

[0082] Deploy models and services to high-performance cloud servers and access them remotely through the Robot App.

[0083] This psychological counseling robot is targeted at psychological counseling service application scenarios. Based on the latest large model and multimodal pattern recognition technology, it strives to provide users with psychological counseling services with high availability and high cost-effectiveness, effectively alleviating the current shortage of professional psychological counselors.

[0084] The working process of the psychological counseling robot based on multimodal data and vertical big models is as follows: the psychological counseling user faces the intelligent robot terminal, conducts a psychological interview (simultaneously detects the user's emotions to improve the interview diagnosis effect) to obtain a preliminary diagnosis result, and then selects a suitable psychological scale test to obtain further diagnosis results, and builds a user portrait based on the above results. Then, based on the user portrait, psychological healing services are provided to the user. In these processes, the content interaction of psychological interviews and psychological healing is all realized based on intelligent agents and psychological big models, and the psychologist is visualized in the image of a digital person on the robot side.

[0085] In the above-mentioned psychological counseling robot based on multimodal data and vertical big models, the psychological counseling robot is used to provide psychological counseling services in scenarios such as families, schools, and enterprises. The front end relies on intelligent robot terminals, with virtual digital people as the main way of human-computer interaction, supplemented by multimodal emotion recognition to enhance the effect of psychological counseling; the back end uses intelligent agents as the control core and psychological big models as the computing platform, using their powerful semantic understanding and dialogue interaction capabilities, combined with multimodal pattern recognition for expressions and voices, to achieve psychological assessment and psychological healing functions, while having the ability to supplement and check knowledge in external knowledge bases, and can provide psychological assessment and healing services. The psychological counseling robot uses structured prompts combined with a strategy of multimodal emotional data enhancement, with the help of retrieval enhancement generation and knowledge graph technology, fully utilizing the performance advantages of the big model while reducing the occurrence of hallucination problems, and adopts digital human interaction methods to reduce interpersonal barriers and improve the quality of consultation.

[0086] In one embodiment, the multimodal emotion recognition module includes: a real-time data acquisition submodule, a dynamic expression recognition submodule, and a speech recognition submodule.

[0087] The real-time data acquisition submodule is used to synchronously capture facial images and voice signal sequences of a preset time period during the user's speech in each interview dialogue.

[0088] Specifically, during each round of interview dialogue, the VAD algorithm in the WebRTC project is used to determine whether the user is speaking. During the user's speech, a 2-second facial image and voice signal sequence is synchronously captured as input data for the multimodal emotion recognition module.

[0089] The dynamic expression recognition submodule is used to use a convolutional neural network encoder to perform deep feature extraction on each facial image to obtain a facial feature sequence, and then use a standard Transformer encoder to extract spatiotemporal features on the facial feature sequence to obtain a word unit encoding sequence, and finally determine the facial expression recognition result based on the word unit encoding sequence.

[0090] Specifically, the dynamic expression recognition submodule realizes facial expression recognition through a dynamic expression recognition model. The input of the dynamic expression recognition model is a dynamically changing facial image sequence collected, recognized and intercepted at fixed time intervals. Figure 4 As shown in FIG. 1 , the dynamic expression recognition model includes: a convolutional neural network encoder (i.e., a CNN encoder), a standard Transformer encoder, and a phrase classifier. The phrase classifier is used to determine the facial expression recognition result according to the word unit encoding sequence.

[0091] The CNN encoder can be but is not limited to ResNet or MobileNet.

[0092] The speech recognition submodule is used to preprocess the speech signal sequence, and then use the lightweight audio classification model architecture to extract the Mel-frequency cepstral coefficient features of the speech signal. Finally, the LSTM neural network is used to classify the Mel-frequency cepstral coefficient features to obtain the speech emotion recognition results.

[0093] Specifically, the speech recognition submodule adopts Figure 5 The speech emotion recognition model described above realizes speech emotion recognition. Considering the limitation of front-end computing power, a lightweight audio classification model architecture is adopted to extract the Mel-frequency cepstral coefficient (MFCC) features of speech signals, and classify them based on the classic long short-term memory (LSTM) neural network to obtain speech emotion recognition results. MFCC features are cepstral parameters extracted in the Mel-scale frequency domain. The Mel-scale describes the nonlinear characteristics of human ear frequency, which is related to the frequency The relationship can be approximated by the following formula:

[0094] (1)

[0095] in, Indicates frequency, represents the Mel scale.

[0096] The LSTM neural network uses frame-level speech features combined with a long short-term memory recursive neural network model based on an attention mechanism for speech classification and recognition, in order to fully utilize the emotional saturation differences in temporal features. Compared with traditional statistical features, frame-level speech features can maintain the temporal relationship in the original speech and improve recognition accuracy.

[0097] In one embodiment, the facial expression recognition result is determined based on the word-meta coding sequence, including: using a phrase classifier to perform calculations based on the word-meta coding sequence to obtain the facial expression recognition result; wherein the phrase classifier includes a word-meta pooling layer and a fully connected layer; the word-meta pooling layer is used to cross-divide the word-meta coding sequence into two groups, for each word-meta in the first group, find a most similar word-meta from the second group as a merging target and record its word-meta similarity, and average-merge a preset number of words with the highest matching word-meta similarity ranking in the first group with the corresponding words in the second group, after the merging is completed, the total number of words will be reduced by a preset number, and the remaining words will be concatenated in sequence to obtain a word-meta pooling result; the fully connected layer is used to classify according to the word-meta pooling result to obtain the facial expression recognition result.

[0098] Specifically, in a phrase classifier, the word pooling layer is an important component used to reduce word-unit information redundancy and speed up classification operations.

[0099] The preset number of terms with the highest similarity ranking among the matching terms in the first group refers to selecting the top preset number of terms by ranking the matching terms in the first group in descending order of similarity. The preset number is a fixed value.

[0100] In one embodiment, the word-unit similarity is:

[0101] (2)

[0102] in, For vector A , B The word similarity of A , B For two non-zero dimensional vector, Represents vectors A and B The elements.

[0103] In one embodiment, if Figure 6 As shown, the intelligent agent module includes: a psychological assessment submodule, a psychological healing submodule, a health management submodule, a result checking submodule, a RAG function submodule and an interactive control submodule.

[0104] The psychological assessment submodule is used to assess the user's psychological state by combining preliminary screening through psychological interviews and professional evaluations with psychological scales.

[0105] The psychological healing sub-module is used to define the content of the dialogue response part based on the psychological big model and use the prompt engineering method to achieve dialogue generation.

[0106] The health management submodule is used to automatically generate user health reports based on user physiological and psychological measurement data to achieve comprehensive health management.

[0107] The result verification submodule is used to provide the verification function of the report results.

[0108] The RAG functional submodule is used to retrieve relevant information from a preset document collection or database when the psychological model needs to generate text or answer questions, and then use the retrieved information to guide text generation.

[0109] Specifically, Retrieval-Augmented Generation (RAG) combines language modeling and information retrieval techniques.

[0110] The interactive control submodule is used to realize the access request to the psychological big model according to the interactive requirements of the intelligent robot terminal. The interactive requirements can be but not limited to interview assessment, counseling and treatment, etc.

[0111] In one embodiment, the working process of the psychological assessment submodule includes: based on the psychological big model, using the prompt engineering method, defining prompt information to complete the interview dialogue guidance; the prompt information includes dialogue roles, task descriptions, round limits, questioning methods and empathy support; the facial expression recognition results and voice emotion recognition results are inserted into the user dialogue text as prefixes in a preset format to perform text feature fusion to obtain a complete interview text sequence; with the interview text sequence as input, according to the guidance of different disease prompt information, the psychological big model gives the user's psychological state assessment results for the interview content; with the user's psychological state assessment results as a reference, a specific scale is recommended for evaluation based on preset custom rules to obtain the scale evaluation results; based on the user's psychological state assessment results and the scale evaluation results, a comprehensive user portrait is created; the user portrait includes personal information, interview analysis results and scale evaluation results.

[0112] In one embodiment, the preset format is:

[0113] [{“role”: “user”, “content”: “<emotion description>X…X”}, {“role”: “assistant”, “content”: “X…X”}, …]; (3)

[0114] Among them, role represents the dialogue role, user corresponds to the user, and assistant corresponds to the large model; content represents the dialogue content of the corresponding role, the prefix <emotion description> contains the user's facial and voice emotion data, such as <happy expression, calm tone>, and X represents the corresponding dialogue text.

[0115] Specifically, the psychological state of the user is assessed by combining preliminary screening through psychological interviews with professional evaluation using psychological scales. The specific steps include:

[0116] (1) Psychological interview

[0117] On the basis of collecting basic information about users, a preliminary assessment of their mental state is conducted through interviews. At present, it is common to use behavioral characteristics in psychological interviews to design automatic detection algorithms for mental illnesses. The feasibility of these methods has been confirmed by psychologists, and their external manifestations can be divided into linguistic characteristics and non-linguistic characteristics. Studies have shown that linguistic characteristics, especially the language habits of patients with mental illness, are often significantly different from those of healthy individuals. Therefore, the linguistic characteristics in psychological interview conversations, including language habits and sentiment analysis, can be used to identify mental illnesses. If combined with non-linguistic characteristics such as user emotions, the recognition accuracy can be further improved.

[0118] 1) Automatic generation of interview dialogues

[0119] Psychological interviews based on psychological counseling robots can adopt a question-and-answer strategy based on goal planning. For each user, a dialogue goal plan is created based on background data, and multiple rounds of dialogues are conducted with the user for each dialogue goal. In each round of dialogue, the psychological counseling robot system, as the active party in the dialogue, raises the topic, analyzes the content of the user's answer, and starts the next round of dialogue based on the user's answer and the dialogue goal. This process is repeated until the pre-set dialogue end condition is met. According to the implementation principle, the dialogue system can be divided into two categories: retrieval-based and generative-based, which are implemented in pipeline and end-to-end modes respectively. The retrieval-based dialogue system adopts a modular design, controllable process, and low computing resource consumption. It has been widely used in task-based dialogue systems; the generative dialogue system is based on deep learning models, especially the breakthrough progress of large model technology, and the dialogue content has been greatly enriched, which is closer to the real human dialogue style.

[0120] The psychological counseling robot proposed in this application is based on a big psychological model and uses prompt engineering methods to define prompt information such as dialogue roles, task descriptions, round limits, questioning methods, and empathy support to complete interview dialogue guidance.

[0121] 2) Emotion Enhancement Interview Assessment

[0122] In the multimodal emotion recognition module, multimodal emotion recognition based on user expressions and voices is implemented, and it is synchronized with the time of each round of user dialogue text. It is inserted into the user dialogue text as a prefix in a fixed format for text feature fusion, which helps to increase the accuracy of mental illness assessment. The format is shown in formula (3), and finally a complete interview text sequence is obtained. With this interview text sequence as input, according to the guidance of different disease prompt information, the psychological big model gives the user's psychological state assessment result based on the interview content.

[0123] (2) Psychological scale

[0124] The method of measuring and evaluating the psychological status of target users through professional questionnaires compiled by psychometricians is an important way to conduct psychological measurement at present. When users evaluate, they first select the scale type according to their own situation, such as "90-item Symptom Checklist (SCL-90)", "Self-Rating Depression Scale (SDS)", "Self-Rating Anxiety Scale (SAS)", "Negative Symptom Rating Scale (SANS)", "Children and Adolescents Mental Health Scale (MHS-CA)", "Chinese Middle School Students' Mental Health Inventory (MMHI-60)", "Undergraduate Mental Health Questionnaire (UPI)", "Post-traumatic Stress Disorder Checklist (PCL)", "Pittsburgh Sleep Quality Index (PSQI)", etc., and then start the evaluation. The evaluation process of different scales is similar. Select the option that best suits your situation in the system questions. When all evaluations are completed, the evaluation scores can be automatically calculated.

[0125] The psychological state of the user is preliminarily determined through psychological interviews, and based on this, a specific scale is recommended for evaluation based on custom rules to obtain the scale evaluation results.

[0126] (3) User Profile

[0127] Combining the results of psychological interviews and scale assessments, a comprehensive user portrait is created, which includes personal information, interview analysis results, scale assessment results, etc., providing sufficient basis for further intervention measures.

[0128] In one embodiment, the working process of the psychological healing submodule includes: following the core principles of cognitive behavioral therapy, the dialogue response of the psychological healing submodule involves five components; among which, the first component is confirmation and empathy, the second component is thinking or belief, the third component is challenge or reflection, the fourth component is strategy or insight, and the fifth component is encouragement and prospect; based on the psychological big model, the prompt engineering method is used to specify the contents of the five components to achieve dialogue generation.

[0129] Specifically, cognitive behavioral therapy (CBT) is one of the most common forms of psychotherapy and has been shown to be effective for a wide range of mental health problems, including anxiety and depression. This approach is centered on cognitive models and describes the interrelationships between situations, thoughts, behaviors, and emotions. It believes that it is not the situation itself that affects people's feelings or reactions, but the interpretation and thinking of the situation. Therefore, it aims to help people recognize, challenge, and reconstruct these negative interpretations and thoughts. CBT places great emphasis on targeting distorted cognition and aims to correct any cognitive bias that causes psychological distress. It combines behavioral experiments and skill training. Following the core principles of CBT, the dialogue response of the psychological healing module involves five components: (1) confirmation and empathy; (2) thinking or beliefs; (3) challenge or reflection; (4) strategy or insight; (5) encouragement and outlook. This application is based on the psychological big model and uses the prompt engineering method to specify the above five parts to achieve dialogue generation.

[0130] In one of the embodiments, the external knowledge base is generated based on graph retrieval enhancement.

[0131] Specifically, the external knowledge base function is implemented based on Graph Retrieval Enhanced Generation (GraphRAG). RAG technology has achieved remarkable success in solving the challenges faced by large models without retraining. By referring to external knowledge bases, RAG effectively alleviates problems such as hallucinations, lack of domain-specific knowledge, and outdated information. However, the complex relational structure between different entities in the database poses a challenge to the RAG system. GraphRAG emerges as an innovative solution to address the limitations of traditional RAG methods. It retrieves graph elements containing relational knowledge from pre-built graph databases, considers the mutual correlation between texts, and can retrieve relational information more accurately and comprehensively. GraphRAG enhances the response of language models by retrieving relevant knowledge from graph databases, and includes three main stages: graph-based indexing (G-Indexing), graph-guided retrieval (G-Retrieval), and graph-enhanced generation (G-Generation). G-Indexing creates a graph database aligned with downstream tasks; G-Retrieval extracts relevant information from the database based on user queries; G-Generation synthesizes output based on the retrieved graph data. The performance of GraphRAG depends heavily on the quality of its graph database, which involves the construction of its own knowledge graph and the optimization of the knowledge retrieval and generation processes.

[0132] In one embodiment, the process of constructing the psychological big model includes the following steps:

[0133] Step S1: Select Llama 3 8B as the base LLM.

[0134] Llama 3 is based on the autoregressive Transformer architecture. Compared with the previous generation Llama 2, Llama 3 has made several key improvements: using a tokenizer with a 128K vocabulary, which can encode language more effectively and significantly improve model performance; using a grouped query attention mechanism, which not only improves big data processing efficiency but also speeds up response speed; using 8K context length, which is twice that of Llama 2. In multiple industry benchmarks, Llama 3 performs significantly better than other models of the same size, showing strong application potential.

[0135] Step S2: Collect Chinese psychological counseling open source data and psychology professional materials, and construct a psychological counseling fine-tuning dataset through manual selection.

[0136] Specifically, we collected open source datasets of Chinese psychological counseling (including SmileChat, SoulChatCorpus, COIG-CQIA, CPsyCounD, D4, etc.) and professional psychology materials, and manually selected and sorted them to form a fine-tuned dataset.

[0137] Step S3: Use the psychological counseling fine-tuning dataset to fully fine-tune the base LLM based on XTuner to obtain a large psychological model suitable for psychological counseling tasks.

[0138] Specifically, full fine-tuning is to adjust all parameters based on the base LLM to adapt it to a specific task. This process uses a smaller learning rate and task-specific data to fully utilize the general features of the pre-trained model and requires more computing resources. XTuner is an efficient, flexible and versatile large model fine-tuning tool library that is compatible with LLM and VLM pre-training and lightweight fine-tuning, and supports multiple fine-tuning algorithms such as LoRA, QLoRA, and full fine-tuning.

[0139] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0140] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A psychological consultation robot based on multimodal data and vertical large model, characterized in that: The psychological consultation robot comprises: an intelligent robot terminal and an intelligent robot backend, wherein the intelligent robot terminal comprises a virtual digital human, a multimodal emotion recognition module, and a data fusion module; the intelligent robot backend comprises: an intelligent agent module, an external knowledge base, and a psychological big model; The virtual digital human is used to realize multi-modal intelligent interaction between man and machine; The multimodal emotion recognition module is used to extract features and classify and recognize images and voice data collected by the voice collection device and image or video collection device carried by the psychological counseling robot when the user is speaking, so as to obtain facial expression recognition results and voice emotion recognition results; The data fusion module is used to fuse the facial expression recognition result and the voice emotion recognition result with the user dialogue text to obtain the emotion and dialogue text fusion result; The intelligent agent module is the control core of the psychological counseling robot, and is used to insert the facial expression recognition results and the voice emotion recognition results into the user dialogue text as prefixes according to a preset format to perform text feature fusion, and obtain a complete interview text sequence; taking the interview text sequence as input, according to the guidance of different symptom prompt information, the psychological big model gives the user's psychological state assessment results based on the interview content, and is also used to realize psychological healing, health management, user health report result checking functions, and is also used to realize RAG functions and interactive control with the psychological big model; The external knowledge base is used to supplement and verify psychological professional knowledge; The psychological big model is a big model fine-tuned based on the open source LLM architecture using the psychological counseling fine-tuning dataset, and is used to implement the computing platform of the psychological counseling robot; The multimodal emotion recognition module includes: a real-time data acquisition submodule, a dynamic expression recognition submodule and a speech recognition submodule; The real-time data acquisition submodule is used to synchronously capture facial images and voice signal sequences of a preset time period during the user's speech during each interview dialogue; The dynamic expression recognition submodule is used to extract deep features from each facial image using a convolutional neural network encoder to obtain a facial feature sequence, then extract spatiotemporal features from the facial feature sequence using a standard Transformer encoder to obtain a word unit encoding sequence, and finally determine the facial expression recognition result based on the word unit encoding sequence; The speech recognition submodule is used to preprocess the speech signal sequence, then use a lightweight audio classification model architecture to extract the Mel-frequency cepstral coefficient features of the speech signal, and finally use an LSTM neural network to classify according to the Mel-frequency cepstral coefficient features to obtain a speech emotion recognition result; Wherein, determining the facial expression recognition result according to the word element encoding sequence includes: A phrase classifier is used to perform calculations based on the word-meta encoding sequence to obtain a facial expression recognition result; wherein the phrase classifier includes a word-meta pooling layer and a fully connected layer; the word-meta pooling layer is used to cross-divide the word-meta encoding sequence into two groups, for each word-meta in the first group, find a most similar word-meta from the second group as a merging target and record its word-meta similarity, and average-merge a preset number of words with the highest matching word-meta similarity ranking in the first group with the corresponding words in the second group. After the merging is completed, the total number of words will be reduced by a preset number, and the remaining words will be spliced ​​in sequence to obtain a word-meta pooling result; the fully connected layer is used to perform classification based on the word-meta pooling result to obtain a facial expression recognition result.

2. The psychological consultation robot based on multimodal data and vertical large model according to claim 1 is characterized in that: The word-unit similarity is: in, For vector A , B The word similarity of A , B For two non-zero dimensional vector, Represents vectors A and B The elements.

3. The psychological consultation robot based on multimodal data and vertical large model according to claim 1 is characterized in that: The intelligent agent module includes: psychological assessment submodule, psychological healing submodule, health management submodule, result checking submodule, RAG function submodule and interactive control submodule; The psychological assessment submodule is used to assess the user's psychological state by combining preliminary screening through psychological interviews and professional assessments using psychological scales; The psychological healing submodule is used to define the content of the dialogue response part based on the psychological big model and use the prompt engineering method to achieve dialogue generation; The health management submodule is used to automatically generate a user health report based on the user's physiological and psychological measurement data to achieve comprehensive health management; The result checking submodule is used to provide a checking function for the report results; The RAG functional submodule is used to first retrieve relevant information from a preset document set or database when the psychological macro model needs to generate text or answer questions, and then use the retrieved information to guide text generation; The interaction control submodule is used to realize the access request to the psychological big model according to the interaction requirements of the intelligent robot terminal.

4. The psychological consultation robot based on multimodal data and vertical large model according to claim 3 is characterized in that: The working process of the psychological assessment submodule includes: Based on the psychological big model, using the prompt engineering method, define prompt information to guide the interview dialogue; the prompt information includes dialogue roles, task descriptions, round limit, questioning methods and empathy support; Inserting the facial expression recognition result and the voice emotion recognition result as prefixes into the user conversation text according to a preset format to perform text feature fusion, thereby obtaining a complete interview text sequence; Taking the interview text sequence as input, according to the guidance of different disease prompt information, the psychological big model gives the user's psychological state assessment result according to the interview content; Using the user's psychological state assessment result as a reference, recommending a specific scale for assessment based on preset custom rules, and obtaining the scale assessment result; A comprehensive user portrait is created based on the user psychological state assessment results and the scale evaluation results; the user portrait includes personal information, interview analysis results and scale evaluation results.

5. The psychological consultation robot based on multimodal data and vertical large model according to claim 4 is characterized in that: The preset format is: [{"role": "user", "content": "<emotion description>X…X"}, {"role": "assistant","content": "X…X"}, …]; Among them, role represents the dialogue role, user corresponds to the user, and assistant corresponds to the big model; content represents the dialogue content of the corresponding role, the prefix <emotion description> contains the user's facial and voice emotion data, and X represents the corresponding dialogue text.

6. The psychological consultation robot based on multimodal data and vertical large model according to claim 3 is characterized in that: The working process of the psychological healing submodule includes: Following the core principles of cognitive behavioral therapy, the dialogue response of the psychological healing submodule involves five components; the first component is confirmation and empathy, the second component is thinking or belief, the third component is challenge or reflection, the fourth component is strategy or insight, and the fifth component is encouragement and prospect; Based on the psychological big model, the prompt engineering method is used to stipulate the contents of the five components to achieve dialogue generation.

7. The psychological consultation robot based on multimodal data and vertical large model according to claim 1 is characterized in that: The external knowledge base is generated based on graph retrieval enhancement.

8. The psychological consultation robot based on multimodal data and vertical large model according to claim 1 is characterized in that: The process of constructing the psychological big model includes: Select Llama 3 8B as the base LLM; Collect Chinese psychological counseling open source data and psychology professional materials, and construct a psychological counseling fine-tuning dataset through manual selection; The psychological counseling fine-tuning dataset is used to fully fine-tune the base LLM based on XTuner to obtain a large psychological model suitable for psychological counseling tasks.

Images