Mental health support dialogue generation method based on multi-modal input

Abstract

The invention discloses a psychological health support dialogue generation method based on multi-modal input. The method comprises the steps that 1, text data are acquired and preprocessed; 2, calculating context similarity to obtain a high-quality example set; 3, extracting key visual clues from the image by utilizing a visual language model, wherein the key visual clues comprise facial expressions, actions and emotions; 4, integrating an external common knowledge base to understand the emotion and intention of the user; 5, generating a consultation strategy by using a large language model; and 6, dialogue generation in combination with the context. According to the method, the visual information, the dialogue context and the consultation strategy are comprehensively utilized, the performance of a large language model in the psychological health dialogue is enhanced, more detailed and similar supportive replies can be generated, the emotion recognition and response quality are remarkably improved, the emotion state of a user can be comprehensively understood and responded, and the user experience is improved. Therefore, the mental health support effect is effectively improved.

Description

Technical Field

[0001] The present invention relates to the technical field of multimodal human-computer interaction and natural language processing, and in particular to a method for generating a psychological health support dialogue based on multimodal input. Background Art

[0002] Currently, artificial intelligence technology is widely used to identify emotional states and predict the development trajectory of psychological disorders to achieve personalized medical support. Furthermore, artificial intelligence-based mental health agents can simulate the conversation between psychotherapists and patients, effectively helping to reduce users' stress, anxiety and depression symptoms. The application of such technology will help improve the accessibility of mental health services in areas where professionals are relatively scarce, so that more people can benefit from high-quality mental health support and services.

[0003] The research on mental health dialogue systems has gone through a process from manual rule construction to the integration of advanced models and technologies. Early methods mainly relied on manually formulated rules to generate responses with empathy characteristics. Although they were effective in specific situations, they lacked adaptability. Subsequently, some scholars combined rules with algorithms to assist in generating supportive responses by analyzing stress-related texts in social media. This type of method has improved the naturalness and relevance of the dialogue to a certain extent, but it is still limited by the scale of the model and the singleness of data features. Since then, generative dialogue models based on neural networks have begun to develop, and deep learning technology has helped chatbots understand and respond to social support needs. However, most of these models rely on large-scale and high-quality pre-training data, which is still insufficient for the scenario in the field of mental health that requires refined and professional processing. Although existing studies have attempted to introduce large language models (LLMs) and sentiment computing, there are still problems with insufficient data customization and professional motivation design in the context of mental health.

[0004] On the other hand, most current mental health dialogue systems are mainly based on text input, which makes it difficult to fully identify and present the user's emotional and behavioral status. Emotional expressions often have multimodal characteristics, such as facial expressions and body movements. Relying solely on text will lead to an incomplete and inaccurate understanding of the user's emotional state. Although some attempts have been made to introduce multimodal information into mental health support, such research is still relatively rare.

[0005] How to effectively integrate multimodal information (such as facial expressions and movements in images) with dialogue context and external knowledge to enhance the depth and accuracy of the dialogue system's understanding of user emotions and behavioral states, thereby meeting the needs of refined and professional data processing and intervention in the field of mental health, is still an issue that urgently needs to be studied and improved. Summary of the invention

[0006] In order to overcome the shortcomings of the prior art, the present invention proposes a method for generating mental health support dialogues based on multimodal input, in order to significantly improve the quality of emotion recognition and response, so as to more comprehensively understand and respond to the user's emotional state, and enhance the ability to generate reply responses for mental health support.

[0007] In order to achieve the above-mentioned purpose, the present invention adopts the following technical scheme:

[0008] The method for generating a psychological health support dialogue based on multimodal input of the present invention is characterized in that it is performed according to the following steps:

[0009] Step 1: Obtain a multimodal dataset , including: training set and test set ; The training set Any training sample in , the test set Any test sample in ;in, and are respectively a training text utterance sequence containing n sentences of dialogue and a test text utterance sequence containing m sentences of dialogue, and They are Visual image data, Visual image data;

[0010] Step 2: and Perform semantic processing and calculate the similarity between the two to obtain a high-quality example set ;

[0011] Step 3: Select a multimodal conversational language model As a visual language model, and constructing prompt text templates for generating facial expressions and actions , and then use formula (6) to generate Facial expressions and action descriptions :

[0012] (6)

[0013] Step 4: Generate according to the relationship type Comprehensive common sense knowledge text ;

[0014] Step 5: Generate using large language model LLM Consulting strategy text ;

[0015] Step 6: , , , as well as Integrate into prompt text And input into the large language model LLM to generate Final response content The method for generating a mental health support dialogue based on multimodal input according to the present invention is also characterized in that step 2 is performed as follows:

[0016] Step 2.1: All text utterances except the last sentence in the dialogue are concatenated to form a test context sequence , and use a sentence-level semantic encoder Will Mapped to a high-dimensional semantic vector space to obtain a test representation vector ;

[0017] Step 2.2: The m sentences in the dialogue are concatenated to form a training context sequence , and use a sentence-level semantic encoder Will Mapped to a high-dimensional semantic vector space to obtain a training representation vector ;

[0018] Step 2.3: Calculate using formula (5) and Similarity :

[0019] (5)

[0020] In formula (5), is the semantic similarity function represented by cosine similarity; represents the vector inner product operation, Represents the Euclidean norm of the vector; Step 2.4, according to the pre-set few sample parameters ,in, is a positive integer, from Select the top sample with the highest semantic similarity to the test sample training samples and form a high-quality example set .

[0021] Further, step 4 is performed as follows:

[0022] Step 4.1: Use the fine-tuned common sense reasoning model right With relationship type Process and generate Common sense inference text ;in, ∈ , Represents a set of relationship types;

[0023] Step 4.2: Use formula (8) to fuse the common sense inference texts of all relationship types in R to generate comprehensive common sense knowledge texts. :

[0024] (8)

[0025] In formula (8), Represents a text string concatenation operation.

[0026] Further, step 5 is performed as follows:

[0027] Step 5.1: The large language model LLM uses formula (9) to infer The psychological and emotional stage :

[0028] (9)

[0029] In formula (9), is a prompt text for psychological emotion stage identification;

[0030] Step 5.2: Construct the consultation strategy guidance text , and with and Input the large language model LLM together to get Consulting strategy text .

[0031] An electronic device of the present invention includes a memory and a processor, wherein the memory is used to store a program that supports the processor to execute the dialogue generation method, and the processor is configured to execute the program stored in the memory.

[0032] The present invention provides a computer-readable storage medium, wherein a computer program is stored on the computer-readable storage medium, and the computer program executes the steps of the dialogue generation method when the computer program is executed by a processor.

[0033] Compared with the prior art, the present invention has the following beneficial effects:

[0034] 1. This paper designs a new multimodal framework that integrates visual information to enhance the mental health support response generated by the LLM model. Compared with the current mainstream psychological support methods, the method of the present invention can more comprehensively understand the user's emotional state and capture richer emotional expressions, thereby generating more accurate and targeted supportive responses.

[0035] 2. The present invention uses zero-sample and few-sample learning techniques to select the most relevant samples by calculating semantic similarity, thereby improving the adaptability of the LLM model in the field of psychotherapy without large-scale training.

[0036] 3. The method of the present invention adopts corresponding strategies to guide the model's response generation. This strategy guidance can help the model generate more appropriate responses at different stages of the dialogue, thereby improving the coherence and effectiveness of the dialogue.

[0037] 4. The present invention constructs dedicated data processing and LLM prompting technology tailored for mental health environments, enhancing the responsiveness and sensitivity of the LLM system, which is crucial for effective mental health support. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 It is a schematic diagram of the process of the present invention;

[0039] Figure 2 This is a template structure diagram of the prompts used in the present invention. DETAILED DESCRIPTION

[0040] In this embodiment, a method for generating mental health support dialogues based on multimodal input is to use multimodal input to enhance the ability of LLM in generating responses for mental health support. The method integrates a visual language model to analyze images and capture facial expressions, movements, and emotions. It then combines this visual data with the conversation context and consultation strategies to produce more subtle and supportive responses, such as Figure 1 As shown, the specific steps are as follows:

[0041] Step 1: Obtain a multimodal dataset , including: training set and test set ; Training set Any training sample in is denoted as , test set Any test sample in ; The test samples , is the number of text utterances in the test sample, , is the number of text utterances in the training sample, and They are Visual image data, Visual image data.

[0042] Step 2: and Perform semantic processing and calculate the similarity between the two to obtain a high-quality example set ;

[0043] Step 2.1: All text utterances except the last sentence in the dialogue are concatenated to form the test context sequence using formula (1): :

[0044] (1) Using sentence-level semantic encoder Will Mapped to a high-dimensional semantic vector space, the test representation vector is obtained using formula (2) : (2)

[0045] Step 2.2: The m sentences in the dialogue are concatenated to form the training context sequence using formula (3) : (3) Using sentence-level semantic encoder Will Mapped to a high-dimensional semantic vector space, the training representation vector is obtained using formula (4) : (4)

[0046] Step 2.3: Calculate using formula (5) and Similarity :

[0047] (5)

[0048] In formula (5), is the semantic similarity function represented by cosine similarity; represents the vector inner product operation, Represents the Euclidean norm of the vector. Step 2.4, according to the pre-set few-sample parameters ,in, is a positive integer, from Select the top sample with the highest semantic similarity to the test sample training samples and form a high-quality example set .

[0049] Step 3: Select a multimodal conversational language model with excellent image recognition capabilities (e.g. GPT-4) as a visual language model and build prompt text templates for generating facial expressions and actions ,in, is a predefined text prompt used to guide the model to extract the visual description of the image. The text prompt used here is: Please describe the expression and action of the person in the picture. Then use formula (6) to generate Facial expressions and action descriptions :

[0050] (6)

[0051] Generated As a visual context element to further the conversation process, it is an The final comprehensive representation, together with the dialogue context and example set, provides richer visual and semantic support for subsequent tasks.

[0052] Step 4: Generate according to the relationship type Comprehensive common sense knowledge text ;

[0053] Step 4.1: Traditional dialogue generation methods rely only on historical conversation data, which makes it difficult to fully understand the user's situation and emotions. Introducing commonsense knowledge graphs in mental health support conversations can enhance the understanding of implicit information, enhance cognitive empathy, and overcome the limitations of relying on conversation history. In order to generate context-related commonsense reasoning, the COMET model based on the GPT architecture and fine-tuned using the ATOMIC dataset is used to introduce commonsense knowledge; the COMET model can generate commonsense for four types of relationships, including: Characterize the conditions required before speaking; Characterize the impact of speech; characterizes the speaker’s intention; Represent the speaker’s emotional or attitudinal response; use a fine-tuned common sense reasoning model right With relationship type Processing is performed to generate the equation (7) about Common sense inference text :

[0054] (7)

[0055] In formula (7), ∈ , Represents a collection of relationship types.

[0056] Step 4.2: Use formula (8) to fuse the common sense inference texts of all relationship types in R to generate comprehensive common sense knowledge texts. :

[0057] (8)

[0058] In formula (8), Represents a text concatenation operation; It can provide more complete contextual common sense support for subsequent dialogue generation and understanding.

[0059] Step 5: Generate using large language model LLM Consulting strategy text ;

[0060] Step 5.1: Introduce the emotional support dialogue framework to help the large language model identify key strategies consistent with the psychotherapy stage in the psychological counseling scenario, thereby promoting emotional exploration, deepening of understanding, and behavioral change in the dialogue, and ensuring its safety and autonomy; the large language model LLM uses formula (9) to infer The psychological and emotional stage :

[0061] (9)

[0062] In formula (9), This is the prompt text for psychological and emotional stage identification. The prompt text here is specific: "Based on the context, these three stages (exploration: helping visitors identify problems; insight: helping visitors reach a new depth of self-understanding; action: helping visitors decide on actions to solve problems) belong to which type of conversation? Based on the stage, explain why you make such a judgment."

[0063] Step 5.2: After identifying the dialogue stage, construct the consultation strategy guidance prompt text , and and Input the large language model LLM together, and then use formula (10) to select the consulting strategy text that matches the recognition stage from the predefined strategy set : (10)

[0064] In formula (10), Indicates the policy text determined by LLM, The corresponding prompt text is: "Based on the different stages, tell me what strategy should be used in the next response. (One of the following seven stages: asking questions, restating or reformulating the problem, reflecting emotions, self-disclosure, affirmation and reassurance, providing advice and information.) (The exploration stage corresponds to: asking questions, restating or reformulating the problem, reflecting emotions, and self-disclosure. The insight stage corresponds to: reflecting emotions, self-disclosure, affirmation and comfort. The action stage corresponds to: self-disclosure, affirmation and reassurance, providing advice and information.) Use [] to enclose the selected strategy. Here is an example: Based on the context, this conversation is in the Action stage. The next response should use the (self-disclosure) strategy; In the above steps, LLM will select one of the seven strategies provided based on the current stage of the conversation and mark the selected strategy in "[ ]" brackets."

[0065] Step 6: Use formula (11) to , , , as well as Integrate into prompt text : (11) In formula (11), Indicates that the mental health support dialogue generates a task prompt, and the specific prompt text is: "This is a psychological counseling task: In this task, the client (the first person) will share their feelings, experiences, and challenges with the counselor. The counselor (the second person) needs to provide mental health and emotional support. The counselor's job is not to provide solutions or advice, but to help clients explore their feelings through listening, empathy, and open-ended questions. The counselor should promote the client's self-understanding and growth, and help them identify and deal with inner conflicts or distress. In addition, the counselor should maintain professional boundaries, maintain confidentiality, and guide the client to seek more professional medical or mental health services when necessary. In this particular conversation, you will play the role of a counselor and should give your next round of responses based on the information provided by the client." It is composed of multiple components, and the order of the components will affect the generation results of the model; a specially designed prompt template for multimodal psychological support scenarios is used to assist LLM in better meeting the requirements of the consultation task in subsequent responses. The content and structure of the prompt template are shown below: Figure 2As shown. The task definition provides a comprehensive description of the tasks and functionalities required by the LLM. Samples for contextual learning are selected based on zero-shot or few-shot settings to enhance the model's understanding of the task. Expression description analyzes visual images to capture the searcher's facial expressions and movements, enriching the text-based input. External knowledge supplements the conversation context with additional relevant knowledge. Policy guidance involves selecting strategic content based on the context to guide the model's response. The final conversation context includes the conversation history, excluding the counselor's last utterance. The main goal of this arrangement is to enable the LLM to simulate the role of the counselor and effectively generate responses for subsequent rounds. Input into the large language model LLM, and then use formula (12) to generate Final response content :

[0066] (12)

[0067] In this embodiment, an electronic device includes a memory and a processor, wherein the memory is used to store a program that supports the processor to execute the above method, and the processor is configured to execute the program stored in the memory.

[0068] In this embodiment, a computer-readable storage medium stores a computer program on the computer-readable storage medium, and the computer program executes the steps of the above method when executed by a processor.

Claims

1. A method for generating mental health support dialogue based on multimodal input, characterized in that: The steps are as follows: Step 1: Obtain multimodal dataset , including: training set and test set ; The training set Any training sample in , the test set Any test sample in ;in, and are respectively a training text utterance sequence containing n sentences of dialogue and a test text utterance sequence containing m sentences of dialogue, and They are Visual image data, Visual image data; Step 2: and Perform semantic processing and calculate the similarity between the two to obtain a high-quality example set ; Step 3: Select a multimodal conversational language model As a visual language model, and constructing prompt text templates for generating facial expressions and actions , and then use formula (6) to generate Facial expressions and action descriptions : (6) Step 4: Generate according to the relationship type Comprehensive common sense knowledge text ; Step 5: Generate using large language model LLM Consulting strategy text ; Step 6: , , , as well as Integrate into prompt text And input into the large language model LLM to generate Final response content .

2. According to claim 1, a method for generating a mental health support dialogue based on multimodal input is characterized in that: Described step 2 is carried out as follows: Step 2.1: All text utterances except the last sentence in the dialogue are concatenated to form a test context sequence , and use a sentence-level semantic encoder Will Mapped to a high-dimensional semantic vector space to obtain a test representation vector ; Step 2.2: The m sentences in the dialogue are concatenated to form a training context sequence , and use a sentence-level semantic encoder Will Mapped to a high-dimensional semantic vector space to obtain a training representation vector ; Step 2.3: Calculate using formula (5) and Similarity : (5) In formula (5), is the semantic similarity function represented by cosine similarity; represents the vector inner product operation, Represents the Euclidean norm of a vector; Step 2.4: According to the pre-set few-sample parameters ,in, is a positive integer, from Select the top sample with the highest semantic similarity to the test sample training samples and form a high-quality example set .

3. The method for generating a mental health support dialogue based on multimodal input according to claim 2, characterized in that: Step 4 is performed as follows: Step 4.1: Use the fine-tuned common sense reasoning model right With relationship type Process and generate Common sense inference text ;in, ∈ , Represents a set of relationship types; Step 4.2: Use formula (8) to fuse the common sense inference texts of all relationship types in R to generate comprehensive common sense knowledge texts. : (8) In formula (8), Represents a text string concatenation operation.

4. The method for generating a mental health support dialogue based on multimodal input according to claim 3, characterized in that: Step 5 is performed as follows: Step 5.1: The large language model LLM uses formula (9) to infer The psychological and emotional stage : (9) In formula (9), is a prompt text for psychological emotion stage identification; Step 5.2: Construct the consultation strategy guidance text , and and Input the large language model LLM together to get Consulting strategy text .

5. An electronic device, comprising a memory and a processor, characterized in that: The memory is used to store a program that supports the processor to execute the dialogue generation method described in any one of claims 1 to 4, and the processor is configured to execute the program stored in the memory.

6. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the dialog generation method according to any one of claims 1 to 4 are executed.

Images