Multi-mode psychological counselor skill recommendation method, device and equipment

Abstract

The invention relates to the technical field of artificial intelligence and psychological health, and discloses a multi-mode psychological counselor skill recommendation method and equipment. The method comprises the following steps: acquiring video, audio and text data in a psychological consultation session, and converting the video, audio and text data into structured data; obtaining the current emotional state of the consultant through multi-modal analysis according to the video, audio and text data after the structure conversion; judging whether psychological intervention skill recommendation is performed or not; if yes, obtaining recommended psychological intervention skills through a trained condition triggering model according to the dialogue between the psychological consultant and the consultant and the current emotional state; and according to the psychological intervention skills and the dialogue between the psychological consultant and the consultant, a dialogue text is generated through a large language model to help the psychological consultant to reply. According to the embodiment of the invention, the recommendation effect of the psychological intervention skills can be improved.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence and mental health technology, and in particular to a multimodal psychological counselor skill recommendation method, device, equipment and storage medium. Background Art

[0002] As the demand for psychological counseling increases, how to help counselors quickly select the optimal intervention techniques in real-time conversations has become the key to improving the effectiveness of psychological counseling. Traditional psychological counseling mainly relies on the experience of counselors, making it difficult to conduct real-time interventions in a systematic manner. In addition, existing technologies are mostly limited to text analysis of a single conversation, resulting in inaccurate recommendations. Summary of the invention

[0003] The main purpose of the present invention is to solve the technical problem of poor recommendation effect of psychological intervention skills.

[0004] A first aspect of the present invention provides a multimodal psychological counselor skill recommendation method, the multimodal psychological counselor skill recommendation method comprising: In a psychological counseling session, video, audio and text data are collected through a camera, a microphone and a text input device, and the video, audio and text data are converted into structured data; According to the video, audio and text data after structural conversion, the current emotional state of the consultant is obtained through multimodal analysis; Determining whether to recommend psychological intervention techniques based on the multimodal analysis of the video, the audio, the text data, and a preset trigger condition; If yes, then based on the conversation between the counselor and the counselee, and the current emotional state, the trained conditional trigger model is used to obtain the recommended psychological intervention skills, wherein the psychological intervention skills include the skill name, skill operation steps, and conversation examples; Based on the psychological intervention techniques and the conversation between the psychological counselor and the counselee, a conversation text is generated through a large language model to help the psychological counselor respond.

[0005] Optionally, in a first implementation of the first aspect of the present invention, generating a dialogue text through a large language model according to the psychological intervention technique and the dialogue between the psychological counselor and the counselee to help the psychological counselor to respond includes: According to the conversation between the psychological counselor and the counselee, identifying the current conversation situation; Inputting the current dialogue situation and the psychological intervention skills into the prompt words of the large language model; According to the prompt words, a dialogue text is generated through the large language model to help the psychological counselor to respond.

[0006] Optionally, in a second implementation of the first aspect of the present invention, judging whether to recommend psychological intervention techniques based on the multimodal analysis of the video, the audio, the text data, and a preset trigger condition includes: Obtaining historical conversation records, wherein the historical conversation records include records of the consultant's emotional changes, feedback on the effectiveness of using historical psychological intervention techniques, and important event nodes; According to the emotion change record, determining whether there is an emotion state in the past that is similar to the current emotion state; If yes, psychological intervention techniques are recommended based on the feedback on the effectiveness of the historical psychological intervention techniques described; Otherwise, it is determined whether to recommend psychological intervention techniques based on the multimodal analysis of the video, the audio, the text data, and the preset trigger conditions.

[0007] Optionally, in a third implementation of the first aspect of the present invention, obtaining the current emotional state of the consultant through multimodal analysis based on the video, audio, and text data after structural conversion includes: According to the video after structural transformation, the facial expression features of the consultant are identified; According to the audio after structural conversion, the voice and intonation features of the consultant are identified; According to the text data after structural conversion, identifying emotional keywords; The current emotional state of the consultant is obtained through multimodal analysis based on the facial expression features, the voice intonation features, and the emotional keywords.

[0008] Optionally, in a fourth implementation of the first aspect of the present invention, the recommended psychological intervention techniques are obtained through a trained conditional triggering model based on the conversation between the psychological counselor and the counselee, and the current emotional state, including: According to the current conversation between the counselor and the counselee and the historical conversation records, the counselee's psychological change trajectory and the effect of the previous psychological intervention techniques are obtained; Obtaining consultation context information according to the psychological change trajectory and the effects of the previous use of psychological intervention techniques; According to the context information and the current emotional state, psychological intervention skills are obtained through a trained conditional trigger model.

[0009] Optionally, in a fifth implementation of the first aspect of the present invention, before obtaining the psychological intervention technique through a trained conditional trigger model according to the context information and the current emotional state, the method further includes: Extract multimodal emotional state data from conversation data between own psychological counselors and counselors; According to the multimodal emotional state data and the preset trigger conditions recommended by the psychological intervention skills, the conversation is analyzed by a large language model to determine whether it meets the preset trigger conditions recommended by the psychological intervention skills, thereby obtaining training data; According to the training data, an initial conditional trigger model is trained to obtain a trained conditional trigger model.

[0010] Optionally, in a sixth implementation of the first aspect of the present invention, after generating a dialogue text through a large language model based on the psychological intervention skills and the dialogue between the psychological counselor and the counselee to help the psychological counselor to respond, the method further includes: Obtaining the effects of using psychological intervention skills fed back by psychological counselors, and filtering the effects of using the psychological intervention skills; According to the filtered effect of the psychological intervention skills, a reinforcement learning algorithm is used to iteratively optimize the large language model.

[0011] The second aspect of the present invention provides a multimodal psychological counselor skill recommendation device, comprising: a memory and at least one processor, the memory storing instructions, the memory and the at least one processor being interconnected via lines; the at least one processor calling the instructions in the memory so that the multimodal psychological counselor skill recommendation device executes the above-mentioned multimodal psychological counselor skill recommendation method.

[0012] The third aspect of the present invention provides a computer-readable storage medium, in which instructions are stored, and when the instructions are run on a computer, the computer executes the above-mentioned multimodal psychological counselor skill recommendation method. In an embodiment of the present invention, in a psychological counseling session, video, audio and text data are collected through a camera, a microphone and a text input device, and the video, the audio and the text data are converted into structured data; according to the video, the audio and the text data after the structure conversion, the current emotional state of the consultant is obtained through multimodal analysis; according to the multimodal analyzed video, the audio, the text data, and the preset trigger conditions, it is determined whether to recommend psychological intervention skills; if so, according to the dialogue between the psychological counselor and the consultant, and the current emotional state, the recommended psychological intervention skills are obtained through the trained conditional trigger model, and the psychological intervention skills include the skill name, skill operation steps and dialogue examples; according to the psychological intervention skills and the dialogue between the psychological counselor and the consultant, a dialogue text is generated through a large language model to help the psychological counselor reply. In the present invention, the current emotional state of the consultant is obtained by analyzing video, audio and text data, and psychological intervention skills are recommended and dialogue text is generated through conditional trigger models and large language models. Through multimodal analysis, preset trigger conditions can be used to trigger the recommendation of psychological intervention skills, thereby fully understanding the content of the dialogue, providing personalized intervention suggestions, and improving the recommendation effect of psychological intervention skills. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 A schematic diagram of an embodiment of a multimodal psychological counselor skill recommendation method in an embodiment of the present invention; Figure 2 A schematic diagram of an embodiment of a multi-modal psychological counselor skill recommendation device in an embodiment of the present invention; Figure 3 A schematic diagram of an embodiment of a multimodal psychological counselor skill recommendation device in an embodiment of the present invention. DETAILED DESCRIPTION

[0014] The embodiments of the present invention provide a multimodal psychological counselor skill recommendation method, device, equipment and storage medium.

[0015] The embodiments disclosed in the present invention will be described in more detail below with reference to the accompanying drawings. Although certain embodiments disclosed in the present invention are shown in the accompanying drawings, it should be understood that the present invention can be implemented in various forms and should not be construed as being limited to the embodiments described herein, which are instead provided for a more thorough and complete understanding of the present invention. It should be understood that the drawings and embodiments disclosed in the present invention are only for exemplary purposes and are not intended to limit the scope of protection disclosed in the present invention.

[0016] In the description of the embodiments disclosed in the present invention, the term "including" and similar terms should be understood as open inclusion, that is, "including but not limited to". The term "based on" should be understood as "based at least in part on". The term "one embodiment" or "the embodiment" should be understood as "at least one embodiment". The terms "first", "second", etc. may refer to different or the same objects. Other explicit and implicit definitions may also be included below.

[0017] For ease of understanding, the specific process of the embodiment of the present invention is described below. Figure 1 In one embodiment of the present invention, a multimodal psychological counselor skill recommendation method includes: S100, in a psychological counseling session, collects video, audio and text data through a camera, a microphone and a text input device, and converts the video, audio and text data into structured data.

[0018] In this embodiment, the video, audio and text data in the video consultation are captured by the AI ​​supervision function. Specifically, first, in the psychological counseling session, real-time video and audio data are collected through the camera and microphone. The camera captures the facial expressions and body movements of the participants, while the microphone records the content of the conversation. Next, the speech recognition module converts the audio data into text information. This process usually includes preprocessing of the speech signal, feature extraction, and speech-to-text conversion using a deep learning model. For example, the use of the Whisper model can improve the accuracy of speech-to-text conversion. At the same time, the video data also needs to be structured, and the video content is analyzed to extract key frames, events, objects, and time information, etc., to form a structured data table. Finally, all these data (text, audio features, video features) are integrated into a structured data set, which can be used for subsequent psychological counseling effect evaluation, sentiment analysis, and behavioral research.

[0019] S200, based on the video, audio and text data after structural conversion, the current emotional state of the consultant is obtained through multimodal analysis.

[0020] In this embodiment, multimodal fusion plays a vital role in the emotion recognition task, and its main purpose is to improve the accuracy and robustness of emotion recognition by integrating information from different modalities (such as video, audio, and text). Specifically, after each modality is predicted independently, these prediction results are fused to generate the final emotional state prediction. For example, the prediction results of video, audio, and text are weighted averaged to generate the final emotional state prediction. In addition, the features of different modalities can be spliced ​​at the model input stage and then input into a joint model for analysis. This method is achieved by creating a single set containing all collected features and then training a single classifier. For example, audio and text features can be spliced ​​into a single feature vector, and then support vector regression is used to predict the emotional value.

[0021] S300, judging whether to recommend psychological intervention techniques based on the multimodal analyzed video, audio, and text data and preset trigger conditions.

[0022] In this embodiment, the system monitors the conversation data in real time by setting a series of trigger conditions (such as specific keywords, emotional thresholds, and intonation changes). When it is detected that the preset trigger conditions are met, the large language model is triggered to recommend psychological intervention skills. Specifically, first, based on the video, audio, and text data, the multimodal fusion model is used to predict the emotional value, analyze the audio features, obtain the consultant's intonation change data during the consultation, and analyze the text data to obtain emotional keywords. Then, the above-mentioned predicted emotional value, intonation change data, and emotional keywords are compared with the specific keywords, emotional thresholds, and intonation changes in the preset trigger conditions to determine whether to recommend psychological intervention skills.

[0023] S400, if yes, then based on the conversation between the counselor and the counselee, as well as the current emotional state, the trained conditional trigger model is used to obtain the recommended psychological intervention techniques.

[0024] In this embodiment, if the preset trigger condition is triggered and it is determined that psychological intervention skills are to be recommended, psychological intervention skills are recommended based on the conversation between the psychological counselor and the counselee, as well as the current emotional state, through the trained conditional trigger model. For example, for emotional problems such as anxiety, stress or depression, the model can give targeted suggestions and encouragement plans, and provide emotional support. Through real-time analysis and automatic recommendation of skills, the system can reduce the decision-making time of counselors in the conversation, enabling them to handle complex psychological counseling scenarios more efficiently.

[0025] Among them, psychological intervention skills include skill names, skill operation steps and dialogue examples. For example, the skill name: Psychological education: The relationship between dieting and eating disorders, aims to help users understand that dieting is the number one culprit of eating disorders, especially anorexia, so that users can understand the harm of dieting and increase their motivation to make changes.

[0026] Operation steps: A1. Recognize how dieting can lead to eating disorders. Goal: Help users understand that when people start to restrict their food intake, their bodies and minds will undergo a series of changes. Studies have shown that hunger and malnutrition can affect brain function, change the structure and chemistry of the brain, and lead to eating disorders. Girls who strictly diet are 18 times more likely to develop eating disorders within six months than those who do not diet. Even if it is not a strict diet, 2 / 3 of new cases of eating disorders come from moderate dieters. This means that any form of dietary control, whether strict or moderate, increases the risk of eating disorders. A2. Explain to users how dieting leads to eating disorders step by step. Goal: Help users learn how dieting leads to eating disorders step by step, and the common manifestations of each stage. First, the process of dieting leading to anorexia usually begins with the restriction of food intake. Many people start to control their diet for the purpose of losing weight or wrong health concepts, which leads to initial weight loss and brings a sense of control and accomplishment. Over time, weight loss gradually stagnates, and the body responds to hunger by reducing the metabolic rate. They will gradually ignore hunger, but the strong hunger response will eventually fight back, prompting them to over-diet or binge eat. At the same time, their emotions and cognition will gradually become distorted, fearing weight gain, worrying too much about food and body shape, and entering a more stringent diet control and vicious cycle.

[0027] Example conversation: 1. User says: “If I don’t control my diet, I will get fat immediately.” 2. User said: "I know I feel weak now, but I can't stop dieting or I will lose control of my weight." S500, based on psychological intervention techniques and the conversation between the counselor and the counselee, generates conversation text through a large language model to help the counselor respond.

[0028] In this embodiment, when generating a reply, a specific role and instruction are set, such as imitating the role of a psychological counselor, and clear instructions are provided to reduce uncertainty. Then, the recommended skills and psychological intervention skills and the dialogue between the psychological counselor and the counselor are input into the prompt words of the large language model, and a suitable dialogue text is generated by the large language model. For example, when the visitor expresses anxiety, the system generates encouraging language to help the counselor respond. When generating a reply, it is necessary to combine the psychological knowledge base and retrieval technology to ensure the relevance and accuracy of the reply and reduce misinformation, which helps to generate a reply that is more in line with the needs of psychological counseling. And letting the model provide multiple versions of reply options, and then selecting the answer that best meets personal needs, can increase the diversity and pertinence of the reply. By providing automated psychological intervention skill recommendations and dialogue generation, the decision-making burden of counselors in high-intensity counseling scenarios can be reduced and the quality of counseling can be improved.

[0029] In an optional implementation of the first aspect of the present invention, based on the psychological intervention skills and the dialogue between the psychological counselor and the counselee, the dialogue text is generated by the large language model to help the psychological counselor to respond, including: Based on the conversation between the psychological counselor and the counselee, the current conversation situation is identified; the current conversation situation and psychological intervention skills are input into the prompt words of the large language model; based on the prompt words, the conversation text is generated through the large language model to help the psychological counselor respond.

[0030] In this embodiment, first, it is necessary to obtain the current psychological counseling dialogue record, including the dialogue content between the counselor and the visitor. This information will be provided to the large language model as input data. Then, the scene recognition model is used to analyze the current dialogue content and identify the specific psychology scene. According to the identified current dialogue situation and current input information, the corresponding first prompt word is generated, and the recommended skills returned by the conditional trigger model are combined with the current dialogue record to construct a clear and specific prompt word. Secondly, after the prompt word is generated, multiple rounds of dialogue tests are performed to ensure that the prompt word is associated with the previous dialogue content. Finally, the constructed prompt word is input into the large language model, and the model will generate a suitable reply based on the prompt word. This process may require multiple iterations to optimize the prompt word to improve the quality and accuracy of the generated reply. In addition, the generated reply can also be evaluated to check whether it meets the expected standards and format requirements. If necessary, the prompt word can be adjusted or the reply can be regenerated to ensure that the final output reply can meet the needs of psychological counseling.

[0031] In an optional implementation of the first aspect of the present invention, judging whether to recommend psychological intervention skills based on multimodal analyzed video, audio, text data, and preset trigger conditions includes: Obtain historical conversation records, which include records of the consultant's emotional changes, feedback on the effectiveness of historical psychological intervention techniques, and important event nodes; based on the records of emotional changes, determine whether there was an emotional state in the past that was similar to the current emotional state; if so, recommend psychological intervention techniques based on the feedback on the effectiveness of historical psychological intervention techniques; otherwise, determine whether to recommend psychological intervention techniques based on multimodal analysis of video, audio, and text data, as well as preset trigger conditions.

[0032] In this embodiment, after each session, the system automatically stores the key information in the consultation process into the memory module to provide personalized continuous conversation support based on the historical conversation records. The memory module includes: the visitor's emotional change records (such as frequent anxiety), historical psychological intervention skills feedback, and important event nodes (such as major life changes). Before recommending psychological intervention skills, first, it is necessary to extract emotional change records, feedback on the use of psychological intervention skills, and important event nodes from the historical conversation records of the memory module. By analyzing the emotional changes in the historical conversation records, it is identified whether the past emotional state is similar to the current emotional state. For example, if the historical records show that the visitor was anxious due to work pressure and was relieved through relaxation training techniques, then it can be considered whether the current emotional state is also related to anxiety. If it is found that the past emotional state is similar to the current emotional state, it is necessary to check the feedback on the use of historical psychological intervention skills and automatically recommend psychological intervention skills that have been effective. For example, if cognitive behavioral therapy has effectively relieved anxiety before, then the skill can be recommended again. If there is no similar emotional state in the historical records, or the current emotional state needs to be further evaluated, it is determined whether the psychological intervention skills need to be recommended based on the results of multimodal analysis and the preset trigger conditions. By integrating multimodal data and memory modules, the system can provide more accurate and personalized psychological intervention skills recommendations based on the client's historical emotional changes and behavioral patterns.

[0033] In an optional implementation of the first aspect of the present invention, the current emotional state of the consultant is obtained through multimodal analysis based on the video, audio and text data after structural conversion, including: Based on the video after structural conversion, the consultant's facial expression features are identified; based on the audio after structural conversion, the consultant's voice and intonation features are identified; based on the text data after structural conversion, emotional keywords are identified; based on the facial expression features, voice and intonation features, and emotional keywords, the consultant's current emotional state is obtained through multimodal analysis.

[0034] In this embodiment, the collected data is analyzed in real time, and technologies such as sentiment analysis, semantic understanding, expression recognition and intonation analysis are used to fully understand the visitor's current emotions and psychological state. Specifically, in sentiment analysis, emotional keywords such as "anxiety" and "loneliness" are identified based on text content. This can use NLP technology to perform sentiment analysis and semantic understanding on the text to capture subtle changes in emotions and potential psychological distress. Specifically, the user's voice text is first passed into the large language model (LLM) to automatically identify one of the 10 fixed emotions. Then, the emotion obtained by text recognition will be passed to the LLM together with the emotion of video and voice recognition for reasoning to obtain empathetic responses. In intonation analysis, potential emotional fluctuations are identified by analyzing the changes in the intonation of the voice, such as rapid speech speed indicating tension, which can be achieved by analyzing the pitch, timbre, speech speed and other features of the voice signal. In facial expression analysis, facial expressions (such as smiling and frowning) are identified based on video data to infer emotional states. This usually includes using deep learning models such as convolutional neural networks (CNNs) to analyze facial action units (AUs) in the video to identify the consultant's facial expressions. The information of the above three modalities (facial expression features, voice intonation features and emotional keywords) are integrated and analyzed to obtain the consultant's current emotional state.

[0035] In an optional implementation of the first aspect of the present invention, based on the conversation between the psychological counselor and the counselee, and the current emotional state, the recommended psychological intervention techniques obtained through the trained conditional trigger model include: Based on the current conversation between the counselor and the counselee and the historical conversation records, the counselee's psychological change trajectory and the previous effects of using psychological intervention skills are obtained; based on the psychological change trajectory and the previous effects of using psychological intervention skills, the contextual information of the consultation is obtained; based on the contextual information and the current emotional state, the psychological intervention skills are obtained through the trained conditional trigger model.

[0036] In this embodiment, by combining the current conversation between the counselor and the counselee with the historical records in the memory module, the system can understand the client's psychological change trajectory and the effects of the previous psychological intervention techniques. Combined with the psychological change trajectory and the effects of the previous psychological intervention techniques, the context information of the consultation is obtained, which also includes understanding the counselee's main complaint, personal background, growth experience and other information, and integrating and evaluating this information. According to the context information and the current emotional state, the appropriate psychological intervention technique is determined through the trained conditional trigger model.

[0037] In an optional implementation of the first aspect of the present invention, before obtaining the psychological intervention skills through the trained conditional trigger model according to the context information and the current emotional state, the method further includes: Extract multimodal emotional state data from the conversation data between the company's own psychological counselors and counselees; analyze whether the conversation meets the preset trigger conditions recommended by the psychological intervention skills based on the multimodal emotional state data and the preset trigger conditions recommended by the psychological intervention skills through a large language model to obtain training data; train an initial conditional trigger model based on the training data to obtain a trained conditional trigger model.

[0038] In this embodiment, first, training data is constructed to extract multimodal emotional state data from the conversation data between the own psychological counselors and the counselors, which includes data in multiple modes such as text, voice, and video to fully capture emotional information. Then, the multimodal emotional state, conversation data, and the trigger conditions of 127 psychological intervention skills summarized by the psychological team are input into the large model to analyze whether the conversation meets the trigger conditions of the psychological intervention skills, and a large amount of training data is obtained. The training data is then used to train the model to obtain a conditional trigger model, which can input the conversation between the psychological counselor and the counselor and the multimodal analysis, and output the intervention analysis and the name of the intervention skill that meets the trigger conditions.

[0039] In an optional implementation of the first aspect of the present invention, after generating a dialogue text through a large language model based on the psychological intervention skills and the dialogue between the psychological counselor and the counselee to help the psychological counselor to respond, the method further includes: Obtain the effects of psychological intervention skills feedback from psychological counselors and filter the effects of psychological intervention skills; based on the filtered effects of psychological intervention skills, use reinforcement learning algorithm to iteratively optimize the large language model.

[0040] In this embodiment, based on the counselor's feedback on the use of psychological intervention techniques, a reinforcement learning algorithm is used to continuously optimize the recommendation strategy, so that the system can better adapt to different consultation scenarios and individual differences. As the system accumulates more session data, the memory module automatically updates the history to ensure that the recommended psychological intervention techniques are always based on the latest contextual information.

[0041] See also Figure 2 The second aspect of the present invention provides a multimodal psychological counselor skill recommendation device, the multimodal psychological counselor skill recommendation device comprising: The data conversion module 10 is used to collect video, audio and text data through a camera, a microphone and a text input device in a psychological counseling session, and convert the video, audio and text data into structured data; The emotional state analysis module 20 is used to obtain the current emotional state of the consultant through multimodal analysis based on the video, audio and text data after the structure conversion; The psychological intervention judgment module 30 is used to judge whether to recommend psychological intervention techniques based on the multimodal analyzed video, audio, and text data and preset trigger conditions; The psychological intervention recommendation module 40 is used to obtain the recommended psychological intervention skills according to the dialogue between the psychological counselor and the counselee and the current emotional state through the trained condition trigger model. The psychological intervention skills include the skill name, skill operation steps and dialogue examples; The dialogue text generation module 50 is used to generate dialogue text through a large language model according to the psychological intervention skills and the dialogue between the psychological counselor and the counselee, so as to help the psychological counselor to respond.

[0042] In an optional implementation of the second aspect of the present invention, the dialogue text generation module 50 is also used to identify the current dialogue situation based on the dialogue between the psychological counselor and the counselee; input the current dialogue situation and psychological intervention skills into the prompt words of the large language model; and generate a dialogue text based on the prompt words through the large language model to help the psychological counselor respond.

[0043] In an optional implementation of the second aspect of the present invention, the psychological intervention judgment module 30 is also used to obtain historical conversation records, which include records of the consultant's emotional changes, historical feedback on the effectiveness of using psychological intervention techniques, and important event nodes; based on the records of emotional changes, it is judged whether there were emotional states in the past that were similar to the current emotional state; if so, psychological intervention techniques are recommended based on historical feedback on the effectiveness of using psychological intervention techniques; otherwise, based on the video, audio, and text data of multimodal analysis, and preset trigger conditions, it is judged whether to recommend psychological intervention techniques.

[0044] In an optional implementation of the second aspect of the present invention, the emotional state analysis module 20 is also used to identify the consultant's facial expression features based on the video after structural conversion; identify the consultant's voice and intonation features based on the audio after structural conversion; identify emotional keywords based on the text data after structural conversion; and obtain the consultant's current emotional state through multimodal analysis based on the facial expression features, voice and intonation features, and emotional keywords.

[0045] In an optional implementation of the second aspect of the present invention, the psychological intervention recommendation module 40 is also used to obtain the consultant's psychological change trajectory and the previous effects of using psychological intervention skills based on the current conversation between the psychological counselor and the consultant and the historical conversation records; obtain the context information of the consultation based on the psychological change trajectory and the previous effects of using psychological intervention skills; and obtain the psychological intervention skills through the trained conditional trigger model based on the context information and the current emotional state.

[0046] In an optional implementation of the second aspect of the present invention, the multimodal psychological counselor skill recommendation device also includes a model training module, which is used to extract multimodal emotional state data from the conversation data between the psychological counselor and the consultant; based on the multimodal emotional state data and the preset trigger conditions for the psychological intervention skill recommendation, the conversation is analyzed through a large language model to see whether it meets the preset trigger conditions for the psychological intervention skill recommendation, and training data is obtained; based on the training data, an initial conditional trigger model is trained to obtain a trained conditional trigger model.

[0047] In an optional implementation of the second aspect of the present invention, the multimodal psychological counselor skill recommendation device also includes a model optimization module, which is used to obtain the effects of using psychological intervention skills fed back by psychological counselors and filter the effects of using psychological intervention skills; based on the filtered effects of using psychological intervention skills, a reinforcement learning algorithm is used to iteratively optimize the large language model.

[0048] Figure 3 1 is a schematic diagram of the structure of a multimodal psychological counselor skill recommendation device provided by an embodiment of the present invention. The multimodal psychological counselor skill recommendation device 500 may have relatively large differences due to different configurations or performances, and may include one or more processors (central processing units, CPU) 510 (for example, one or more processors) and a memory 520, and one or more storage media 530 (for example, one or more mass storage devices) storing application programs 533 or data 532. Among them, the memory 520 and the storage medium 530 can be short-term storage or permanent storage. The program stored in the storage medium 530 may include one or more modules (not shown in the figure), and each module may include a series of instruction operations in the multimodal psychological counselor skill recommendation device 500. Furthermore, the processor 510 may be configured to communicate with the storage medium 530 to execute a series of instruction operations in the storage medium 530 on the multimodal psychological counselor skill recommendation device 500.

[0049] The multimodal psychologist skill recommendation device 500 may also include one or more power supplies 540, one or more wired or wireless network interfaces 550, one or more input and output interfaces 560, and / or one or more operating systems 531, such as Windows Server, Mac OS X, Unix, Linux, Free BSD, etc. Those skilled in the art will appreciate that Figure 3 The structure of the multimodal psychologist skill recommendation device shown does not constitute a limitation on the multimodal psychologist skill recommendation device, and may include more or fewer components than shown in the figure, or a combination of certain components, or a different arrangement of components.

[0050] The present invention also provides a computer-readable storage medium, which may be a non-volatile computer-readable storage medium or a volatile computer-readable storage medium. The computer-readable storage medium stores instructions, which, when executed on a computer, enable the computer to execute the steps of the multimodal psychological counselor skill recommendation method.

[0051] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0052] In addition, although each operation is described in a specific order, this should be understood as requiring such operation to be performed in the specific order shown or in a sequential order, or requiring that all illustrated operations should be performed to obtain desired results. Under certain circumstances, multitasking and parallel processing may be advantageous. Similarly, although some specific implementation details are included in the above discussion, these should not be interpreted as limiting the scope of the present disclosure. Some features described in the context of a separate embodiment can also be implemented in a single implementation in combination. On the contrary, the various features described in the context of a single implementation can also be implemented in multiple implementations individually or in any suitable sub-combination mode.

[0053] Although the subject matter has been described in language specific to structural features and / or methodological logical actions, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. On the contrary, the specific features and actions described above are merely example forms of implementing the claims.

Claims

1. A multimodal psychological counselor skill recommendation method, characterized in that: The multimodal psychological counselor skill recommendation method includes: In a psychological counseling session, video, audio and text data are collected through a camera, a microphone and a text input device, and the video, audio and text data are converted into structured data; According to the video, audio and text data after structural conversion, the current emotional state of the consultant is obtained through multimodal analysis; Determining whether to recommend psychological intervention techniques based on the multimodal analysis of the video, the audio, the text data, and a preset trigger condition; If yes, then based on the conversation between the counselor and the counselee, and the current emotional state, the trained conditional trigger model is used to obtain the recommended psychological intervention skills, wherein the psychological intervention skills include the skill name, skill operation steps, and conversation examples; Based on the psychological intervention techniques and the conversation between the psychological counselor and the counselee, a conversation text is generated through a large language model to help the psychological counselor respond.

2. The multimodal psychological counselor skill recommendation method according to claim 1, characterized in that: The generating of the dialogue text by the large language model according to the psychological intervention skills and the dialogue between the psychological counselor and the counselee to help the psychological counselor to respond includes: According to the conversation between the psychological counselor and the counselee, identifying the current conversation situation; Inputting the current dialogue situation and the psychological intervention skills into the prompt words of the large language model; According to the prompt words, a dialogue text is generated through the large language model to help the psychological counselor to respond.

3. The multimodal psychological counselor skill recommendation method according to claim 1, characterized in that: The determining whether to recommend psychological intervention techniques based on the multimodal analysis of the video, the audio, the text data, and the preset trigger condition includes: Obtaining historical conversation records, wherein the historical conversation records include records of the consultant's emotional changes, feedback on the effectiveness of using historical psychological intervention techniques, and important event nodes; According to the emotion change record, determining whether there is an emotion state in the past that is similar to the current emotion state; If yes, psychological intervention techniques are recommended based on the feedback on the effectiveness of the historical psychological intervention techniques described; Otherwise, it is determined whether to recommend psychological intervention techniques based on the multimodal analysis of the video, the audio, the text data, and the preset trigger conditions.

4. The multimodal psychological counselor skill recommendation method according to claim 1, characterized in that: The video, audio and text data after structural conversion are analyzed through multimodal analysis to obtain the current emotional state of the consultant, including: According to the video after structural transformation, the facial expression features of the consultant are identified; According to the audio after structural conversion, the voice and intonation features of the consultant are identified; According to the text data after structural conversion, identifying emotional keywords; The current emotional state of the consultant is obtained through multimodal analysis based on the facial expression features, the voice intonation features, and the emotional keywords.

5. The multimodal psychological counselor skill recommendation method according to claim 1, characterized in that: The psychological intervention techniques recommended by the trained conditional trigger model based on the conversation between the psychological counselor and the counselee, as well as the current emotional state, include: According to the current conversation between the counselor and the counselee and the historical conversation records, the counselee's psychological change trajectory and the effect of the previous psychological intervention techniques are obtained; Obtaining consultation context information according to the psychological change trajectory and the effects of the previous use of psychological intervention techniques; According to the context information and the current emotional state, psychological intervention skills are obtained through a trained conditional trigger model.

6. The multimodal psychological counselor skill recommendation method according to claim 5, characterized in that: Before obtaining the psychological intervention skills through the trained conditional trigger model according to the context information and the current emotional state, the method further includes: Extract multimodal emotional state data from conversation data between own psychological counselors and counselors; According to the multimodal emotional state data and the preset trigger conditions recommended by the psychological intervention skills, the conversation is analyzed by a large language model to determine whether it meets the preset trigger conditions recommended by the psychological intervention skills, thereby obtaining training data; According to the training data, an initial conditional trigger model is trained to obtain a trained conditional trigger model.

7. The multimodal psychological counselor skill recommendation method according to claim 1, characterized in that: After generating a dialogue text through a large language model based on the psychological intervention skills and the dialogue between the psychological counselor and the counselee to help the psychological counselor to respond, the method further includes: Obtaining the effects of using psychological intervention skills fed back by psychological counselors, and filtering the effects of using the psychological intervention skills; According to the filtered effect of the psychological intervention skills, a reinforcement learning algorithm is used to iteratively optimize the large language model.

8. A multimodal psychological counselor skill recommendation device, characterized in that: The multimodal psychological counselor skill recommendation device includes: A data conversion module is used to collect video, audio and text data through a camera, a microphone and a text input device in a psychological counseling session, and convert the video, audio and text data into structured data; An emotional state analysis module, used to obtain the current emotional state of the consultant through multimodal analysis based on the video, audio and text data after structural conversion; A psychological intervention judgment module, used to judge whether to recommend psychological intervention techniques based on the multimodal analysis of the video, the audio, the text data, and a preset trigger condition; A psychological intervention recommendation module is used to obtain recommended psychological intervention skills based on the conversation between the psychological counselor and the counselee, as well as the current emotional state, through the trained conditional trigger model. The psychological intervention skills include skill names, skill operation steps, and conversation examples; The dialogue text generation module is used to generate dialogue text through a large language model based on the psychological intervention skills and the dialogue between the psychological counselor and the counselee to help the psychological counselor respond.

9. A multimodal psychological counselor skill recommendation device, characterized in that: The multimodal psychological counselor skill recommendation device comprises: a memory and at least one processor, the memory stores instructions, and the memory and the at least one processor are interconnected via a line; The at least one processor calls the instructions in the memory to enable the multimodal psychological counselor skill recommendation device to execute the multimodal psychological counselor skill recommendation method as described in any one of claims 1-7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the multimodal psychological counselor skill recommendation method as described in any one of claims 1-7 is implemented.

Images