Intelligent psychological counseling system based on multi-mode large model and cognitive behavior therapy

Abstract

The invention provides an intelligent psychological counseling system based on a multi-mode large model and a cognitive behavior therapy, and the system comprises the steps: recognizing voice from a user side through the multi-mode large model, and obtaining text information and emotional state information; analyzing the text information by using a large language model, carrying out conversation with the user side, and extracting psychological state information of the user side from the conversation; a cognitive behavior treatment strategy matched with the user side is determined according to the emotional state information, so that a voice response to the user side is generated according to the matched cognitive behavior treatment strategy by using a multi-mode large model; by combining a multi-modal large model and a large language model, accurate and efficient natural language understanding of a user side is realized, highly anthropomorphic voice response is provided, dependence on professional psychological consultants is reduced, limitation of psychological consultation services on time and regions can be broken through, large-scale consultation service requirements can be met in an all-weather and low-cost manner, and the psychological consultation service experience is improved. The man-machine interaction experience in the consultation process is improved, and high-quality and high-practicability psychological consultation and treatment services are provided.

Description

Technical Field

[0001] The present invention relates to the field of psychological counseling, and in particular to an intelligent psychological counseling system based on a multimodal large model and cognitive behavioral therapy. Background Art

[0002] Currently, there are few resources for psychological problem counseling and treatment, and these resources are usually only open to specific groups such as students. At the same time, many people are unable to realize that they have psychological problems and thus cannot seek professional counseling help in a timely manner. In addition, traditional psychological counseling services have the following problems: the resources of professional psychological counselors are scarce and cannot meet the needs of large-scale mental health services; the cost of psychological counseling is high and exceeds the affordability of most people; the service time and region are limited and it is difficult to provide psychological counseling services in a timely manner; existing intelligent psychological counseling systems have limited natural language understanding ability, the response is too mechanical, it is difficult to achieve a highly anthropomorphic service, and the suggestions provided are too one-sided and cannot achieve the expected treatment effect. In view of the above deficiencies in existing psychological counseling and treatment services, it is necessary to provide psychological counseling services with high-efficiency, accurate natural language understanding ability, highly anthropomorphic response, all-weather and low cost. Summary of the Invention

[0003] The purpose of the present invention is to provide an intelligent psychological counseling system based on a multimodal large model and cognitive behavioral therapy. The multimodal large model is used to identify the speech from the user terminal to obtain text information and emotional state information; the large language model is used to analyze the text information and conduct a conversation with the user terminal, and the psychological state information of the user terminal is extracted from the conversation; and the cognitive behavioral therapy strategy matching the user terminal is determined according to the emotional state information. Then, the multimodal large model is used to generate a voice response to the user terminal according to the matching cognitive behavioral therapy strategy; combining the multimodal large model and the large language model to achieve accurate and efficient natural language understanding of the user terminal, provide a highly anthropomorphic voice response, reduce the dependence on professional psychological counselors, be able to break through the time and region limitations of psychological counseling services, meet the needs of large-scale counseling services all-weather and at low cost, improve the human-computer interaction experience in the counseling process, and provide high-quality, highly practical and professionally targeted psychological counseling and treatment services.

[0004] The present invention is realized through the following technical solutions:

[0005] An intelligent psychological counseling system based on a multimodal large model and cognitive behavioral therapy, comprising:

[0006] A multimodal input module, which uses a multimodal large model to identify the speech from the user terminal to obtain text information and emotional state information;

[0007] A dialogue management module, which uses a large language model to analyze the text information and thus conduct a conversation with the user terminal;

[0008] A psychological element analysis module that uses a large language model to analyze the conversation and extract the psychological state information of the client;

[0009] A cognitive behavioral therapy determination module that determines a cognitive behavioral therapy strategy matching the client according to the emotional state information and the psychological state information;

[0010] A multimodal output model that uses a multimodal large model to generate a voice response to the client according to the matching cognitive behavioral therapy strategy.

[0011] Optionally, the multimodal input module includes:

[0012] A speech recognition and transcription unit that, after identifying the language type corresponding to the speech from the client, uses a speech transcription large model to identify the text information corresponding to the speech; wherein, the speech transcription large model has an end-to-end deep neural network architecture;

[0013] A speech emotion feature extraction unit that performs acoustic feature analysis on the speech to obtain the multi-dimensional speech acoustic features of the client, thereby determining the emotional state features of the client; wherein, the multi-dimensional speech acoustic features include speech frequency, speech volume, and speech speed.

[0014] Optionally, the multimodal input module further includes:

[0015] A facial expression analysis unit that visually recognizes the facial key point movements during the speech of the client and uses a deep learning model to analyze the facial key point movements to obtain the facial micro-expression features of the client, thereby determining the emotional state features of the client.

[0016] Optionally, the multimodal input module further includes:

[0017] A facial image processing module for collecting facial images during the speech of the client to obtain a plurality of facial image data;

[0018] Performing image region extraction on the facial key points in every two adjacent facial image data to obtain an image region corresponding to the facial key points;

[0019] Comparing the similarity of the image regions corresponding to the corresponding facial key points in every two adjacent facial image data to obtain the similarity value of the image regions corresponding to the facial key points included in every two adjacent facial image data;

[0020] Comparing the similarity value of the image regions corresponding to the facial key points included in every two adjacent facial image data with a preset similarity threshold;

[0021] Select two adjacent facial image data with similarity values lower than a preset similarity threshold as the target image data group;

[0022] Extract the image data collected at the first moment after the target image data group;

[0023] Retrieve the image data collected at the first moment and compare its similarity with two pieces of image data in the target image data to obtain the similarity values between the image data collected at the first moment and the two pieces of image data in the target image data respectively;

[0024] Take the similarity value between the image data collected at the first moment and the facial image data with an earlier collection time in the target image data as the first reference similarity value;

[0025] Take the similarity value between the image data collected at the first moment and the facial image data with a later collection time in the target image data as the second reference similarity value;

[0026] Use the first reference similarity value, the second reference similarity value, and the similarity value corresponding to the target image data group to obtain a comprehensive similarity value;

[0027] Among them, the comprehensive similarity value is obtained through the following formula:

[0028] S = max(S1, S2) × (1 - f × S0)

[0029] Among them, S represents the comprehensive similarity value; S1 and S2 represent the first reference similarity value and the second reference similarity value respectively; S0 represents the similarity value corresponding to the target image data group; f represents an adjustment coefficient used to adjust the influence sensitivity of the similarity value on the comprehensive similarity value, and the adjustment coefficient is obtained through the following formula:

[0030]

[0031] Among them, f represents the adjustment coefficient, n represents the total number of target image data groups; S i represents the similarity value corresponding to the i-th target image data group; S i+1 represents the similarity value corresponding to the (i + 1)-th target image data group; S b represents the standard deviation of the similarity values corresponding to n target image data groups;

[0032] When the comprehensive similarity value exceeds the preset comprehensive similarity threshold, then eliminate the facial image data corresponding to the larger reference similarity value among the first reference similarity value and the second reference similarity value corresponding to the target image data.

[0033] Optionally, it further includes a memory module for storing the historical conversation data of the user terminal;

[0034] The dialogue management module uses a large language model to conduct a multi-dimensional structured evaluation of the text information to obtain a multi-dimensional evaluation form; according to the historical conversation data and the multi-dimensional evaluation form, it generates a dialogue response regarding the speech and controls the dialogue process.

[0035] Optionally, the dialogue management module generates a dialogue response regarding the speech and controls the dialogue process according to the historical conversation data and the multi-dimensional evaluation form, including:

[0036] According to the multi-dimensional evaluation form, determine the content of the dialogue response regarding the speech; according to the historical conversation data, adjust the words of the dialogue response content and intercept the dialogue sentences to generate a dialogue response regarding the speech and control the dialogue process; wherein, the dialogue process refers to the number of dialogue sentence interactions with the user terminal.

[0037] Optionally, the psychological factor analysis module uses a large language model to analyze the dialogue and extract the psychological state information of the user terminal, including:

[0038] Use the embeding model to convert the dialogue into a vector in the semantic space;

[0039] Use a large language model to perform semantic detection based on the vector to obtain semantic keywords;

[0040] According to the semantic keywords, determine the factors related to the psychological state of the user terminal; wherein, the factors related to the psychological state include the type of the psychological state of the user terminal.

[0041] According to the factors related to the psychological state, determine the psychological state information of the user terminal.

[0042] Optionally, the cognitive behavioral therapy determination module determines the cognitive behavioral therapy strategy matching the user terminal according to the emotional state information and the psychological state information, including:

[0043] Use a large language model to analyze the emotional state information, the psychological state information, and the global historical conversation of the user terminal to generate several alternative cognitive behavioral therapy strategies;

[0044] Use a large language model to perform a simulation score on all alternative cognitive behavioral therapy strategies to obtain the treatment effect score results of all alternative cognitive behavioral therapy strategies;

[0045] Use a genetic algorithm to analyze the treatment effect score results of each alternative cognitive behavioral therapy strategy and determine the cognitive behavioral therapy strategy matching the user terminal from them.

[0046] Optionally, the multimodal output model includes:

[0047] A corpus extraction unit that uses a multimodal large model to extract corpus data applicable to cognitive behavioral therapy for the client from a corpus database according to the matched cognitive behavioral therapy strategy;

[0048] A statement generation unit that uses a multimodal large model to organize the semantic text of the corpus data and generate statement data for cognitive behavioral therapy for the client;

[0049] A speech synthesis unit that uses a multimodal large model to perform speech conversion on the statement data and generate a speech response for cognitive behavioral therapy for the client.

[0050] Optionally, the multimodal output model further includes:

[0051] A voice adjustment unit that uses a multimodal large model to analyze the facial expression feedback during the client's reception of the speech response; and adjusts the tone and / or intonation of the speech response according to the facial expression feedback.

[0052] Compared with the prior art, the present invention has the following beneficial effects:

[0053] An intelligent psychological counseling system based on a multimodal large model and cognitive behavioral therapy provided by the present application uses a multimodal large model to identify the speech from the client to obtain text information and emotional state information; uses a large language model to analyze the text information and have a conversation with the client, and extracts the psychological state information of the client from the conversation; and also determines a cognitive behavioral therapy strategy matched with the client according to the emotional state information, so as to use the multimodal large model to generate a speech response for the client according to the matched cognitive behavioral therapy strategy; combines the multimodal large model and the large language model to achieve accurate, efficient and natural language understanding of the client, provides a highly anthropomorphic speech response, reduces the dependence on professional psychological counselors, can break through the time and geographical limitations of psychological counseling services, meet the large-scale counseling service needs all-weather and at low cost, improve the human-computer interaction experience in the counseling process, and provide high-quality, highly practical and professionally targeted psychological counseling and treatment services. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art. Obviously, the following drawings are only some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained according to these drawings. Among them:

[0055] Figure 1 The structural schematic diagram of an intelligent psychological counseling system based on a multimodal large model and cognitive behavioral therapy provided by the present invention.

[0056] Figure 2 The structural schematic diagram of the multimodal input module in the present invention.

[0057] Figure 3 The structural schematic diagram of the multimodal output module in the present invention.

[0058] Figure 4 The working process schematic diagram of the psychological element analysis module in the present invention. Detailed implementation manners

[0059] To make the above objects, features, and advantages of the present application more obvious and understandable, the following will describe the detailed implementation manners of the present application in conjunction with the accompanying drawings. It can be understood that the specific embodiments described herein are only used to explain the present application, rather than limiting the present application. Additionally, it should be noted that for the sake of convenience of description, only parts related to the present application are shown in the drawings, rather than all the structures. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts belong to the scope of protection of the present application.

[0060] The terms "include" and "have" in the present application and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, system, product, or device that includes a series of steps or units is not limited to the listed steps or units, but optionally further includes steps or units not listed, or optionally further includes other steps or units inherent to these processes, systems, products, or devices.

[0061] Referring to

[0062] Please refer to Figure 1 As shown, an intelligent psychological counseling system based on a multimodal large model and cognitive behavioral therapy provided by an embodiment of the present application. The intelligent psychological counseling system based on a multimodal large model and cognitive behavioral therapy includes:

[0063] A multimodal input module that uses a multimodal large model to identify the speech from the user side to obtain text information and emotional state information;

[0064] The dialogue management module analyzes text information using a large language model to conduct a dialogue with the user side;

[0065] The psychological element analysis module analyzes the dialogue using a large language model to extract the psychological state information of the user side;

[0066] The cognitive behavioral therapy determination module determines the cognitive behavioral therapy strategy matching the user side based on the emotional state information and the psychological state information;

[0067] The multimodal output model uses a multimodal large model to generate a voice response to the user side according to the matching cognitive behavioral therapy strategy.

[0068] The beneficial effects of the above embodiments. This intelligent psychological counseling system based on a multimodal large model and cognitive behavioral therapy uses a multimodal large model to identify the voice from the user side to obtain text information and emotional state information; uses a large language model to analyze the text information and conduct a dialogue with the user side, and extracts the psychological state information of the user side from the dialogue; also determines the cognitive behavioral therapy strategy matching the user side according to the emotional state information, so as to use the multimodal large model to generate a voice response to the user side according to the matching cognitive behavioral therapy strategy; combines the multimodal large model and the large language model to achieve accurate, efficient and natural language understanding of the user side, provides a highly anthropomorphic voice response, reduces the dependence on professional psychological counselors, can break through the time and geographical limitations of psychological counseling services, meet the large-scale counseling service needs all-weather and at low cost, improve the human-computer interaction experience in the counseling process, and provide high-quality, highly practical and professionally targeted psychological counseling and treatment services.

[0069] In another embodiment, the multimodal input module includes:

[0070] The voice recognition and transcription unit, after identifying the language type corresponding to the voice from the user side, uses a voice transcription large model to identify the text information corresponding to the voice; wherein, the voice transcription large model has an end-to-end deep neural network architecture;

[0071] The voice emotion feature extraction unit conducts acoustic feature analysis on the voice to obtain the multi-dimensional voice acoustic features of the user side, so as to determine the emotional state features of the user side; wherein, the multi-dimensional voice acoustic features include voice frequency, voice volume and voice speed.

[0072] Beneficial effects of the above embodiments: The speech recognition and transcription unit adopts an end-to-end deep neural network architecture, supporting the recognition of multiple languages and dialects. For example, usually, the speech recognition and transcription unit selects Chinese or English as the default recognition language; when the user end does not use Chinese or English for voice interaction, the user end can select and change the language type used for current voice interaction by itself, and the speech recognition and transcription unit can match to a suitable language database according to the actually selected language type of the user end, so as to meet the language needs of different user ends. Using language databases corresponding to different language types to meet the needs of different language scenarios of the end-to-end deep neural network architecture can effectively reduce the speech recognition error rate, and the speech recognition error rate is lower than 3%. In addition, the speech recognition and transcription unit also has noise suppression and speech separation functions. After receiving the voice from the user end, it performs background noise filtering on the voice, and then uses the deep neural network to separate the voice signals corresponding to different channels in the voice, so as to separate the real speaker's voice of the user end, and can also ensure a high speech recognition accuracy in a complex environment.

[0073] Considering that there are significant differences in the pitch (voice frequency), volume, and speech rate of a user speaking in different emotional states. Generally speaking, when the user is in a more excited or frightened emotional state, such as excitement or panic, the pitch, volume, and speech rate of the user's speech are all higher; when the user's emotion is relatively calm, the pitch, volume, and speech rate of the user's speech are all lower. By performing multi-dimensional acoustic feature analysis on the speech, the speech frequency, speech volume, and speech rate of the user end are extracted, and then the multi-modal large model is used to analyze the speech frequency, speech volume, and speech rate to identify various types of emotional state characteristics of the user end, such as anger, sadness, anxiety, excitement, panic, etc.

[0074] In another embodiment, the multi-modal input module further includes:

[0075] A facial expression analysis unit that visually recognizes the facial key point movements during the process of the user end sending voice, and uses a deep learning model to analyze the facial key point movements to obtain the facial micro-expression characteristics of the user end, so as to determine the emotional state characteristics of the user end.

[0076] Beneficial effects of the above embodiments: The emotional state of the user terminal is not only reflected in the voice acoustic features emitted by the user, but also in the facial expressions of the user during the voice emission process. Merely analyzing the voice acoustic features emitted by the user cannot comprehensively identify the emotional state of the user. At this time, it is necessary to synchronously analyze the facial expressions of the user in order to maximize the identification of all emotional states of the user during the voice process and effectively avoid misidentification or omission of emotions. The facial expression analysis unit collects facial dynamic images during the voice emission process of the user terminal and performs facial key point action recognition on the above facial dynamic images. The facial key point action recognition may include, but is not limited to, recognizing the actions of facial features such as the mouth, nose, eyes, and eyebrows. For example, the actions of 68 key points on the face can be tracked, and a deep learning model is used to integratively analyze all facial key point actions to obtain the user's facial micro-expression, thereby further determining the emotional state characteristics of the user terminal, which is an effective supplementary recognition for determining the emotional state characteristics through voice acoustic analysis of the user terminal.

[0077] In another embodiment, the multi-modal input module further includes:

[0078] A facial image processing module for collecting facial images during the voice emission process of the user terminal to obtain a plurality of facial image data;

[0079] Extract the image regions corresponding to the facial key points from every two adjacent facial image data to obtain the image regions corresponding to the facial key points;

[0080] Compare the similarity of the image regions corresponding to the corresponding facial key points in every two adjacent facial image data to obtain the similarity values of the image regions corresponding to the facial key points included in every two adjacent facial image data;

[0081] Compare the similarity values of the image regions corresponding to the facial key points included in every two adjacent facial image data with a preset similarity threshold;

[0082] Select two adjacent facial image data with similarity values lower than the preset similarity threshold as the target image data group;

[0083] Extract the image data collected at the first moment after the target image data group;

[0084] Retrieve the similarity comparison between the image data collected at the first moment and the two image data in the target image data to obtain the similarity values of the image data collected at the first moment and the two image data in the target image data respectively;

[0085] Use the similarity value between the image data collected at the first moment and the facial image data with an earlier collection time in the target image data as the first reference similarity value;

[0086] Use the similarity values between the image data collected at the first moment and the facial image data with a later collection moment in the target image data as the second reference similarity values;

[0087] Use the first reference similarity value, the second reference similarity value, and combine with the similarity values corresponding to the target image data group to obtain a comprehensive similarity value;

[0088] Among them, the comprehensive similarity value is obtained through the following formula:

[0089] S = max(S1, S2) × (1 - f × S0)

[0090] Among them, S represents the comprehensive similarity value; S1 and S2 respectively represent the first reference similarity value and the second reference similarity value; S0 represents the similarity value corresponding to the target image data group; f represents an adjustment coefficient, which is used to adjust the influence sensitivity of the similarity value on the comprehensive similarity value, and the adjustment coefficient is obtained through the following formula:

[0091]

[0092] Among them, f represents the adjustment coefficient, n represents the total number of target image data groups; S i represents the similarity value corresponding to the i-th target image data group; S i+1 represents the similarity value corresponding to the (i + 1)-th target image data group; S b represents the standard deviation of the similarity values corresponding to n target image data groups;

[0093] When the comprehensive similarity value exceeds the preset comprehensive similarity threshold, then eliminate the facial image data corresponding to the larger reference similarity value among the first reference similarity value and the second reference similarity value corresponding to the target image data.

[0094] The beneficial effects of the above embodiments are as follows. By calculating the similarity values of adjacent facial image data and screening out the image data groups below the preset threshold, facial images that may have abnormalities (such as sudden expression changes, unnatural movements, etc.) can be effectively identified, avoiding the interference of such abnormal data on subsequent processing (such as expression recognition, sentiment analysis, etc.). Using the comprehensive similarity value to further evaluate the target image data group and adjusting the influence sensitivity of the similarity value by combining the adjustment coefficient makes the calculation of the comprehensive similarity value more scientific and reasonable, reducing the possibility of misjudgment. The introduction of the adjustment coefficient enables the system to adaptively adjust the influence sensitivity of the similarity value on the comprehensive similarity value according to the standard deviation of the similarity values of the target image data group. This helps the system maintain stable performance in different scenarios (such as light changes, expression changes, etc.). When the comprehensive similarity value exceeds the preset threshold, the system can automatically eliminate abnormal facial image data, thereby enhancing the robustness of the system and ensuring the accuracy and reliability of subsequent processing. By eliminating abnormal facial image data, the system can reduce misjudgments and interference caused by abnormal data, thus improving the user experience. For example, during voice interaction, the accurate recognition of the user's expression helps the system better understand the user's intention and provide more considerate services. Accurate facial image processing can reduce the system processing time and improve the interaction efficiency. Users do not need to wait for the system to process a large amount of abnormal data, thus obtaining a smoother interaction experience. By screening and eliminating abnormal facial image data, the system can reduce the amount of data to be processed, thereby reducing the computational complexity and improving the system performance. The system can allocate more computing resources to key tasks (such as expression recognition, sentiment analysis, etc.), improve the resource utilization efficiency, and further enhance the system performance.

[0095] In another embodiment, the intelligent psychological counseling system further includes a memory module for storing the historical conversation data of the user terminal;

[0096] The dialogue management module uses a large language model to conduct multi-dimensional structured evaluation on the text information to obtain a multi-dimensional evaluation form; according to the historical conversation data and the multi-dimensional evaluation form, it generates a dialogue reply regarding the voice and controls the dialogue process.

[0097] The beneficial effects of the above embodiments are as follows. Actual psychological counseling needs to combine and store the historical conversation data of the user terminal. By identifying the historical conversation data, the emotional state of the user terminal during the historical psychological counseling service process can be determined, which is convenient for comparing with the current emotional state of the user terminal, and then pertinently determining the psychological problems of the user terminal and providing appropriate psychological counseling suggestions. Setting a memory module with a data storage function in the intelligent psychological counseling system to store the historical conversation data of the user terminal provides a reliable data comparison basis for subsequent psychological counseling.

[0098] The dialogue management module is responsible for tracking and converting the dialogue state during the psychological counseling process. After converting the voice dialogue content from the user side into text information through the multi-modal large model, it uses the large language model to ask questions to the user side, generate responses to the user side, and control the dialogue process to ensure the coherence and personalization of the psychological counseling process. At the same time, it also has the function of adjusting the dialogue strategy in real time. Specifically, the dialogue management module can use reinforcement learning algorithms to optimize the dialogue strategy, has the ability to understand the context, and can control the dialogue process naturally and smoothly. The dialogue management module generates dialogue responses about voice and controls the dialogue process according to historical dialogue data and multi-dimensional evaluation tables, can make accurate and timely voice responses according to the real-time dialogue voice of the user side, ensure that the content of the voice response matches the real-time dialogue voice of the user side in text logic, and can also adjust the theme content and speed of the voice interaction with the user side to effectively control the psychological counseling dialogue process. Generally speaking, the dialogue management module has the following functional effects: First, establish a good psychological treatment relationship, establish a trust relationship through techniques such as empathy expression and active listening, and automatically adjust the tone and content of the voice response according to the expression method and emotional state of the user side; Second, be able to accurately identify problems, use structured evaluation tools to accurately identify the core problems of the user side, and use multi-dimensional evaluation tables in combination with historical dialogue data to form a comprehensive psychological problem diagnosis report; Third, be able to conduct cognitive intervention, adopt the ABC theory framework to systematically carry out cognitive restructuring, track the cognitive change process of the user side, and provide reinforcement and consolidation exercises in a timely manner; Fourth, be able to conduct behavior guidance, formulate a phased behavior change plan, and conduct tracking feedback, and dynamically adjust the plan difficulty according to the implementation situation of the user side.

[0099] In another embodiment, the dialogue management module generates dialogue responses about voice and controls the dialogue process according to historical dialogue data and multi-dimensional evaluation tables, including:

[0100] Determine the content of the dialogue response about voice according to the multi-dimensional evaluation table; adjust the words and intercept the dialogue sentences of the dialogue response content according to the historical dialogue data to generate the dialogue response about voice and control the dialogue process; where the dialogue process refers to the number of dialogue sentence interactions with the user side.

[0101] The beneficial effects of the above embodiments are that the multi-dimensional evaluation table multi-dimensionally represents the emotional state of the user side, comprehensively represents the emotional state of the user side, and provides a reference for automatically adjusting the content and tone of the voice response to the user side. Also, according to the historical dialogue data, adjust the words of the dialogue response content and intercept the dialogue sentences, and adjust the number of dialogue sentence interactions with the user side to ensure that the user side receives and understands the voice response sent by the dialogue management module within a limited time during the voice interaction with the user side.

[0102] In another embodiment, the psychological factor analysis module uses a large language model to analyze the conversation and extract the psychological state information of the user side, including:

[0103] Using an embedding model to convert the conversation into vectors in the semantic space;

[0104] Using a large language model to perform semantic detection based on the vectors to obtain semantic keywords;

[0105] According to the semantic keywords, determine the factors related to the psychological state of the user side; among them, the factors related to the psychological state include the type of the psychological state of the user side;

[0106] According to the factors related to the psychological state, determine the psychological state information of the user side.

[0107] The beneficial effects of the above embodiments are that the psychological factor analysis module may include but is not limited to an irrational belief recognition unit, a cognitive restructuring guidance unit, and a personalized behavior suggestion generation unit; among them, the irrational belief recognition unit can accurately identify at least ten common irrational belief patterns such as over-conceptualization and black-and-white thinking from the user's expressions and establish a complete classification system of cognitive biases; the cognitive restructuring guidance unit can use the Socratic questioning method to help users discover and question irrational beliefs and dynamically adjust the difficulty of questions according to the user's cognitive level; the personalized behavior suggestion generation unit provides feasible suggestions that conform to the SMART principle based on the specific situation of the user, and will consider factors such as the user's living environment and social support.

[0108] The psychological factor analysis module uses a large language model to extract factors closely related to the psychological state of the user side during the conversation, estimates the psychological state of the user side by analyzing these factors, and issues early warnings for possible critical situations. Specifically, the conversation is converted into vectors in the semantic space through an embedding model, and the semantic retrieval function of the large language model is used to retrieve texts closely related to the keywords of the CL90 scale factors of psychological measurement; the accuracy rate of each piece of conversation content retrieved semantically (denoted as a i ) and the score of the large language model for the retrieved conversation content (denoted as b i ) are multiplied to obtain a conversation segment, and the sum of all retrieved conversation segments is calculated. Finally, it is divided by the total length of the retrieved conversation (denoted as L) to obtain the final score P. The above score P is the basis for early warning of critical situations. The above calculation process is as follows: For example, the keywords corresponding to "depression" in the CL90 scale factors of psychological measurement include: melancholy, despair, loss of interest, mental retardation, sleep disorder, fatigue, decision-making difficulty, social withdrawal, etc. Processing the conversation segments retrieved by the above keywords can obtain the score for "depression".

[0109] In addition, the Pearson correlation coefficient between the factor scores of each psychological measurement CL90 scale in multiple periods and the intensity of the main events encountered in the corresponding periods can be calculated to determine the main stimulating factors of the psychological measurement CL90 scale factors, providing important information for the strategies of cognitive behavioral therapy. For the scenario of judging whether the user is depressed, the above process can be implemented through the following formula:

[0110]

[0111] In the above formula, P i is the depression factor score in the i-th period, is the average depression factor score in all periods, X i is the intensity value of the work-related events in the i-th period, is the average intensity value of the work-related events in all periods, and r is the Pearson correlation coefficient; the larger the coefficient r, the greater the correlation between the user's depression and work events.

[0112] In another embodiment, the cognitive behavioral therapy determination module determines the cognitive behavioral therapy strategy matching the user terminal according to the emotional state information and the psychological state information, including:

[0113] Analyze the emotional state information, psychological state information, and the global historical conversation of the user terminal using a large language model to generate several alternative cognitive behavioral therapy strategies;

[0114] Use the large language model to simulate and score all alternative cognitive behavioral therapy strategies to obtain the treatment effect score results of all alternative cognitive behavioral therapy strategies;

[0115] Use a genetic algorithm to analyze the treatment effect score results of each alternative cognitive behavioral therapy strategy to determine the cognitive behavioral therapy strategy matching the user terminal.

[0116] Beneficial effects of the above embodiments. The cognitive behavioral therapy determination module uses a large language model to provide personalized professional services that comply with the ethical norms of psychological counseling. By analyzing emotional state information, psychological state information, and the global historical conversation on the user side, it gives treatment strategies and key points from the perspective of cognitive behavioral therapy, generates several alternative cognitive behavioral therapy strategies, and provides the above information to the large language model of the dialogue management module. The large language model used by the cognitive behavioral therapy determination module is fine-tuned with classic cases of cognitive behavioral therapy and a knowledge base of cognitive behavioral therapy, and can accurately master the theory and application methods of cognitive behavioral therapy. Then, the large language model is used to simulate and score all alternative cognitive behavioral therapy strategies to obtain the treatment effect scoring results of each alternative cognitive behavioral therapy strategy. For example, the large language model retrieves all the conversation records of the user, simulates the user's reactions and final psychological state during the implementation of each alternative cognitive behavioral therapy strategy, and gives the treatment effect scoring results. Finally, the genetic algorithm is used to analyze the treatment effect scoring results of each alternative cognitive behavioral therapy strategy to determine the cognitive behavioral therapy strategy that matches the user side.

[0117] In another embodiment, the multimodal output model includes:

[0118] A corpus extraction unit that uses a multimodal large model to extract corpus data applicable to cognitive behavioral therapy for the user side from a corpus database;

[0119] A sentence generation unit that uses a multimodal large model to organize the semantic text of the corpus data and generate sentence data for cognitive behavioral therapy for the user side;

[0120] A speech synthesis unit that uses a multimodal large model to perform speech conversion on the sentence data and generate a voice response for cognitive behavioral therapy for the user side.

[0121] Beneficial effects of the above embodiments. The multimodal output model integrates speech synthesis and virtual image generation technologies to provide natural and emotional voice responses, and is accompanied by appropriate virtual image expression feedback; it adopts the latest emotion computing technology and can automatically adjust the tone and intonation according to the conversation content. The multimodal output model includes a corpus extraction unit, a sentence generation unit, and a speech synthesis unit; it uses a multimodal large model to extract corpus data applicable to cognitive behavioral therapy for the user side from a corpus database, uses a multimodal large model to organize the semantic text of the corpus data and generate sentence data for cognitive behavioral therapy for the user side, and uses a multimodal large model to perform speech conversion on the sentence data and generate a voice response for cognitive behavioral therapy for the user side, which can provide users with a personalized psychological counseling voice response feedback and improve the experiential nature of psychological counseling services.

[0122] In another embodiment, the multimodal output model further includes:

[0123] A voice adjustment unit that uses the multimodal large model to analyze the facial expression feedback during the process of the user receiving the voice response; adjusts the tone and / or intonation of the voice response according to the facial expression feedback.

[0124] The beneficial effects of the above embodiment are as follows. Considering that there are significant differences among users who need psychological counseling services, and different users have different requirements for the tone and intonation of the voice response feedback during the psychological counseling process, the multimodal large model is used to analyze the facial expression feedback during the process of the user receiving the voice response; the tone and / or intonation of the voice response is adjusted according to the facial expression feedback, so as to improve the real-time performance and effectiveness of the adjustment of the voice response feedback during the psychological counseling service process.

[0125] Generally speaking, this intelligent psychological counseling system based on the multimodal large model and cognitive behavioral therapy uses the multimodal large model to identify the voice from the user side, obtaining text information and emotional state information; uses the large language model to analyze the text information and conduct a conversation with the user side, and extracts the psychological state information of the user side from the conversation; also determines the cognitive behavioral therapy strategy matching the user side according to the emotional state information, so as to use the multimodal large model to generate a voice response to the user side according to the matching cognitive behavioral therapy strategy; combines the multimodal large model and the large language model to achieve accurate, efficient and natural language understanding of the user side, provides a highly anthropomorphic voice response, reduces the dependence on professional psychological counselors, can break through the time and geographical limitations of psychological counseling services, meet the large-scale counseling service needs all-weather and at low cost, improve the human-computer interaction experience in the counseling process, and provide high-quality, highly practical and professionally targeted psychological counseling and treatment services.

[0126] The above is only a specific implementation manner of the present invention, and any improvement made on the premise of the present invention's concept is regarded as the protection scope of the present invention.

Claims

1. An intelligent psychological counseling system based on a multimodal large model and cognitive behavioral therapy, characterized in that Including: A multi-modal input module that uses a multi-modal large model to recognize the speech from the user side and obtain text information and emotional state information; A dialogue management module that uses a large language model to analyze the text information and thus conduct a dialogue with the user side; A psychological element analysis module that uses a large language model to analyze the dialogue and extract the psychological state information of the user side; A cognitive behavioral therapy determination module that determines a cognitive behavioral therapy strategy matching the user side according to the emotional state information and the psychological state information; A multi-modal output model that uses a multi-modal large model to generate a voice response to the user side according to the matching cognitive behavioral therapy strategy.

2. The intelligent psychological counseling system based on a multi-modal large model and cognitive behavioral therapy according to claim 1, characterized in that: The multi-modal input module includes: A speech recognition and transcription unit that, after identifying the language type corresponding to the speech from the user side, uses a speech transcription large model to recognize the text information corresponding to the speech; wherein, the speech transcription large model has an end-to-end deep neural network architecture; A speech emotion feature extraction unit that performs acoustic feature analysis on the speech to obtain the multi-dimensional speech acoustic features of the user side, thereby determining the emotional state features of the user side; wherein, the multi-dimensional speech acoustic features include speech frequency, speech volume, and speech speed.

3. The intelligent psychological counseling system based on a multi-modal large model and cognitive behavioral therapy according to claim 2, characterized in that: The multi-modal input module further includes: A facial expression analysis unit that visually recognizes the facial key point movements during the speech of the user side and uses a deep learning model to analyze the facial key point movements to obtain the facial micro-expression features of the user side, thereby determining the emotional state features of the user side.

4. The intelligent psychological counseling system based on a multi-modal large model and cognitive behavioral therapy according to claim 3, characterized in that: The multi-modal input module further includes: A facial image processing module for collecting facial images during the speech of the user side to obtain a plurality of facial image data; Extracting the image regions corresponding to the facial key points from every two adjacent facial image data to obtain the image regions corresponding to the facial key points; Comparing the similarity of the image regions corresponding to the corresponding facial key points in every two adjacent facial image data to obtain the similarity value of the image regions corresponding to the facial key points included in every two adjacent facial image data; Comparing the similarity value of the image regions corresponding to the facial key points included in every two adjacent facial image data with a preset similarity threshold; Selecting every two adjacent facial image data with similarity values lower than the preset similarity threshold as a target image data group; Extracting the image data collected at the first moment after the target image data group; Retrieving the image data collected at the first moment and comparing its similarity with the two image data in the target image data to obtain the similarity values of the image data collected at the first moment and the two image data in the target image data respectively; Take the similarity values between the image data collected at the first moment and the facial image data with an earlier collection moment in the target image data as the first reference similarity values; Take the similarity values between the image data collected at the first moment and the facial image data with a later collection moment in the target image data as the second reference similarity values; Use the first reference similarity value, the second reference similarity value, and the similarity value corresponding to the target image data group to obtain a comprehensive similarity value; Among them, the comprehensive similarity value is obtained through the following formula: S = max(S1, S2) × (1 - f × S0) Among them, S represents the comprehensive similarity value; S1 and S2 respectively represent the first reference similarity value and the second reference similarity value; S0 represents the similarity value corresponding to the target image data group; f represents an adjustment coefficient used to adjust the influence sensitivity of the similarity value on the comprehensive similarity value, and the adjustment coefficient is obtained through the following formula: where f represents the adjustment coefficient and n represents the total number of target image data groups; S i represents the similarity value corresponding to the i-th target image data group; S i+1 represents the similarity value corresponding to the (i + 1)-th target image data group; S b represents the standard deviation of the similarity values corresponding to n target image data groups; When the comprehensive similarity value exceeds the preset comprehensive similarity threshold, then eliminate the facial image data corresponding to the larger reference similarity value among the first reference similarity value and the second reference similarity value corresponding to the target image data.

5. An intelligent psychological counseling system based on a multi-modal large model and cognitive behavioral therapy according to claim 1, characterized in that: It further includes a memory module for storing the historical conversation data of the user terminal; The dialogue management module uses a large language model to conduct a multi-dimensional structured evaluation of the text information to obtain a multi-dimensional evaluation form; Generate a dialogue response regarding the voice and control the dialogue process based on the historical conversation data and the multi-dimensional evaluation form.

6. An intelligent psychological counseling system based on a multi-modal large model and cognitive behavioral therapy according to claim 5, characterized in that: The dialogue management module generates a dialogue response regarding the voice and controls the dialogue process based on the historical conversation data and the multi-dimensional evaluation form, including: Determine the content of the dialogue response regarding the voice according to the multi-dimensional evaluation form; adjust the words of the dialogue response content and intercept the dialogue sentences according to the historical conversation data to generate a dialogue response regarding the voice and control the dialogue process; where the dialogue process refers to the number of dialogue sentence interactions with the user terminal.

7. An intelligent psychological counseling system based on a multi-modal large model and cognitive behavioral therapy according to claim 1, characterized in that: The psychological element analysis module uses a large language model to analyze the dialogue and extract the psychological state information of the user terminal, including: Use the embeding model to convert the dialogue into a vector in the semantic space; Use a large language model to perform semantic detection based on the vector to obtain semantic keywords; Determine the psychological state related factors of the user terminal according to the semantic keywords; where the psychological state related factors include the type of the psychological state of the user terminal; Determine the psychological state information of the user terminal according to the psychological state related factors.

8. An intelligent psychological counseling system based on a multimodal large model and cognitive behavioral therapy according to claim 1, characterized in that: The cognitive behavioral therapy determination module determines a cognitive behavioral therapy strategy matching the client according to the emotional state information and the psychological state information, including: Analyzing the emotional state information, the psychological state information, and the global historical conversation of the client by using a large language model to generate several alternative cognitive behavioral therapy strategies; Using a large language model to simulate and score all alternative cognitive behavioral therapy strategies to obtain the treatment effect scoring results of all alternative cognitive behavioral therapy strategies; Analyzing the treatment effect scoring results of each alternative cognitive behavioral therapy strategy by using a genetic algorithm to determine a cognitive behavioral therapy strategy matching the client therefrom.

9. An intelligent psychological counseling system based on a multimodal large model and cognitive behavioral therapy according to claim 1, characterized in that: The multimodal output model includes: A corpus extraction unit that uses a multimodal large model to extract corpus data applicable to cognitive behavioral therapy for the client from a corpus database; a sentence generation unit that uses a multimodal large model to organize the semantic text of the corpus data to generate sentence data for cognitive behavioral therapy for the client; A speech synthesis unit that uses a multimodal large model to perform speech conversion on the sentence data to generate a speech response for cognitive behavioral therapy for the client.

10. An intelligent psychological counseling system based on a multimodal large model and cognitive behavioral therapy according to claim 9, characterized in that: The multimodal output model further includes: A speech adjustment unit that uses a multimodal large model to analyze the expression feedback during the client's reception of the speech response; adjusting the tone and / or intonation of the speech response according to the expression feedback.

Images