Multi-modal fusion psychological assessment report generation method, device and equipment
By using a multimodal fusion-based psychological assessment method, which combines facial expressions and speech tone features from video data with spoken text, a psychological assessment report is generated. This addresses the problem of insufficient comprehensiveness in existing technologies and improves the accuracy of the assessment.
Patent Information
- Application Number
- CN202411500527.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-10-25
- Publication Date
- 2024-11-22
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
Existing mental health assessment methods rely on user self-reported data, which is subjective and results in assessments that are not comprehensive or accurate enough.
By acquiring users' video data, extracting facial expression features and voice tone features, and combining them with spoken text, a multimodal big data model is used to analyze emotional and behavioral characteristics and generate a psychological assessment report.
It enables multi-dimensional psychological state analysis, improving the accuracy and comprehensiveness of psychological state assessment.
Smart Images

Figure CN119007993A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of mental health assessment, and in particular to a multi-modal fusion psychological assessment report generation method, device, equipment and storage medium. Background Art
[0002] Mental health refers to a continuous and positive mental state in which the subject can adapt well and fully develop his or her physical and mental potential. How to maintain people's mental health and actively intervene and guide those with psychological problems is of great importance to the entire family, society and company.
[0003] The assessment of mental health status involves the collection, analysis, synthesis and judgment of an individual's specific behavioral symptoms. It can be achieved through a variety of technical means such as "interviews", "questionnaires", and "projective tests". However, most of these assessment methods rely on the user's self-reported data and are subjective, which may lead to incomplete and inaccurate judgments on the user's mental state. Summary of the invention
[0004] The main purpose of the present invention is to solve the technical problem of how to achieve multi-dimensional psychological state analysis and improve the accuracy of psychological state assessment.
[0005] A first aspect of the present invention provides a method for generating a multimodal fusion psychological assessment report, the method comprising: Acquire video data of the user, wherein the video data is video data recorded when the user answers questions in the DSM-5 psychological assessment question bank, and the video data includes user images and audio data; Extracting facial expression features and voice intonation features of the user from the video data; Acquire the user's speech text, wherein the speech text is text extracted from the user's answers to questions in the DSM-5 psychological assessment question bank; According to the facial expression features, the voice intonation features, and the speech text, the emotional behavior features of the user are obtained through a multimodal large model; According to the emotional behavior characteristics, a corresponding psychological assessment report is generated for the user.
[0006] Optionally, in a first implementation of the first aspect of the present invention, obtaining the emotional behavior characteristics of the user through a multimodal large model according to the facial expression characteristics, the voice intonation characteristics, and the speech text includes: According to the facial expression features and the voice intonation features, the emotional state of the user is identified through a multimodal large model; The emotional state and the speech text are integrated to obtain the emotional behavior characteristics of the user; The historical question and answer records of the DSM-5 psychological assessment question bank corresponding to the user are obtained, and an empathy response is generated through an empathy model according to the emotional behavior characteristics and the historical question and answer records.
[0007] Optionally, in a second implementation of the first aspect of the present invention, before fusing the emotional state and the speech text to obtain the emotional behavior characteristics of the user, the method further includes: Acquire the historical emotional state and the current emotional state of the user, wherein the historical emotional state and the current emotional state are obtained by multimodal fusion based on the same facial expression features and the voice intonation features; Determining whether the historical emotional state is consistent with the current emotional state; If they are inconsistent, weighting the historical emotional state and the current emotional state to calculate the emotional state of the user; If they are consistent, the step of fusing the emotional state and the speech text to obtain the emotional behavior characteristics of the user is entered.
[0008] Optionally, in a third implementation manner of the first aspect of the present invention, weighting the historical emotional state and the current emotional state to calculate the emotional state of the user includes: Performing a weighted sum of the historical emotional state and the current emotional state with a time-attenuated weight to obtain a weight of the historical emotional state and the current emotional state; The corresponding emotional state with the highest weight between the historical emotional state and the current emotional state is selected as the emotional state of the user through a statistical analysis method.
[0009] Optionally, in a fourth implementation of the first aspect of the present invention, before obtaining the historical question and answer records of the DSM-5 psychological assessment question bank corresponding to the user and generating an empathy response through an empathy model according to the emotional behavior characteristics and the historical question and answer records, the method further includes: Acquire a training data set, the data set including high-quality manually collected emotional behavior feature data and historical question and answer record data, as well as synthesized emotional behavior feature data and historical question and answer record data; Based on the data set, an initial empathy model is trained using a deep learning framework and algorithm; The trained empathy model is fine-tuned using lightweight fine-tuning technology to provide appropriate empathic responses to the user for different psychological counseling scenarios.
[0010] Optionally, in a fifth implementation of the first aspect of the present invention, identifying the user's emotional state through a multimodal large model according to the facial expression features and the voice intonation features includes: According to the facial expression features and the voice and intonation features, the correlation between the facial expression features and the voice and intonation features is calculated through an attention mechanism, and corresponding attention weights are generated for the facial expression features and the voice and intonation features; According to the attention weight, the facial expression features and the voice and intonation features are weighted, and the weighted facial expression features and the voice and intonation features are fused through a cross-modal attention mechanism. Based on the fused facial expression features and the voice intonation features, the emotional state of the user is identified through a multimodal large model.
[0011] Optionally, in a sixth implementation of the first aspect of the present invention, generating a corresponding psychological assessment report for the user according to the emotional behavior characteristics includes: Generate multi-dimensional evaluation reference indicators based on the emotional behavior characteristics; Based on the evaluation reference indicators, a corresponding psychological evaluation report is generated for the user.
[0012] The second aspect of the present invention provides a multimodal fusion psychological assessment report generation device, comprising: a memory and at least one processor, the memory storing instructions, the memory and the at least one processor being interconnected via lines; the at least one processor calling the instructions in the memory so that the multimodal fusion psychological assessment report generation device executes the above-mentioned multimodal fusion psychological assessment report generation method.
[0013] A third aspect of the present invention provides a computer-readable storage medium, which stores instructions. When the computer-readable storage medium is run on a computer, it enables the computer to execute the above-mentioned multi-modal fusion psychological assessment report generation method.
[0014] In an embodiment of the present invention, the video data of the user is obtained, and the video data is the video data recorded when the user answers questions in the DSM-5 psychological assessment question bank, and the video data includes user images and audio data; the facial expression features and voice and intonation features of the user are extracted from the video data; the speech text of the user is obtained, and the speech text is the text extracted from the user's answers to the questions in the DSM-5 psychological assessment question bank; according to the facial expression features, the voice and intonation features, and the speech text, the emotional behavior features of the user are obtained through a multimodal large model; according to the emotional behavior features, a corresponding psychological assessment report is generated for the user. In the present invention, by fusing facial expression features, the voice and intonation features, and the speech text, the emotional behavior features of the user are multimodally identified, and a multi-dimensional psychological state analysis can be achieved, which more comprehensively reflects the user's mental health status, thereby improving the accuracy of psychological state assessment. BRIEF DESCRIPTION OF THE DRAWINGS
[0015] Figure 1 A schematic diagram of an embodiment of a method for generating a multimodal fusion psychological assessment report in an embodiment of the present invention; Figure 2 A schematic diagram of an embodiment of a multi-modal fusion psychological assessment report generating device in an embodiment of the present invention; Figure 3 The figure is a schematic diagram of an embodiment of a multimodal fusion psychological assessment report generation device in an embodiment of the present invention. DETAILED DESCRIPTION
[0016] The embodiments of the present invention provide a method, apparatus, device and storage medium for generating a multi-modal fusion psychological assessment report.
[0017] The embodiments disclosed in the present invention will be described in more detail below with reference to the accompanying drawings. Although certain embodiments disclosed in the present invention are shown in the accompanying drawings, it should be understood that the present invention can be implemented in various forms and should not be construed as being limited to the embodiments described herein, which are instead provided for a more thorough and complete understanding of the present invention. It should be understood that the drawings and embodiments disclosed in the present invention are only for exemplary purposes and are not intended to limit the scope of protection disclosed in the present invention.
[0018] In the description of the embodiments disclosed in the present invention, the term "including" and similar terms should be understood as open inclusion, that is, "including but not limited to". The term "based on" should be understood as "based at least in part on". The term "one embodiment" or "the embodiment" should be understood as "at least one embodiment". The terms "first", "second", etc. may refer to different or the same objects. Other explicit and implicit definitions may also be included below.
[0019] For ease of understanding, the specific process of the embodiment of the present invention is described below. Figure 1 , an embodiment of a method for generating a multimodal fusion psychological assessment report in an embodiment of the present invention includes: S100, obtaining video data of a user.
[0020] In this embodiment, a video of a user answering questions in a DSM-5 psychological assessment question bank is recorded in the form of human-computer interaction, wherein DSM-5 (Diagnostic and Statistical Manual of Mental Disorders, Fifth Edition) is a widely used psychological assessment tool for diagnosing various mental disorders. In practical applications, video data can be captured and parsed through a video interface API, which can provide technical support for obtaining video content when a user answers questions.
[0021] S200, extracting the user's facial expression features and voice intonation features from the video data.
[0022] In this embodiment, after obtaining the user's video data, the user's facial expression features and voice and intonation features are extracted from the video data. In order to achieve more accurate and efficient facial expression recognition, a visual language model (VLM) is used to complete this task. First, the Vision Transformer (ViT) is used to extract facial expression features. ViT learns the global features of the image by dividing the image into several small patches and processing these patches as input. Then, the image features and text are embedded in the same vector space through the Contrastive Language-Image Pretraining (CLIP) technology, so that the image features can be more accurately captured through language description. Finally, the image feature embedding and the corresponding text prompt words are input into the pre-trained visual language model to identify specific facial expressions. Voice and intonation features are also an important part of emotion recognition. Voice features can be divided into language features and acoustic features, where acoustic features include the speaker's tone, intonation, etc. By using the Audio-LanguageModel (ALM), the user's language features and acoustic features are extracted at the same time, and emotional analysis and processing are performed to identify the user's emotional state. In order to more accurately identify the emotional state, facial expression features and voice intonation features can be fused. Specifically, if the emotion recognition results of the image and voice are inconsistent, the historical emotional state and the current emotional state are weighted and summed with the time decay weight, and the emotional state with the highest weight (highest probability) is finally selected.
[0023] S300, obtaining the user's speech text.
[0024] In this embodiment, the speech text extracted from the user's answers to questions in the DSM-5 psychological assessment question bank can be extracted from the audio data, and the audio signal is converted into a digital signal through automatic speech recognition (Automatic Speech Recognition, ASR) technology, and analyzed using machine learning technology, so as to convert the spoken words into written text. Specifically, the audio data needs to be preprocessed first, such as converting the audio data into a standard format and adjusting the sampling rate and the number of channels, and then the preprocessed audio data is transcribed using an ASR engine. For example, a Whisper model can be used to efficiently extract text. These ASR engines usually output text with punctuation to facilitate subsequent editing and analysis. The transcribed text can then be further edited and optimized to improve readability and accuracy. For example, tools can be used to add appropriate punctuation and paragraph divisions.
[0025] S400, based on facial expression features, voice intonation features, and speech text, the user's emotional behavior features are obtained through a multimodal large model.
[0026] In this embodiment, the natural language output capabilities of the visual multimodal and speech multimodal models are used to input the user's real-time video data and summarize it into a descriptive text of the user's emotions, thereby extracting the user's facial expressions and voice intonation emotions in real time, and then combining the user's speech text to integrate and analyze the user's real emotions and psychological state on the large language model. By integrating the recognition results of multiple modalities, a rich information dimension is provided and the recognition accuracy is greatly improved. Multimodal real-time recognition of 8 major categories of emotional behavior characteristics: emotions, feelings, attitudes, psychosis, clothing, speech, eye contact, and physical activity status. Emotions include: afraid, calm, depressed, anxious, angry, happy, disgusted, stressed, surprised, and excited; Feelings include: rich emotions, weak emotions, lack of emotions, and inappropriate emotions; Attitudes include: cooperative, uncooperative, defensive, and aggressive; stiff; Negative symptoms; Clothing: neat, untidy, and weird; Speech includes: normal speech, off-topic speech, excessive speech, and poor speech; Eye contact includes: normal eye contact and avoidant eye contact; Body activity states include: normal body activity state, restless body activity state, slow body activity state, and twitchy body activity state.
[0027] S500, generating a corresponding psychological assessment report for the user according to the emotional behavior characteristics.
[0028] In this embodiment, based on the user's emotional behavior characteristics obtained through the multimodal large model, a corresponding psychological assessment report is generated for the user to help the user better understand his or her own psychological condition and provide corresponding support and suggestions.
[0029] In an optional implementation of the first aspect of the present invention, based on facial expression features, voice intonation features, and speech text, the user's emotional behavior features are obtained through a multimodal large model, including: Based on facial expression features and voice intonation features, the user's emotional state is identified through a multimodal large model; the emotional state and speech text are integrated to obtain the user's emotional behavior characteristics; the user's corresponding DSM-5 psychological assessment question bank's historical question and answer records are obtained, and based on the emotional behavior characteristics and historical question and answer records, an empathy response is generated through an empathy model.
[0030] In this embodiment, a large multimodal model of vision and speech is introduced to recognize the user's facial emotions and voice emotions in real time, and obtain highly accurate user emotions (such as happiness, sadness, anxiety, etc.). The large multimodal model of vision and speech can capture and analyze the user's emotional state in real time by integrating facial expression information from video probes, eye gaze angle analysis technology, and speech emotion analysis technology. These usually use deep learning models, such as Transformer structures, to model speech emotion signals and adaptively focus on valuable fine-grained emotion information in speech signals. At the same time, the speech text is also integrated to obtain the user's emotional behavior characteristics, and then the user's emotional behavior characteristics and historical question and answer records are passed into the empathy large language model to generate empathy responses.
[0031] In an optional implementation of the first aspect of the present invention, before fusing the emotional state and the speech text to obtain the emotional behavior characteristics of the user, the method further includes: The user's historical emotional state and current emotional state are obtained by multimodal fusion based on the same facial expression features and voice intonation features; whether the historical emotional state is consistent with the current emotional state is determined; if not, the historical emotional state and the current emotional state are weighted to calculate the user's emotional state; if consistent, the emotional state and the speech text are fused to obtain the user's emotional behavior characteristics.
[0032] In this embodiment, the emotional state (historical emotional state and current emotional state) obtained by fusing the same facial expression features and voice intonation features at different time nodes is obtained, and it is determined whether the historical emotional state is consistent with the current emotional state. If not, the historical emotional state and the current emotional state are weighted to obtain the final emotional state of the user. If they are consistent, the step of fusing the emotional state and the speech text to obtain the emotional behavior characteristics of the user is entered.
[0033] In an optional implementation of the first aspect of the present invention, weighting the historical emotional state and the current emotional state to calculate the emotional state of the user includes: The historical emotional state and the current emotional state are weighted and added with the time decay weight to obtain the weights of the historical emotional state and the current emotional state; and the corresponding emotional state with the highest weight between the historical emotional state and the current emotional state is selected as the user's emotional state through a statistical analysis method.
[0034] In this embodiment, in the weighted calculation, different weights are assigned to data at different times due to different degrees of influence at different times. If the historical emotional state is inconsistent with the current emotional state, the historical emotional state and the current emotional state are weighted and summed with the time decay weight, that is, the weight is determined according to the time window and half-life of the data, and then the emotional state with the highest weight is selected as the final result through a statistical analysis method, thereby integrating the emotional signals of facial expressions and voice intonation to obtain accurate emotional evaluation results. This comprehensive analysis can correct the error of a single modality, making emotional evaluation more reliable.
[0035] The specific calculation method of time decay weight is usually based on the exponential decay function, which assumes that data will decay over time and the weight of each point will decrease over time. The specific calculation formula can be expressed as:
[0036] Where λ is the decay rate and t is the time variable.
[0037] In practical applications, two key parameters need to be determined: time window and half-life. The time window refers to the time period from which the data comes, while the half-life refers to the time required for the weight to decay to half. For example, if 252 days are selected as the time window and 63 days as the half-life, when the current day is t, the weight of the t-63rd day is 0.5, and the farther away from t, the smaller the weight.
[0038] In an optional implementation of the first aspect of the present invention, before obtaining the historical question and answer records of the DSM-5 psychological assessment question bank corresponding to the user, and generating an empathy response through an empathy model based on the emotional behavior characteristics and the historical question and answer records, the method further includes: Obtain a training data set, which includes high-quality manually collected emotional behavior feature data and historical question and answer record data, as well as synthetic emotional behavior feature data and historical question and answer record data; based on the data set, use the deep learning framework and algorithm to train the initial empathy model; use lightweight fine-tuning technology to fine-tune the trained empathy model to provide users with appropriate empathetic responses for different psychological counseling scenarios.
[0039] In this embodiment, the empathy model is a large psychological vertical model, which is trained using high-quality artificial data, high-quality synthetic data, and advanced optimization technology, and can provide users with appropriate empathic responses for different psychological counseling scenarios. The use of synthetic data in the training data set can be used to supplement the deficiencies of real data, improve data quality and quantity, and reduce the cost of data collection and processing. An important role of synthetic data in the training of large psychological vertical models is to improve the alignment ability of the model and the vertical application effect. By using synthetic data, ordinary optimization data can be replaced to improve the performance of the model. This method is particularly suitable for scenarios that require highly customized features, such as hoping that a large language model has a specific bias when answering certain questions. Since the quality of the acquired data is often uneven, it is necessary to manually collect high-quality data. After training, the empathy model is fine-tuned using lightweight fine-tuning technology to improve the performance of the model on specific tasks. For example, in the field of psychology, it may be necessary to fine-tune specific psychological tests or questionnaire data to ensure that the model can accurately predict psychological states or behaviors.
[0040] In an optional implementation of the first aspect of the present invention, identifying the user's emotional state through a multimodal large model based on facial expression features and voice intonation features includes: According to the facial expression features and speech intonation features, the correlation between the facial expression features and the speech intonation features is calculated through the attention mechanism, and the corresponding attention weights are generated for the facial expression features and the speech intonation features; according to the attention weights, the facial expression features and the speech intonation features are weighted, and the weighted facial expression features and speech intonation features are fused through the cross-modal attention mechanism; according to the fused facial expression features and speech intonation features, the user's emotional state is identified through a multimodal large model.
[0041] In this embodiment, in multimodal fusion, the correlation between different modalities is enhanced by the attention mechanism. First, the attention mechanism can automatically learn the correlation between different modalities and apply these correlations to information fusion between modalities, thereby helping the model to better understand and process multimodal data. Secondly, the introduction of the attention mechanism can help the model focus on the most important information in different modalities. By learning the attention weights of each modality, the model can dynamically adjust the contribution of each modality in a specific task. Optionally, the cross-attention mechanism can be used to enhance the representation ability of the model by calculating the mutual influence between different modalities, helping the model to better capture the correlation between different modalities. Specifically, the attention mechanism is applied to facial expression features and speech intonation feature vectors to obtain a weighted combination of features. This method generates attention weights by calculating the correlation between different modal features, and then uses these weights to perform weighted combination of facial expression features and speech intonation features. Then, the cross-modal attention mechanism is used to achieve pairwise feature fusion between visual and speech modalities, that is, to establish a connection between different modalities.
[0042] In an optional implementation of the first aspect of the present invention, generating a corresponding psychological assessment report for a user according to the emotional behavior characteristics includes: Based on emotional behavior characteristics, multi-dimensional evaluation reference indicators are generated; based on the evaluation reference indicators, a corresponding psychological evaluation report is generated for the user.
[0043] In this embodiment, based on the emotional behavior characteristics generated by the multimodal large model, multi-dimensional evaluation reference indicators are generated to provide a more comprehensive analysis basis for the user's mental health status.
[0044] See also Figure 2 The second aspect of the present invention provides a multi-modal fusion psychological assessment report generation device, the multi-modal fusion psychological assessment report generation device comprising: The video data acquisition module 10 is used to acquire the user's video data, which is the video data recorded when the user answers questions in the DSM-5 psychological assessment question bank, and the video data includes the user's image and audio data; A feature extraction module 20 is used to extract the user's facial expression features and voice intonation features from the video data; A text extraction module 30 is used to obtain the user's speech text, which is the text extracted from the user's answers to questions in the DSM-5 psychological assessment question bank; The multimodal fusion module 40 is used to obtain the user's emotional behavior characteristics through a multimodal large model according to facial expression characteristics, voice intonation characteristics, and speech text; The report generation module 50 is used to generate a corresponding psychological assessment report for the user according to the emotional behavior characteristics.
[0045] In an optional implementation of the second aspect of the present invention, the multimodal fusion module 40 is also used to identify the user's emotional state based on facial expression features and voice intonation features through a multimodal large model; fuse the emotional state and speech text to obtain the user's emotional behavior characteristics; obtain the user's corresponding DSM-5 psychological assessment question bank's historical question and answer records, and generate empathic responses through an empathy model based on the emotional behavior characteristics and historical question and answer records.
[0046] In an optional implementation of the second aspect of the present invention, the multimodal fusion psychological assessment report generation device further includes: The emotional state judgment module is used to obtain the user's historical emotional state and current emotional state. The historical emotional state and the current emotional state are obtained by multimodal fusion based on the same facial expression features and voice intonation features; judge whether the historical emotional state is consistent with the current emotional state; if not, weight the historical emotional state and the current emotional state to calculate the user's emotional state; if consistent, enter the step of fusing the emotional state and the speech text to obtain the user's emotional behavior characteristics.
[0047] In an optional implementation of the second aspect of the present invention, the emotional state judgment module is also used to perform a weighted sum of the historical emotional state and the current emotional state with a time-attenuated weight to obtain the weights of the historical emotional state and the current emotional state; and select the corresponding emotional state with the highest weight between the historical emotional state and the current emotional state as the user's emotional state through a statistical analysis method.
[0048] In an optional implementation of the second aspect of the present invention, the multimodal fusion psychological assessment report generation device further includes: The model training module is used to obtain training data sets, which include high-quality manually collected emotional behavior feature data and historical question and answer record data, as well as synthetic emotional behavior feature data and historical question and answer record data; based on the data set, the initial empathy model is trained using a deep learning framework and algorithm; the trained empathy model is fine-tuned using lightweight fine-tuning technology to provide users with appropriate empathetic responses for different psychological counseling scenarios.
[0049] In an optional implementation of the second aspect of the present invention, the multimodal fusion module 40 is also used to calculate the correlation between facial expression features and speech intonation features through an attention mechanism based on facial expression features and speech intonation features, and generate corresponding attention weights for the facial expression features and speech intonation features; according to the attention weights, the facial expression features and speech intonation features are weighted, and the weighted facial expression features and speech intonation features are fused through a cross-modal attention mechanism; based on the fused facial expression features and speech intonation features, the user's emotional state is identified through a multimodal large model.
[0050] In an optional implementation of the second aspect of the present invention, the report generation module 50 is also used to generate multi-dimensional evaluation reference indicators according to the emotional behavior characteristics; and generate a corresponding psychological evaluation report for the user according to the evaluation reference indicators.
[0051] Figure 3 is a schematic diagram of the structure of a multimodal fusion psychological assessment report generating device provided by an embodiment of the present invention. The multimodal fusion psychological assessment report generating device 500 may have relatively large differences due to different configurations or performances, and may include one or more processors (central processing units, CPU) 510 (for example, one or more processors) and a memory 520, and one or more storage media 530 (for example, one or more mass storage devices) storing application programs 533 or data 532. Among them, the memory 520 and the storage medium 530 can be short-term storage or permanent storage. The program stored in the storage medium 530 may include one or more modules (not shown in the figure), and each module may include a series of instruction operations in the multimodal fusion psychological assessment report generating device 500. Furthermore, the processor 510 may be configured to communicate with the storage medium 530 to execute a series of instruction operations in the storage medium 530 on the multimodal fusion psychological assessment report generating device 500.
[0052] The psychological assessment report generating device 500 based on multimodal fusion may also include one or more power supplies 540, one or more wired or wireless network interfaces 550, one or more input and output interfaces 560, and / or one or more operating systems 531, such as Windows Serve, Mac OS X, Unix, Linux, Free BSD, etc. Those skilled in the art will appreciate that Figure 3 The structure of the multimodal fusion psychological assessment report generation device shown does not constitute a limitation on the multimodal fusion-based psychological assessment report generation device, and may include more or fewer components than shown in the figure, or a combination of certain components, or a different arrangement of components.
[0053] The present invention also provides a computer-readable storage medium, which may be a non-volatile computer-readable storage medium or a volatile computer-readable storage medium. The computer-readable storage medium stores instructions, and when the instructions are executed on a computer, the computer executes the steps of the multimodal fusion psychological assessment report generation method.
[0054] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
[0055] In addition, although each operation is described in a specific order, this should be understood as requiring such operation to be performed in the specific order shown or in a sequential order, or requiring that all illustrated operations should be performed to obtain desired results. Under certain circumstances, multitasking and parallel processing may be advantageous. Similarly, although some specific implementation details are included in the above discussion, these should not be interpreted as limiting the scope of the present disclosure. Some features described in the context of a separate embodiment can also be implemented in a single implementation in combination. On the contrary, the various features described in the context of a single implementation can also be implemented in multiple implementations individually or in any suitable sub-combination mode.
[0056] Although the subject matter has been described in language specific to structural features and / or methodological logical actions, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. On the contrary, the specific features and actions described above are merely example forms of implementing the claims.
Claims
1. A multimodal fusion psychological assessment report generation method, characterized in that: The multimodal fusion psychological assessment report generation method includes: Acquire video data of the user, wherein the video data is video data recorded when the user answers questions in the DSM-5 psychological assessment question bank, and the video data includes user images and audio data; Extracting facial expression features and voice intonation features of the user from the video data; Acquire the user's speech text, wherein the speech text is text extracted from the user's answers to questions in the DSM-5 psychological assessment question bank; According to the facial expression features, the voice intonation features, and the speech text, the emotional behavior features of the user are obtained through a multimodal large model; According to the emotional behavior characteristics, a corresponding psychological assessment report is generated for the user.
2. The method for generating a multimodal fusion psychological assessment report according to claim 1, characterized in that: The step of obtaining the user's emotional behavior features through a multimodal large model based on the facial expression features, the voice intonation features, and the speech text includes: According to the facial expression features and the voice intonation features, the emotional state of the user is identified through a multimodal large model; The emotional state and the speech text are integrated to obtain the emotional behavior characteristics of the user; The historical question and answer records of the DSM-5 psychological assessment question bank corresponding to the user are obtained, and an empathy response is generated through an empathy model according to the emotional behavior characteristics and the historical question and answer records.
3. The method for generating a multimodal fusion psychological assessment report according to claim 2, characterized in that: Before fusing the emotional state and the speech text to obtain the emotional behavior characteristics of the user, the method further includes: Acquire the historical emotional state and the current emotional state of the user, wherein the historical emotional state and the current emotional state are obtained by multimodal fusion based on the same facial expression features and the voice intonation features; Determining whether the historical emotional state is consistent with the current emotional state; If they are inconsistent, weighting the historical emotional state and the current emotional state to calculate the emotional state of the user; If they are consistent, the step of fusing the emotional state and the speech text to obtain the emotional behavior characteristics of the user is entered.
4. The method for generating a multimodal fusion psychological assessment report according to claim 3, characterized in that: The weighting of the historical emotional state and the current emotional state to calculate the emotional state of the user includes: Performing a weighted sum of the historical emotional state and the current emotional state with a time-attenuated weight to obtain a weight of the historical emotional state and the current emotional state; The corresponding emotional state with the highest weight between the historical emotional state and the current emotional state is selected as the emotional state of the user through a statistical analysis method.
5. The method for generating a multimodal fusion psychological assessment report according to claim 2, characterized in that: The method further includes: obtaining the historical question and answer records of the DSM-5 psychological assessment question bank corresponding to the user, and generating an empathy response through an empathy model according to the emotional behavior characteristics and the historical question and answer records. Acquire a training data set, the data set including high-quality manually collected emotional behavior feature data and historical question and answer record data, as well as synthesized emotional behavior feature data and historical question and answer record data; Based on the data set, an initial empathy model is trained using a deep learning framework and algorithm; The trained empathy model is fine-tuned using lightweight fine-tuning technology to provide appropriate empathic responses to the user for different psychological counseling scenarios.
6. The method for generating a multimodal fusion psychological assessment report according to claim 2, characterized in that: The identifying the user's emotional state according to the facial expression features and the voice intonation features through a multimodal large model includes: According to the facial expression features and the voice and intonation features, the correlation between the facial expression features and the voice and intonation features is calculated through an attention mechanism, and corresponding attention weights are generated for the facial expression features and the voice and intonation features; According to the attention weight, the facial expression features and the voice and intonation features are weighted, and the weighted facial expression features and the voice and intonation features are fused through a cross-modal attention mechanism. Based on the fused facial expression features and the voice intonation features, the emotional state of the user is identified through a multimodal large model.
7. The method for generating a multimodal fusion psychological assessment report according to claim 1, characterized in that: Generating a corresponding psychological assessment report for the user according to the emotional behavior characteristics includes: Generate multi-dimensional evaluation reference indicators based on the emotional behavior characteristics; Based on the evaluation reference indicators, a corresponding psychological evaluation report is generated for the user.
8. A multi-modal fusion psychological assessment report generation device, characterized in that: The multimodal fusion psychological assessment report generating device comprises: A video data acquisition module, used to acquire video data of the user, wherein the video data is the video data recorded when the user answers questions in the DSM-5 psychological assessment question bank, and the video data includes user images and audio data; A feature extraction module, used to extract the facial expression features and voice intonation features of the user from the video data; A text extraction module, used to obtain the user's speech text, wherein the speech text is text extracted from the user's answers to questions in the DSM-5 psychological assessment question bank; A multimodal fusion module, used to obtain the emotional behavior characteristics of the user through a multimodal large model according to the facial expression characteristics, the voice intonation characteristics, and the speech text; The report generation module is used to generate a corresponding psychological assessment report for the user according to the emotional behavior characteristics.
9. A multi-modal fusion psychological assessment report generation device, characterized in that: The multimodal fusion psychological assessment report generation device comprises: a memory and at least one processor, the memory stores instructions, and the memory and the at least one processor are interconnected via a line; The at least one processor calls the instructions in the memory to enable the multimodal fusion psychological assessment report generation device to execute the multimodal fusion psychological assessment report generation method as described in any one of claims 1-7.
10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method for generating a multimodal fusion psychological assessment report as described in any one of claims 1 to 7 is implemented.
Citation Information
Patent Citations
Method for enhancing positive psychological state
CN109977100A
Method for constructing memory network and applying memory network to emotion prediction
CN110222026A
Psychological assessment system and platform, electronic equipment and storage medium
CN115862868A
Common-condition reply generation method and device, terminal and storage medium
CN115934909A
Cross-modal video sentiment analysis method and device, equipment and storage medium
CN117633613A
Cited By
Robot for simulating interrogation of psychologists, equipment and storage medium
CN119248139A
Psychological state prediction system and method based on large language model
CN119271790A
Multi-modal mental health prediction method, electronic equipment and medium
CN119581025A
