Dialogue understanding and psychological analysis method and system based on psychological health model

Abstract

The invention relates to the technical field of deep learning, in particular to a dialogue understanding and psychological analysis method and system based on a psychological health model, and the method comprises the steps: obtaining first dialogue data and first face data; performing feature extraction and fusion on the first dialogue data and the first face data, and updating the fused features into historical context features; generating second dialogue data according to the historical context features; performing feature extraction and fusion on the second face data and the second dialogue data, and updating historical context features; and generating a psychological assessment result according to the updated historical context features. And predicting second dialogue data through the first dialogue data and the historical context features during dialogue. And when the dialogue is completed, predicting a psychological assessment result through the updated historical context features. According to the method, the historical context features fusing the face data and the dialogue data are directly used as input for training, so that the parameter quantity can be greatly reduced.

Description

Technical Field

[0001] The present application relates to the field of deep learning technology, and in particular to a method and system for dialogue understanding and psychological analysis based on a mental health model. Background Art

[0002] At present, neural network models are increasingly used in the medical field for auxiliary treatment, especially in the fields of psychoanalysis and psychological assessment. The application of large language models in psychological assessment has begun to take shape. Like intelligent chat robots, large language models obtain patients' conversation text data for depression detection, suicide detection, cognitive distortion detection and relationship counseling.

[0003] Large language models require input of a large amount of natural language data and a dataset of associations between psychological symptoms during pre-training, so that they can generate specific psychological diagnostic outputs.

[0004] In the existing large language models, correctly understanding human natural language and determining intent is the main technical problem currently faced. The current neural network model understands natural language through structures such as state machines and decision machines, and generates corresponding answer texts. At present, the large language model can only understand the literal intent in the dialogue text, but cannot identify the true intent hidden in the literal meaning.

[0005] In psychotherapy, the psychologist's task is more to identify the patient's hidden true intentions and guide the patient to express them. In order to identify the patient's hidden true intentions, the existing psychological analysis model uses the method of obtaining the patient's micro-expressions to understand human emotions, thereby assisting in judging the patient's true intentions.

[0006] Therefore, in order to enable the neural network model to understand human emotions in psychological assessment, multimodal data such as conversation content, facial expressions, and body language will be collected as input to the model to improve the accuracy of the psychological assessment results.

[0007] Large language models are constantly making breakthroughs in simulating psychological counseling. Multimodal data will cause a sharp increase in the amount of data in the neural network model, resulting in a larger amount of computing power for the device, making it impossible to achieve real-time psychological counseling conversations.

[0008] In order to solve the above problems, this application provides a dialogue understanding and psychological analysis method based on the mental health model. Summary of the invention

[0009] In order to overcome the problems existing in the related art, the first aspect of the present application provides a conversation understanding and psychological analysis method based on a mental health model, comprising: Acquire first conversation data and first facial data; Extracting and fusing features of the first conversation data and the first facial data, and updating the fused features into historical context features; Generate second conversation data according to the historical context feature; the second conversation data is used to send to the user device to trigger the next round of conversation and second facial data; Extracting and fusing features of the second facial data and the second conversation data, and updating the historical context features; A psychological assessment result is generated according to the updated historical context features.

[0010] In one embodiment, comprising: The obtaining of the first conversation data and the first facial data specifically includes: Get interactive video; The interactive video is parsed and extracted to obtain the first conversation data and the first facial data.

[0011] In one implementation, parsing and extracting the interactive video to obtain the first conversation data and the first facial data specifically includes: The first conversation data includes the conversation content, and the first facial data includes facial frame images sorted by timestamps, wherein the facial frame images are 3D depth maps; is an integer greater than or equal to 1.

[0012] In one implementation, the first dialogue data is dialogue data sent by the user to the human-computer interaction device, and the second dialogue data is dialogue data sent by the human-computer interaction device to the user; The feature extraction of the conversation data includes: The word segments in the first conversation data and the second conversation data are encoded into word vector representations and arranged according to timestamps.

[0013] In one embodiment, the first facial data is the facial data of the user in the process of sending the first conversation data, and the second facial data is the facial data of the user in the process of receiving the second conversation data; The feature extraction of the facial data specifically includes: Each frame of the first facial data or the second facial data is encoded and arranged according to timestamps.

[0014] In one embodiment, the feature extraction of the facial data further includes: Acquire facial data; the facial data includes facial frame images, is an integer greater than or equal to 1; Preprocessing the facial frame image; Determine the facial contour according to the laser dot matrix in the facial frame image; determining an emotion score based on the facial contour; The emotion scores are encoded into facial features.

[0015] In one implementation, determining the emotion score according to the facial contour specifically includes: The calculation formula of the emotion score is:

[0016] in, for the facial contour, For emotional type , The facial contour In the emotion type The sentiment score in For a given category, the emotion type The facial contour The conditional probability of For the emotion type The prior probability of For the facial contour The total probability of occurrence.

[0017] In one embodiment, generating a psychological assessment result according to the historical context features specifically includes: Constructing a psychological evaluation form; the psychological evaluation form includes user information, case information and evaluation information; The user information, the case information and the evaluation information are generated by inputting a pre-trained mental health model according to the features in the historical context features.

[0018] The second aspect of the present application provides a conversation understanding and psychological analysis system based on a mental health model, which is used to execute the steps of the method described in the first aspect of the present application, including: a pre-trained mental health model; The mental health model includes a data processing module, an LLM dialogue module and a psychological assessment module; The data processing module is used for feature extraction and fusion; The LLM dialogue module is used to predict the second dialogue data according to the historical context feature; The psychological assessment module is used to predict the psychological assessment result according to the updated historical context features.

[0019] The technical solution provided by this application may have the following beneficial effects: In order to improve the processing speed of the model for multimodal data, the present application fuses the conversation data and facial data after extracting the features, and constructs the historical context features arranged by timestamp. When the conversation is in progress, the second conversation data is predicted by the first conversation data and the historical context features. When the conversation is completed, the psychological assessment result is predicted by the updated historical context features. In addition, the model can be trained directly based on the historical context features that fuse the facial data and conversation data as input, which can greatly reduce the number of parameters.

[0020] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The above and other objects, features and advantages of the present application will become more apparent through a more detailed description of exemplary embodiments of the present application in conjunction with the accompanying drawings, wherein the same reference numerals generally represent the same components in the exemplary embodiments of the present application.

[0022] Figure 1 A flowchart of the conversation understanding and psychological analysis method shown in the embodiment of the present application; Figure 2 for Figure 1 A flow chart of step S1 in the method for dialogue understanding and psychological analysis shown; Figure 3 for Figure 1 Schematic diagram of the feature extraction process of facial data in the conversation understanding and psychological analysis method shown. DETAILED DESCRIPTION

[0023] The preferred embodiments of the present application will be described in more detail below with reference to the accompanying drawings. Although the preferred embodiments of the present application are shown in the accompanying drawings, it should be understood that the present application can be implemented in various forms and should not be limited by the embodiments described herein. On the contrary, these embodiments are provided to make the present application more thorough and complete, and to fully convey the scope of the present application to those skilled in the art.

[0024] The terms used in this application are for the purpose of describing specific embodiments only and are not intended to limit this application. The singular forms of "a", "said" and "the" used in this application and the appended claims are also intended to include plural forms unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used herein refers to and includes any or all possible combinations of one or more associated listed items.

[0025] It should be understood that although the terms "first", "second", "third", etc. may be used in this application to describe various information, this information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of this application, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Thus, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. In the description of this application, the meaning of "multiple" is two or more, unless otherwise clearly and specifically defined.

[0026] Embodiment 1 In the psychological counseling system based on the neural network model, although collecting multimodal data such as user conversations and facial images can provide richer information, it also leads to huge computing power required to achieve real-time conversations.

[0027] In order to improve the response speed of the psychological counseling system to multimodal data and realize real-time psychological counseling dialogue, the embodiment of the present application provides a dialogue understanding and psychological analysis method based on a mental health model, such as Figure 1 As shown, the following steps are included: S1, obtaining first conversation data and first facial data; S2, extracting and fusing features of the first conversation data and the first facial data, and updating the fused features into historical context features; S3, generating second conversation data according to the historical context feature; the second conversation data is used to send to the user device to trigger the next round of conversation and second facial data; S4, extracting and fusing features of the second facial data and the second conversation data, and updating the historical context features; S5. Generate a psychological assessment result according to the updated historical context features.

[0028] It should be noted that the conversation understanding and psychological analysis method described in the embodiment of the present application is implemented based on a pre-trained mental health model, using a human-computer interaction device to capture the user's interactive video and use it as input data. The model processes the data through various modules to generate and reply to the user's conversation content.

[0029] Specifically, the first dialogue data is dialogue data sent by the user to the human-computer interaction device, and the second dialogue data is dialogue data sent by the human-computer interaction device to the user. The first facial data is facial data of the user in the process of sending the first dialogue data, and the second facial data is facial data of the user in the process of receiving the second dialogue data.

[0030] Further, such as Figure 2 As shown, the specific steps of S1 include: S101, obtaining an interactive video; the interactive video is recorded by a human-computer interaction device.

[0031] S102: Parse and extract the interactive video to obtain the first dialogue data and the first facial data.

[0032] Specifically, in S102, the first conversation data and the first facial data are respectively exported from the interactive video.

[0033] The first conversation data includes the conversation content, and the first facial data includes facial frame images sorted by timestamps, wherein the facial frame images are 3D depth maps; is an integer greater than or equal to 1.

[0034] Furthermore, step S2 and step S4 are feature engineering steps, which respectively perform agent extraction and feature fusion on the first conversation data, the second conversation data, the first facial data, and the second facial data.

[0035] Specifically, the word segmentations in the first conversation data and the second conversation data are encoded as word vector representations and arranged according to timestamps.

[0036] Exemplarily, the first conversation data is represented as:

[0037] in, is the first conversation data word vectors, For the The timestamp of the word vector.

[0038] Each frame of the first facial data or the second facial data is encoded and arranged according to the timestamp. The first facial data or the second facial data is represented as:

[0039] in, For the timestamp of the facial frame image, For the face frame images.

[0040] Exemplarily, the matrix of the first facial feature or the second facial feature is expressed as:

[0041] in, For timestamp The facial features of the facial frame image at the moment.

[0042] In the fusion step, the first conversation feature and the first facial feature, the second conversation feature and the second facial feature are fused, and the fused matrix is:

[0043] In the embodiment of the present application, the fused matrix is ​​added to the historical context features. The historical context features are stored in the order of timestamps after the above feature extraction and fusion.

[0044] In step S3, after the first conversation data and the first facial data are acquired and processed, they are input into the model for prediction to obtain the second conversation data. When the user receives the second conversation data, the second facial data is acquired. Similarly, the second conversation data and the second facial data are subjected to feature extraction and the historical context features are added. After the conversation is finished, the psychological assessment result is predicted based on the historical context features.

[0045] Before pre-training, a dialogue-face dataset is first established. The dialogue-face dataset is obtained by collecting dialogue records between patients and doctors in psychological counseling cases and taking facial images of patients, annotating the dialogue records with text, and annotating the facial images with expression.

[0046] In step S5, it specifically includes: Constructing a psychological evaluation form; the psychological evaluation form includes user information, case information and evaluation information; The user information, the case information and the evaluation information are generated by inputting a pre-trained mental health model according to the features in the historical context features.

[0047] In order to improve the processing speed of the model for multimodal data, the embodiment of the present application fuses the conversation data and the facial data after the features are extracted, and constructs the historical context features arranged according to the timestamp. When the conversation is in progress, the second conversation data is predicted by the first conversation data and the historical context features. When the conversation is completed, the psychological assessment result is predicted by the updated historical context features. In addition, the model can be trained directly based on the historical context features that fuse the facial data and the conversation data as input, which can also reduce the number of parameters of the model.

[0048] Embodiment 2 Based on the first embodiment, the present application embodiment describes in detail the feature engineering performed on the first facial data and the second facial data shown in the first embodiment.

[0049] In the field of psychology, experienced counselors will conduct psychological analysis by comprehensively analyzing information such as conversations, micro-expressions, and intonation. Micro-expressions last about 0.04 to 0.2 seconds, and there are more than 80 types of micro-expressions.

[0050] Therefore, in order to enable the mental health model shown in Example 1 to understand more types of micro-expressions, the micro-expression information and dialogue information are integrated to better simulate the counseling activities of the psychological counselor. This embodiment of the application provides a dialogue understanding and psychological analysis method based on the mental health model, such as Figure 1 As shown, the following steps are included: S1, obtaining first conversation data and first facial data; S2, extracting and fusing features of the first conversation data and the first facial data, and updating the fused features into historical context features; S3, generating second conversation data according to the historical context feature; the second conversation data is used to send to the user device to trigger the next round of conversation and second facial data; S4, extracting and fusing features of the second facial data and the second conversation data, and updating the historical context features; S5. Generate a psychological assessment result according to the updated historical context features.

[0051] In the feature engineering of the first facial data and the second facial data, the facial data is obtained from a video stream captured by a depth camera.

[0052] Specifically, the depth camera is a structured light depth camera, which includes a camera and a projector. The projector projects a laser dot matrix onto the user's face. The camera captures an image of the face and records the deformation information of the laser dot matrix on the face, thereby identifying changes in the user's expression.

[0053] Further, such as Figure 3 As shown, the feature engineering steps for facial data include: S201, obtaining facial data; the facial data includes facial frame images, is an integer greater than or equal to 1.

[0054] S202, preprocessing the facial frame image; Specifically, the facial region in the facial frame image is cut out and stored in a size of 840×460 pixels.

[0055] Specifically, the facial frame image The matrix after truncation is expressed as:

[0056] in, is a pixel point on the facial frame image.

[0057] S203, determining the facial contour according to the laser dot matrix in the facial frame image; It can be understood that step S203 is implemented based on a speckle structured light recognition algorithm.

[0058] Specifically, firstly, the light spot offset value in the laser dot matrix is ​​determined according to the infrared brightness of each pixel point on the facial frame image, and the facial contour is calculated according to the light spot offset value.

[0059] S204, determining an emotion score according to the facial contour; Specifically, the emotion score is the probability of predicting that the facial contour belongs to each emotion type. The emotion type includes 7 basic emotions, such as anger, disgust, fear, happiness, sadness, surprise and depression.

[0060] S205: Encode the emotion score into facial features.

[0061] Specifically, the facial features The encoding matrix is:

[0062] in, Indicates that the facial contour is Medium type of sentiment score.

[0063] Furthermore, the calculation formula of the emotion score is:

[0064] in, for the facial contour, For emotional type , The facial contour In the emotion type The sentiment score in For a given category, the emotion type The facial contour The conditional probability of For the emotion type The prior probability of For the facial contour The total probability of occurrence.

[0065] In the embodiment of the present application, the k facial frame images in the facial data represent the user's emotional change information. After extracting the user's facial contour, a probability prediction is performed on the emotion type presented by the user's facial contour. The probability of the emotion type is encoded to obtain facial features. Therefore, when predicting, the model can predict and generate the second dialogue data according to the user's emotion information.

[0066] Embodiment 3 A conversation understanding and psychological analysis system based on a mental health model, used to execute the steps in the conversation understanding and psychological analysis method of implementation one or implementation two, comprising: a pre-trained mental health model; The mental health model includes a data processing module, an LLM dialogue module and a psychological assessment module.

[0067] The data processing module is used to perform feature engineering on the conversation data and the facial data. The LLM conversation module is used to predict the second conversation data according to the historical context features. The psychological assessment module is used to predict the psychological assessment result according to the updated historical context features.

[0068] Regarding the device in the above embodiment, the specific way in which each module performs the operation has been described in detail in the embodiment of the method, and will not be elaborated here.

[0069] The scheme of the present application has been described in detail above with reference to the accompanying drawings. In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments. Those skilled in the art should also be aware that the actions and modules involved in the description are not necessarily required for the present application.

[0070] In addition, it can be understood that the steps in the method of the embodiment of the present application can be adjusted in order, combined and deleted according to actual needs, and the modules in the device of the embodiment of the present application can be combined, divided and deleted according to actual needs.

[0071] In addition, the method according to the present application may also be implemented as a computer program or a computer program product, which includes computer program code instructions for executing some or all of the steps in the above method of the present application.

[0072] Alternatively, the present application can also be implemented as a non-temporary machine-readable storage medium (or computer-readable storage medium, or machine-readable storage medium) on which executable code (or computer program, or computer instruction code) is stored. When the executable code (or computer program, or computer instruction code) is executed by a processor of an electronic device (or electronic device, server, etc.), the processor executes part or all of the steps of the above-mentioned method according to the present application.

[0073] Those skilled in the art will further appreciate that the various illustrative logical blocks, modules, circuits, and algorithm steps described in connection with the application herein may be implemented as electronic hardware, computer software, or combinations of both.

[0074] The flow chart and block diagram in the accompanying drawings show the possible architecture, function and operation of the system and method according to multiple embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a part of a module, a program segment or a code, and the part of the module, the program segment or the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two continuous square boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or the flow chart, and the combination of the square boxes in the block diagram and / or the flow chart can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0075] The embodiments of the present application have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and changes will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The selection of terms used herein is intended to best explain the principles of the embodiments, practical applications, or improvements to the technology in the market, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein.

Claims

1. A method for conversation understanding and psychological analysis based on a mental health model, characterized in that: include: Acquire first conversation data and first facial data; Extracting and fusing features of the first conversation data and the first facial data, and updating the fused features into historical context features; generating second conversation data according to the historical context feature; The second conversation data is used to be sent to the user device to trigger the next round of conversation and the second facial data; Extracting and fusing features of the second facial data and the second conversation data, and updating the historical context features; A psychological assessment result is generated according to the updated historical context features.

2. The method for dialogue understanding and psychological analysis based on a psychological health model according to claim 1, characterized in that: include: The obtaining of the first conversation data and the first facial data specifically includes: Get interactive video; The interactive video is parsed and extracted to obtain the first conversation data and the first facial data.

3. The method for dialogue understanding and psychological analysis based on a psychological health model according to claim 2 is characterized in that: The parsing and extracting the interactive video to obtain the first dialogue data and the first facial data specifically includes: The first conversation data includes the conversation content, and the first facial data includes facial frame images sorted by timestamps, wherein the facial frame images are 3D depth maps; is an integer greater than or equal to 1.

4. The method for dialogue understanding and psychological analysis based on a mental health model according to claim 1, characterized in that: The first dialogue data is dialogue data sent by the user to the human-computer interaction device, and the second dialogue data is dialogue data sent by the human-computer interaction device to the user; The feature extraction of the conversation data includes: The word segments in the first conversation data and the second conversation data are encoded into word vector representations and arranged according to timestamps.

5. The method for dialogue understanding and psychological analysis based on a psychological health model according to claim 4 is characterized in that: The first facial data is the facial data of the user in the process of sending the first conversation data, and the second facial data is the facial data of the user in the process of receiving the second conversation data; The feature extraction of the facial data specifically includes: Each frame of the first facial data or the second facial data is encoded and arranged according to timestamps.

6. The method for dialogue understanding and psychological analysis based on a psychological health model according to claim 5, characterized in that: The feature extraction of the facial data also includes: Acquire facial data; the facial data includes facial frame images, is an integer greater than or equal to 1; Preprocessing the facial frame image; Determine the facial contour according to the laser dot matrix in the facial frame image; determining an emotion score based on the facial contour; The emotion scores are encoded into facial features.

7. The method for dialogue understanding and psychological analysis based on a psychological health model according to claim 6, characterized in that: Determining the emotion score according to the facial contour specifically includes: The calculation formula of the emotion score is: in, for the facial contour, For emotional type , The facial contour In the emotion type The sentiment score in For a given category, the emotion type The facial contour The conditional probability of For the emotion type The prior probability of For the facial contour The total probability of occurrence.

8. The method for dialogue understanding and psychological analysis based on a mental health model according to claim 1, characterized in that: Generate psychological assessment results based on the historical context features, specifically including: Constructing a psychological evaluation form; the psychological evaluation form includes user information, case information and evaluation information; The user information, the case information and the evaluation information are generated by inputting a pre-trained mental health model according to the features in the historical context features.

9. A conversation understanding and psychological analysis system based on a psychological health model, used to perform the steps in the method according to any one of claims 1 to 8, characterized in that: Including: the pre-trained mental health model; The mental health model includes a data processing module, an LLM dialogue module and a psychological assessment module; The data processing module is used for feature extraction and fusion; The LLM dialogue module is used to predict the second dialogue data according to the historical context feature; The psychological assessment module is used to predict the psychological assessment result according to the updated historical context features.

Images