A multimodal depression prediction system based on topic cues

Abstract

A multimodal depression prediction system based on topic prompts. The invention belongs to the field of multimodal recognition prediction technology, and specifically relates to the field of multimodal depression prediction technology. The multimodal depression prediction system based on topic prompts includes a personal information memory module, a prompt pool module, a topic prompt query module, a prompt pool optimization module, a text vision and voice prompt coupling module, a deep language prompt module, a deep visual prompt module, a deep voice prompt module and a joint training module. The personal information is split, and the wide part is used to remember the personal information and the deep part is used for multimodal joint training to realize the personalization of depression detection. The personal information is personalized and adapted by memorizing the personal information, which is more accurate than identifying the speaker. The combination of topic prompts and multimodality provides richer diagnostic information than single modalities such as voice and text.

Description

Technical Field

[0001] The present invention belongs to the technical field of multimodal recognition and prediction, and specifically relates to the technical field of multimodal depression degree prediction. Background Art

[0002] Exploring simple, objective, and accurate auxiliary diagnostic methods for depression has long been a major challenge to be solved in the field of medical public health. However, in the absence of multimodal depression patient data, current depression detection technology has problems such as unstable detection performance and insufficient personalization capabilities, which cannot meet the needs of practical applications.

[0003] A multimodal depression recognition system that integrates different channels and cues is expected to provide more accurate recognition than single-modal methods. Some auxiliary depression diagnosis methods use multimodal input, in which different modalities are integrated, such as the text of the conversation, facial expressions, and speech prosody. Some studies have done some work on the problem of early fusion and late fusion, providing references for multimodal depression recognition.

[0004] The same expression conveys different information under different interview topics. For example, a smile under the topic of discussing family is more discerning than a smile under the topic of greeting others, because the latter may be just for etiquette reasons. In addition, in the actual interview process, individual answers to different topics also vary.

[0005] The practical application of existing objective observation multimodal depression recognition methods still faces many urgent problems that need to be solved: (1) Most studies ignore the impact of interview topics on depression recognition. (2) Existing multimodal depression observation methods are difficult to effectively address the impact of individual differences such as age and gender. Therefore, further improving the stability and generalization ability of depression detection is an urgent need to enable this technology to move towards practical application. Summary of the invention

[0006] In order to solve the technical defects of the existing objective observation multimodal depression recognition method, the present invention provides a multimodal depression degree prediction system based on topic prompts, the system includes a personal information memory module, a prompt pool module, a topic prompt query module, a prompt pool optimization module, a text vision and voice prompt coupling module, a deep text prompt module, a deep visual prompt module, a deep voice prompt module and a joint training module;

[0007] The personal information memory module is used to perform cross-feature conversion and memory on the personal information of the interviewee;

[0008] The prompt pool module constructs a key-value pair composed of a key and a value as a text topic prompt of the interview text;

[0009] The topic prompt query module is used to pair the interview text with the text topic prompt;

[0010] The prompt pool optimization module optimizes the text topic prompts paired with the interview text and outputs the optimal text topic prompts to the text vision and voice prompt coupling module and the deep language prompt module;

[0011] The text-vision and voice prompt coupling module: couples the optimal text topic prompt to the dimension of the visual modality through the text-vision coupling function to obtain the visual topic prompt; couples the optimal text topic prompt to the dimension of the voice modality through the text-voice coupling function to obtain the voice topic prompt;

[0012] The deep text prompt module: performs word embedding operation on the interview text to obtain interview text features, and uses k-layer Transformer layers to encode the optimal text topic prompts and interview text features;

[0013] The deep visual cue module: performs image block embedding operation on the interview video to obtain interview video features, and uses k-layer Transformer layers to encode visual topic cues and interview video features;

[0014] The deep voice prompt module: performs voice embedding operation on the interview audio to obtain interview audio features, and uses k-layer Transformer layers to encode the voice topic prompts and interview audio features;

[0015] The joint training module obtains a prediction result based on the input of the personal information memory module, the deep text prompt module, the deep visual prompt module and the deep voice prompt module, and then feeds the prediction result back to the loss function to jointly train the personal information memory module, the deep text prompt module, the deep visual prompt module and the deep voice prompt module until the output prediction result meets the requirements.

[0016] The beneficial effects of the method of the present invention are:

[0017] (1) Split personal information, use the wide part to remember personal information and the deep part to perform multimodal joint training to achieve personalized depression detection. Personalized adaptation through memorizing personal information is more accurate than identifying the speaker. Through topic prompts and multimodal combination, it has richer diagnostic information than single modalities such as voice and text.

[0018] (2) During training, corresponding prompts are provided for the interview text features, interview video features, and interview audio features, namely the optimal text topic prompts, visual topic prompts, and voice topic prompts, which can further improve the accuracy of subsequent prediction results. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 This is a structural diagram of a multimodal depression level prediction system based on topic prompts described in an embodiment of the present invention;

[0020] Figure 2 A schematic diagram of a personal information memory module according to an embodiment of the present invention;

[0021] Figure 3 This is a schematic diagram of converting personal information into sparse binary features through one-hot encoding in an embodiment of the present invention;

[0022] Figure 4 It is a structural diagram of the deep text prompt module, the deep visual prompt module and the deep voice prompt module described in the embodiment of the present invention. DETAILED DESCRIPTION

[0023] The technical solution of the present invention will be described clearly and completely below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the protection scope of the present invention.

[0024] This embodiment provides a multimodal depression level prediction system based on topic prompts, such as Figure 1 As shown, it includes a personal information memory module, a prompt pool module, a topic prompt query module, a prompt pool optimization module, a text vision and voice prompt coupling module, a deep text prompt module, a deep visual prompt module, a deep voice prompt module and a joint training module;

[0025] The personal information memory module is used to perform cross-feature conversion and memory on the personal information of the interviewee;

[0026] The prompt pool module constructs a key-value pair composed of a key and a value as a text topic prompt of the interview text;

[0027] The topic prompt query module is used to pair the interview text with the text topic prompt;

[0028] The prompt pool optimization module optimizes the text topic prompts paired with the interview text and outputs the optimal text topic prompts to the text vision and voice prompt coupling module and the deep language prompt module;

[0029] The text-vision and voice prompt coupling module: couples the optimal text topic prompt to the dimension of the visual modality through the text-vision coupling function to obtain the visual topic prompt; couples the optimal text topic prompt to the dimension of the voice modality through the text-voice coupling function to obtain the voice topic prompt;

[0030] The deep text prompt module: performs word embedding operation on the interview text to obtain interview text features, and uses k-layer Transformer layers to encode the optimal text topic prompts and interview text features;

[0031] The deep visual cue module: performs a patch embedding operation on the interview video to obtain interview video features, and uses k-layer Transformer layers to encode the visual topic cues and interview video features;

[0032] The deep voice prompt module: performs audio embedding operation on the interview audio to obtain the interview audio features, and uses k-layer Transformer layers to encode the voice topic prompts and the interview audio features;

[0033] The joint training module obtains a prediction result based on the input of the personal information memory module, the deep text prompt module, the deep visual prompt module and the deep voice prompt module, and then feeds the prediction result back to the loss function to jointly train the personal information memory module, the deep text prompt module, the deep visual prompt module and the deep voice prompt module until the output prediction result meets the requirements.

[0034] A detailed introduction to the personal information memory module:

[0035] Personal information memory module such as Figure 2 As shown in the figure, the specific data processing method of the human information memory module is explained by taking the age and gender of the interviewee as an example:

[0036] The personal information is converted into sparse binary features through one-hot encoding, such as Figure 3 As shown in the figure, the basic principle of one-hot encoding is to set the value corresponding to each category in the categorical variable to 0, and set it to 1 only when the category is consistent. The categorical variable age is divided into 7 possible categories (0-10, ..., 60-70) with 10 years old as the dividing line. Then after one-hot encoding, age will be represented by 7 binary features. If the interviewee is 20-30 years old, the value corresponding to the 20-30 years old category is 1, and the values ​​corresponding to other categories are 0. The categorical variable gender is processed in the same way.

[0037] The cross feature transformation of sparse binary features is performed by:

[0038] ;

[0039] To achieve, d represents the total number of sparse binary features, xi Indicates i The categories corresponding to the binary features are is a Boolean variable. i When the categories corresponding to the binary features meet the requirements, The value is 1 otherwise, 0.

[0040] The feature vector after cross feature transformation is passed through the logistic regression model Conduct training, including represents the model parameters, Represents the input data to be trained, b The purpose of training is to memorize personal information and provide a corresponding basis for the personalization of different personal information.

[0041] A detailed introduction to the prompt pool module:

[0042] The motivation for introducing the cue pool module is threefold: first, the subject individuals are unknown at test time, so training topic cues that are unrelated to the individual's depression is infeasible; second, even if the topic cues for different individuals are known at test time, it will prevent possible knowledge sharing between similar individuals; third, although the simple approach of learning a single shared cue for all individuals can achieve knowledge sharing, it still leads to serious forgetting problems; ideally, a model should be able to share knowledge when individuals are similar, while keeping knowledge independent in other cases. Therefore, the design uses a cue pool to store the encoded cue knowledge, which can be flexibly grouped as input to the model.

[0043] The prompt pool module constructs key-value pairs that combine keys and values ​​as text topic prompts for interview texts:

[0044] Each set of key-value pairs consists of a pair of corresponding keys and values, so the prompt pool can be expressed as P = ,in Represents any key in the prompt pool, Representation and The corresponding value;

[0045] The LDA (Latent Dirichlet Allocation) model is trained on all interview texts. , is a D k ×1 matrix;

[0046] yes Feature representation after word embedding operation.

[0047] The key generation of the prompt pool here is obtained by training all interview texts using the LDA model, and the value is the feature representation after the word embedding corresponding to the generated topic (key). The key-value pair here is automatically matched. The key is the topic generated by LDA, and the value is the feature generated by the topic using word embedding. Therefore, when the subsequent interview text is input, since the features of the interview text are generated by word embedding, the features of the text input at this time must be matched with the keys in the prompt pool, and similarity calculation is used to find the keys with the highest similarity. The key-value pair is found, and the key and value are spliced ​​together. Only the key is used when searching (similarity calculation).

[0048] A detailed introduction to the topic prompt query module:

[0049] The topic prompt query module pairs the interview text with the text topic prompt. The pairing method is as follows:

[0050] First, the interview text of the interviewee is input into the LDA (Latent Dirichlet Allocation) model to obtain the interview text features. ;

[0051] Then, by calculating the interview text features The distance between the key and the key in the prompt pool is used to select the N keys with the smallest distance. The key-value pairs corresponding to the N keys are the text topic prompts. The selection is performed by:

[0052] ;

[0053] To achieve, represents the set of N keys selected from the hint pool, , indicating the calculation and The matching function of the distance between the vectors and is mapped to a real number, which is and The distance between them is preferably calculated by using a matching function that calculates the cosine distance.

[0054] A detailed introduction to the prompt pool optimization module:

[0055] The prompt pool optimization module optimizes the text topic prompts paired with the interview text and outputs the optimal text topic prompts to the language vision and voice prompt coupling module and the deep language prompt module;

[0056] The optimization method is as follows: optimize the loss function used in the joint training module to train the deep text prompt module to obtain an optimized loss function. The specific optimization method is to add the following to the original loss function:

[0057] ;

[0058] in, It is a scalar used for weighted loss, and its preferred value is 0.2.

[0059] Under normal circumstances, in each training step, after selecting N key-value pairs to form text topic prompts according to the query strategy of the topic prompt query module, the text topic prompts are input into the deep text prompt module, and finally the prediction results are obtained in the joint training module. The mean square error between the prediction results and the true results is used as the loss function during training. The purpose of training is to minimize the loss function. The prompt pool optimization module optimizes the loss function and adds the N key-value pairs selected by the topic prompt query module to the loss function. When the loss function is minimized, the interview text features The distance to the keys in the prompt pool is also minimized, thus optimizing the text topic prompt.

[0060] A detailed introduction to the text vision and voice prompt coupling module:

[0061] The text-vision and voice prompt coupling module uses the text-vision coupling function The optimal text topic prompt is coupled to the dimension of the visual modality to obtain the visual topic prompt; through the text-speech coupling function The optimal text topic cue is coupled to the dimension of the speech modality to obtain the speech topic cue; the text-visual coupling function and text-to-speech coupling function Each is a linear layer.

[0062] A detailed introduction to the deep text prompt module:

[0063] The deep text prompt module performs word embedding operations on the interview text to obtain interview text features, and uses k-layer Transformer layers to encode the optimal text topic prompts and interview text features; Figure 4 As shown, the operation of the deep text prompt module is carried out according to the following rules:

[0064] In the L1-L j In the L layer of the deep text prompt module, the optimal text topic prompt input in each layer is not transferred, but the optimal text topic prompt is re-input into each layer. j —L k In each layer, the optimal text topic prompt input from each layer is transferred between layers, that is, L j The optimal text topic prompts in the layer are encoded and then input into L j+1 Layer; such as Figure 4As shown in the figure, the optimal text topic prompt for input L1 layer is recorded as P1, and so on. k The optimal text topic hint at the layer is denoted as P k ; The final output text feature is represented as z, ; The operation represented by is to project the text data to be processed into a multimodal common feature space. Indicates L k All features after layer coding, including optimal text topic prompts and interview text features.

[0065] For the choice of the value j, 1 / 3 of the total number of network layers is recommended; for the entire deep text prompt module, cross-stage shared prompts are better than independent prompts, because as the number of network layers increases, the features are more relevant due to the processing of consecutive Transformer layers, so the subsequent stages do not provide independent prompts compared to the early stages.

[0066] A detailed introduction to the deep visual cue module:

[0067] The deep visual cue module performs image block embedding operations on the interview video to obtain interview video features, and uses k-layer Transformer layers to encode visual topic cues and interview video features; Figure 4 As shown, the operation performed by the deep visual cue module is performed according to the following rules:

[0068] In the depth visual cue module V1-V j In the layers, the visual theme cues input to each layer are not passed on, but the visual theme cues are re-input to each layer. j —V k In each layer, the visual theme prompts input by each layer are transmitted between layers, that is, V j The visual thematic cues in the layer are encoded and then input into V j+1 Layer; such as Figure 4 As shown, the visual theme cue of the input V1 layer is recorded as , and so on, input V k The visual theme hint of the layer is denoted as ; The final output image feature representation x, ; The operation represented by is to project the image data to be processed into a multi-modal common feature space. Indicates V k All features after layer encoding, including visual topic cues and interview video features.

[0069] A detailed introduction to the deep voice prompt module:

[0070] The deep voice prompt module performs voice embedding operations on the interview audio to obtain the interview audio features, and uses k-layer Transformer layers to encode the voice topic prompts and interview audio features; Figure 4 As shown, the operation performed by the deep voice prompt module is carried out according to the following rules:

[0071] In the A1-A of the deep voice prompt module j In the layers, the voice topic prompts input in each layer are not transmitted, but the voice topic prompts are re-input into each layer. j —A k In each layer, the speech topic prompts input from each layer are transmitted between layers, that is, A j The speech topic prompts in the layer are encoded and then input into A j+1 As shown in the title, the voice topic prompt input A1 layer is recorded as , and so on, input A k The voice topic prompt of the layer is recorded as ; The final output audio feature representation a, , The operation represented by is to project the speech data to be processed into a multi-modal common feature space. Indicates A k All features after layer encoding, including speech topic cues and interview audio features.

[0072] A detailed introduction to the joint training module:

[0073] The joint training module obtains a prediction result based on the input of the human information memory module, the deep text prompt module, the deep visual prompt module and the deep voice prompt module, and then feeds the prediction result back to the loss function to jointly train the human information memory module, the deep text prompt module, the deep visual prompt module and the deep voice prompt module until the output prediction result meets the prediction requirements.

[0074] Before the text feature z, image feature x, and audio feature a are input into the joint training module, self-attention is used to assign weights to the three, and then the joint feature is input.

[0075] The joint training module includes an embedding layer, a first activation layer, a second activation layer and a logistic regression layer from input to output, wherein the first activation layer adopts a ReLu activation function, the second activation layer adopts a ReLu activation function, and the logistic regression layer adopts a Sigmoid function to convert the output of the linear regression into a probability value between 0 and 1, so as to obtain a predicted score label as a prediction result.

[0076] Further introduce how the joint training module trains the personal information memory module, deep text prompt module, deep visual prompt module and deep voice prompt module:

[0077] The joint training module uses a joint loss function to train the personal information memory module, deep text prompt module, deep visual prompt module and deep voice prompt module, where the personal information memory module is the Wide part of the system, and the deep text prompt module, deep visual prompt module and deep voice prompt module constitute the Deep part of the system. The joint loss function is the sum of the Wide part loss function and the Deep part loss function plus a bias term b, where the Wide part loss function uses the existing logistic regression loss function, and the Deep part loss function is the optimized loss function obtained in the prompt pool optimization module.

[0078] During training, the Follow-the-regularized-leader (FTRL) algorithm is used as the optimizer for the Wide part of the system, and the AdaGrad algorithm is used as the optimizer for the Deep part of the system. During training, the parts corresponding to the interview text features, interview video features, and interview audio features in the Transformer layer are not trained.

Claims

1. A multimodal depression prediction system based on topic cues, characterized in that: The system includes a personal information memory module, a prompt pool module, a topic prompt query module, a prompt pool optimization module, a text vision and voice prompt coupling module, a deep text prompt module, a deep visual prompt module, a deep voice prompt module and a joint training module; The personal information memory module performs cross-feature conversion and memory on the personal information of the interviewee, specifically: The personal information is converted into sparse binary features through one-hot encoding, and the cross-feature conversion of sparse binary features is performed through: ； To achieve, d represents the total number of sparse binary features, x i Indicates i The categories corresponding to the binary features are is a Boolean variable. i When the categories corresponding to the binary features meet the requirements, The value is 1, otherwise it is 0; The feature vector after cross feature transformation is passed through the logistic regression model Conduct training, including represents the model parameters, Represents the input data to be trained, b Representation bias, the purpose of training is to memorize personal information; The prompt pool module constructs a key-value pair composed of a key and a value as a text topic prompt of the interview text; The topic prompt query module is used to pair the interview text with the text topic prompt; The prompt pool optimization module optimizes the text topic prompts paired with the interview text and outputs the optimal text topic prompts to the text vision and voice prompt coupling module and the deep language prompt module; The text-vision and voice prompt coupling module: couples the optimal text topic prompt to the dimension of the visual modality through the text-vision coupling function to obtain the visual topic prompt; couples the optimal text topic prompt to the dimension of the voice modality through the text-voice coupling function to obtain the voice topic prompt; The deep text prompt module: performs word embedding operation on the interview text to obtain interview text features, and uses k-layer Transformer layers to encode the optimal text topic prompts and interview text features; The deep visual cue module: performs image block embedding operation on the interview video to obtain interview video features, and uses k-layer Transformer layers to encode visual topic cues and interview video features; The deep voice prompt module: performs voice embedding operation on the interview audio to obtain interview audio features, and uses k-layer Transformer layers to encode the voice topic prompts and interview audio features; The joint training module obtains a prediction result based on the input of the personal information memory module, the deep text prompt module, the deep visual prompt module and the deep voice prompt module, and then feeds the prediction result back to the loss function to jointly train the personal information memory module, the deep text prompt module, the deep visual prompt module and the deep voice prompt module until the output prediction result meets the requirements.

2. The multimodal depression level prediction system based on topic prompts according to claim 1 is characterized in that: In the personal information memory module, personal information is converted into sparse binary features through one-hot encoding, and then the sparse binary features are cross-feature transformed. The feature vectors after cross-feature transformation are trained through a logistic regression model.

3. The multimodal depression level prediction system based on topic prompts according to claim 2 is characterized in that: In the prompt pool module, each set of key-value pairs consists of a pair of corresponding keys and values, so the prompt pool is represented as: P= ,in Represents any key in the prompt pool, Representation and The corresponding value; Obtained by training the LDA model for all interview texts; yes Feature representation after Embedding operation.

4. The multimodal depression level prediction system based on topic prompts according to claim 3 is characterized in that: In the topic prompt query module, the interview text and the text topic prompt are paired. The pairing method is as follows: First, the interview text of the interviewee is input into the LDA model to obtain the interview text features. ; Then, by calculating the interview text features The distance between the key and the key in the prompt pool is used to select the N keys with the smallest distance. The key-value pairs corresponding to these N keys are the text topic prompts.

5. The multimodal depression level prediction system based on topic prompts according to claim 4 is characterized in that: In the prompt pool optimization module, the optimization method is as follows: optimize the loss function used in the joint training module to train the deep text prompt module, add the N key-value pairs selected by the topic prompt query module to the loss function, and when the loss function is minimized, the interview text features The distance between the key and the key in the prompt pool is also the smallest, thereby optimizing the text topic prompt and obtaining the optimal text topic prompt.

6. The multimodal depression level prediction system based on topic prompts according to claim 5, characterized in that: In the text-vision and voice prompt coupling module, the text-vision coupling function and the text-voice coupling function are both a linear layer.

7. The multimodal depression level prediction system based on topic prompts according to claim 6, characterized in that: The operations performed by the deep text prompt module are carried out according to the following rules: In the L1-L j In the L layer of the deep text prompt module, the optimal text topic prompt input in each layer is not transferred, but the optimal text topic prompt is re-input into each layer. j —L k In each layer, the optimal text topic prompt input from each layer is transferred between layers, that is, L j The optimal text topic prompts in the layer are encoded and then input into L j+1 layer; the optimal text topic prompt for the input L1 layer is recorded as P1, and so on. k The optimal text topic hint at the layer is denoted as P k , the final output text feature is represented as z, , The operation represented by is to project the text data to be processed into a multimodal common feature space. Indicates L k All features after layer coding, including optimal text topic prompts and interview text features.

8. The multimodal depression level prediction system based on topic prompts according to claim 7, characterized in that: The operations performed by the depth visual cue module are performed according to the following rules: In the depth visual cue module V1-V j In the layers, the visual theme cues input to each layer are not passed on, but the visual theme cues are re-input to each layer. j —V k In each layer, the visual theme prompts input by each layer are transmitted between layers, that is, V j The visual thematic cues in the layer are encoded and then input into V j+1 layer; the visual topic cue input to the V1 layer is recorded as , and so on, input V k The visual theme hint of the layer is denoted as ; The final output image feature representation x, , The operation represented by is to project the image data to be processed into a multi-modal common feature space. Indicates V k All features after layer encoding, including visual topic cues and interview video features.

9. The multimodal depression level prediction system based on topic prompts according to claim 8, characterized in that: The operation of the deep voice prompt module is carried out according to the following rules: In the A1-A of the deep voice prompt module j In the layers, the voice topic prompts input in each layer are not transmitted, but the voice topic prompts are re-input into each layer. j —A k In each layer, the speech topic prompts input from each layer are transmitted between layers, that is, A j The speech topic prompts in the layer are encoded and then input into A j+1 Layer; the voice topic prompt input A1 layer is recorded as , and so on, input A k The voice topic prompt of the layer is recorded as ; The final output audio feature representation a, , The operation represented by is to project the speech data to be processed into a multi-modal common feature space. Indicates A k All features after layer encoding, including speech topic cues and interview audio features.

10. The multimodal depression level prediction system based on topic prompts according to claim 9, characterized in that: Before the text feature z, image feature x and audio feature a are input into the joint training module, self-attention is used to assign weights of the three, and then the joint feature is input. The joint training module includes an embedding layer, a first activation layer, a second activation layer and a logistic regression layer from input to output, wherein the first activation layer adopts a ReLu activation function, the second activation layer adopts a ReLu activation function, and the logistic regression layer adopts a Sigmoid function to convert the output of the linear regression into a probability value between 0 and 1, so as to obtain a predicted score label as a prediction result.

Images