Mental health evaluation method, system and equipment based on prompt learning and storage medium
By deconstructing and reorganizing the psychological scale based on prompt learning, and combining multimodal data for mental health assessment, the problem of insufficient effectiveness and accuracy of the evaluation methods in the existing technology is solved, and efficient and accurate psychology is achieved Disease judgment and prediction reduce the impact of cost and bias.
Patent Information
- Application Number
- CN202410980019.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-22
- Publication Date
- 2024-11-12
AI Technical Summary
The existing mental health assessment methods have shortcomings in terms of effectiveness, accuracy, comprehensiveness, convenience, civilianity and privacy, which are difficult to take into account, and are costly and have limited coverage.
Using a method based on prompt learning, the psychological scale is deconstructed and reorganized, and the similarity comparison calculation is performed by combining multimodal data (images and texts), network models are constructed, and classified predictions of mental illness are reduced. Evaluation cost to improve the accuracy of evaluation.
It improves the accuracy of accurate judgment and prediction of psychological diseases, reduces the influence of scale measurement results by social expectations and personal cognitive bias, enhances the interpretability of predicted results, and reduces training parameters and time overhead.
Smart Images

Figure CN118942635A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to a mental health assessment method, system, device and storage medium, and in particular to a mental health assessment method, system, device and storage medium based on prompt learning, which belongs to the field of prompt engineering of natural language processing and the field of multimodal analysis of computer vision. Background Art
[0002] With the continuous advancement of artificial intelligence technology, the use of computer technology for multimodal detection and analysis of mental health disorders has become a research hotspot. It involves extracting and integrating information from multiple types of data sources, such as text, images, audio, and video. Natural language processing (NLP), as an important branch of artificial intelligence, focuses on enabling computers to understand and generate human language. In recent years, prompt learning, as an emerging NLP technology, has demonstrated powerful capabilities in unimodal text processing. Its core idea is to stimulate the potential knowledge of pre-trained language models (such as BERT, GPT, etc.) by designing clever prompts, so as to complete specific downstream tasks without significantly modifying the model.
[0003] At present, the mainstream methods for detecting mental health problems can be divided into three categories: 1. In the industrial world, traditional assessment methods such as mental health scales and mental health counseling are used. These scales usually include a series of standardized questionnaires designed to measure an individual's emotional state, stress level, and life satisfaction. Mental health counseling involves face-to-face communication between professionals and individuals to assess and intervene in mental health problems through conversation. 2. In the medical world, a diagnostic scheme that combines interview records, facial expressions, brain wave information, and patient self-reports for multidimensional analysis. These data are usually not open to the public and are only used by medical professionals for diagnosis and treatment. 3. In academia, sentiment analysis of social media data indirectly obtains a prediction scheme for potential psychological problems. The academic community has explored a sentiment analysis method based on social media data, which indirectly predicts potential mental health problems by analyzing the text, pictures, and videos posted by individuals on social platforms. This method uses big data analysis and machine learning technology to try to capture clues to mental health from individuals' public expressions.
[0004] In the first type of solution, the scale has a clear scoring standard, which is convenient for quantitative analysis and easy to implement, but it relies on individual self-reporting and may be affected by social expectations and personal cognitive biases. At the same time, psychological counseling has limited coverage and professional counseling services often require high fees. The second type of solution can provide a more comprehensive and accurate assessment of mental health conditions. Doctors can develop personalized treatment plans based on the patient's specific situation, but it usually requires expensive equipment and professional medical personnel, which is costly and generally only available in large medical institutions, making it difficult to popularize to all populations. The third type of solution has a wide coverage and can monitor and analyze individual social media activities in real time, but its accuracy and professionalism are questionable. These solutions have their own advantages and disadvantages in terms of effectiveness, accuracy, comprehensiveness, convenience, civilianness, and privacy, but it is difficult to balance them all. Summary of the invention
[0005] Purpose of the invention: The purpose of the present invention is to provide a psychological health assessment method, system, device and storage medium based on prompt learning that can increase assessment speed, reduce assessment costs and improve assessment accuracy.
[0006] Technical solution: The mental health assessment method based on prompt learning described in the present invention comprises:
[0007] Step 1: Collect original images and text descriptions of mental illness from public datasets and the public network, organize the data and construct the dataset, and divide it into training set, test set, and validation set;
[0008] Step 2: According to the needs of the mental illness category to be evaluated, select the corresponding standardized psychological scale, deconstruct the psychological scale, and construct the macro- and micro-classification relationship of mental illness; reorganize the scale deconstruction items based on prompt learning, and generate two-way prompts including global prompts and local prompts of mental illness based on the evaluation criteria of the reorganized scale;
[0009] Step 3: Input the bidirectional prompts and the text description of the training set into the text encoder, input the images of the training set into the image encoder, freeze the parameters of the text encoder and the image encoder, perform similarity comparison calculations, and then use weighted alignment to obtain the fusion prediction probability, build a network model, train the network model, use the trained model to predict the classification of mental illness, and combine it with the standardized psychological scale to complete the mental health assessment.
[0010] Furthermore, the step 1 specifically includes:
[0011] Step 1.1: Collect pictures of mental illness and corresponding text descriptions from public datasets and the public network to generate collected data;
[0012] Step 1.2: For images with missing text description information in the collected data, the BLIP-2 model is used to generate images and text, determine appropriate text information, and generate completed data;
[0013] Step 1.3: preprocessing the completed data, wherein the preprocessing includes removing redundant information, correcting errors and unifying the format;
[0014] Step 1.4: Use labelImg to label the preprocessed data based on the categories of mental illness, establish a mapping association between image-text pairs and the severity of mental illness, and construct a complete dataset containing images and text descriptions;
[0015] Step 1.5: Divide the complete data set into training set, test set and validation set according to the preset ratio, and keep the data distribution ratio of different categories in each data set consistent, and save the segmented data set into a data file.
[0016] Furthermore, the step 2 specifically includes:
[0017] Step 2.1: According to the needs of the mental illness category to be evaluated, select the corresponding standardized psychological scale. For the standardized psychological scale S, use the BERT model to deconstruct the standardized psychological scale S into three parts: the mental illness category Γ included in the scale = {γ1, γ2, γ3, ..., γ H}、Evaluation items of the scale and the scoring criteria of the scale Among them, γ H is the Hth category of mental illness, is the Mth evaluation item, is the Nth scoring standard in the kth evaluation item; the scale deconstruction formula is:
[0018] Γ,Φ,A=BERT(S)
[0019] Step 2.2: Assign disease category γ h The macro dimension α reflected by the evaluation criteria k Mapped to global feature F G :
[0020] F G =f r_global (γ h , α k )
[0021] Among them, f r_global () is the global mapping;
[0022] The micro dimensions included in the evaluation items Portfolio scoring criteria Reflected as local feature FL :
[0023]
[0024] Among them, f r_local () is a local mapping;
[0025] Step 2.3: Based on the paradigm in the cue learning field, we adopt the unified context UC and category-specific context CSC to combine UC with the global feature F G Combined into global prompts G :
[0026] t G =f c_global (F G , t UC )
[0027] Among them, t UC To unify the context of text embedding, f c_global () indicates a global combination;
[0028] Combine CSC with local features F L Combined into local prompts L :
[0029] f L =f c_local (F L , t CSC )
[0030] Among them, t CSC is the context-specific text embedding, f c_local () indicates a local combination.
[0031] Furthermore, the step 3 specifically includes:
[0032] Step 3.1: Set the global prompt G , local prompts L and the text description of the training set D = {d1, d2, d3, ..., d K} Input text encoder Enc T (·), the images of the training set X = {x1, x2, x3, ..., x J} Input image encoder ENc I (·), the text encoder uses transformer, the image encoder uses ResNet variant, and then freezes the parameters of the image and text encoders trained based on the pre-trained large visual language model, and converts the global prompt t G Compare the similarity with image X and make a global probability prediction:
[0033]
[0034] Among them, x represents the image, exp() is the exponential function with the natural constant e as the base, and C G represents global coarse-grained classification, <·,·> is used to calculate cosine similarity, is the global hint of the ith category, τ is the adjustment parameter;
[0035] The local prompt t L Compare the similarity with the text description D and calculate the local probability:
[0036]
[0037] Among them C L represents local fine-grained classification, is the local hint for the i-th category;
[0038] Step 3.2: Set the local probability p L Projected to the global probability p G The same dimensions as:
[0039]
[0040] Among them, f proj () is the projection function;
[0041] The final fusion probability is obtained by weighting to complete the classification prediction of mental illness:
[0042]
[0043] Where λ is the hyperparameter; combined with the standardized psychological scale, the mental health assessment is completed.
[0044] Based on the same inventive concept, the present invention also provides a mental health assessment system based on prompt learning, comprising:
[0045] The preprocessing module is used to collect original images and text descriptions of mental illness from public datasets and public networks, organize the data and construct the dataset, and divide it into training sets, test sets, and validation sets;
[0046] The prompt generation module is used to select the corresponding standardized psychological scale according to the needs of the psychological disease category to be evaluated, deconstruct the psychological scale, and construct the macro- and micro-classification relationship of the psychological disease; reorganize the scale deconstruction items based on prompt learning, and generate two-way prompts including global prompts and local prompts of the psychological disease based on the evaluation criteria of the reorganized scale;
[0047] The evaluation module is used to input the bidirectional prompts and the text description of the training set into the text encoder, input the images of the training set into the image encoder, freeze the parameters of the text encoder and the image encoder, perform similarity comparison calculations, and then use weighted alignment to obtain the fusion prediction probability, build a network model, train the network model, use the trained model to predict the classification of mental illness, and combine it with a standardized psychological scale to complete the mental health evaluation.
[0048] Furthermore, the specific implementation method of the preprocessing module is:
[0049] Collect pictures and corresponding text descriptions of mental illness from public datasets and the public network to generate collected data;
[0050] For images that lack text description information in the collected data, the BLIP-2 model is used to generate images and text, determine appropriate text information, and generate completed data;
[0051] Preprocessing the completed data, wherein the preprocessing includes removing redundant information, correcting errors, and unifying the format;
[0052] Use labelImg to label the preprocessed data based on the categories of mental illness, establish a mapping association between image-text pairs and the severity of mental illness, and construct a complete dataset containing images and text descriptions;
[0053] The complete data set is divided into training set, test set and validation set according to the preset ratio, and the data distribution ratio of different categories in each data set is kept consistent, and the segmented data sets are stored in data files.
[0054] Furthermore, the prompt generation module is specifically implemented as follows:
[0055] According to the needs of the categories of mental illnesses to be evaluated, the corresponding standardized psychological scale is selected. For the standardized psychological scale S, the BERT model is used to deconstruct the standardized psychological scale S into three parts: the categories of mental illnesses contained in the scale Γ = {γ1, γ2, γ3, ..., γ H}、Evaluation items of the scale and the scoring criteria of the scale Among them, γ H is the Hth category of mental illness, is the Mth evaluation item, is the Nth scoring standard in the kth evaluation item; the scale deconstruction formula is:
[0056] Γ,Φ,A=BERT(S)
[0057] Classify disease h The macro dimension α reflected by the evaluation criteria kMapped to global feature F G :
[0058] F G ={f r_global (γ h , α k )
[0059] Among them, f r_global () is the global mapping;
[0060] The micro dimensions included in the evaluation items Portfolio scoring criteria Reflected as local feature F L :
[0061]
[0062] Among them, f r_local () is a local mapping;
[0063] Based on the paradigm in the field of cue learning, we adopt unified context UC and category-specific context CSC to combine UC with the global feature F G Combined into global prompts G :
[0064] t G =f c_global (F G ,t UC )
[0065] Among them, t UC To unify the context of text embedding, f c_global () indicates a global combination;
[0066] Combine CSC with local features F L Combined into local prompts L :
[0067] t L =f c_local (F L , t CSC )
[0068] Among them, t CSC is the context-specific text embedding, f c_local () indicates a local combination.
[0069] Furthermore, the specific implementation method of the evaluation module is:
[0070] Set the global prompt G , local prompts L and the text description of the training set D = {d1, d2, d3, ..., d K} Input text encoder Enc T (·), the images of the training set X = {χ1, χ2, χ3, ..., χ J} Input image encoder Enc I (·), the text encoder uses transformer, the image encoder uses ResNet variant, and then freezes the parameters of the image and text encoders trained based on the pre-trained large visual language model, and converts the global prompt t G Compare the similarity with image X and make a global probability prediction:
[0071]
[0072] Among them, χ represents the image, exp() is the exponential function with the natural constant e as the base, and C G represents global coarse-grained classification, <·,·> is used to calculate cosine similarity, is the global hint of the ith category, τ is the adjustment parameter;
[0073] The local prompt t L Compare the similarity with the text description D and calculate the local probability:
[0074]
[0075] Among them C L represents local fine-grained classification, is the local hint for the i-th category;
[0076] The local probability p L Projected to the global probability p G The same dimensions as:
[0077]
[0078] Among them, f proj () is the projection function;
[0079] The final fusion probability is obtained by weighting to complete the classification prediction of mental illness:
[0080]
[0081] Where λ is the hyperparameter; combined with the standardized psychological scale, the mental health assessment is completed.
[0082] Based on the same inventive concept, the present invention also provides a computing device, comprising: one or more processors, one or more memories and one or more programs, wherein the programs are stored in the memories and configured to be executed by the processors, and when the programs are loaded into the processors, the steps of the mental health assessment method based on prompt learning as described in any one of the above items are implemented.
[0083] Based on the same inventive concept, the present invention also provides a storage medium, which stores a computer program, and the computer program includes program instructions. When the program instructions are executed by a processor, the processor executes the steps of the mental health assessment method based on prompt learning according to any of the above items.
[0084] Beneficial effects: Compared with the prior art, the method proposed in the present invention deconstructs the informal psychological scale and combines it into a formal evaluation standard based on prompt learning, analyzes and extracts the characteristics of mental illness from multiple dimensions, and uses multimodal data to improve the accurate judgment of the specified mental illness and improve the accurate prediction of the psychological condition reflected by the given data; the present invention integrates the psychological scale as the evaluation standard, which not only retains the standardization and convenience of the scale, but also reduces the subjective influence of social expectations and personal cognitive bias on the scale measurement results; the two-way prompt input mechanism integrating the specific scale proposed in the present invention is very effective in extracting While measuring the macro and micro features of the scale, the two-dimensional features of the scale are explicitly enhanced by combining learnable prompt templates; the present invention performs group similarity comparison calculations in the training stage, and compared with traditional single-modal and multi-modal data analysis, another dimension of learning is added, and the modal interpretability of the prediction results is higher; the model proposed in the present invention is based on the existing pre-trained large-scale visual language model LLVM, and prompt fine-tuning is used to replace the original fine-tuning, which greatly reduces training parameters, has low training time and space overhead, and can obtain relatively accurate disease prediction results. BRIEF DESCRIPTION OF THE DRAWINGS
[0085] Figure 1 is a flow chart of a method according to an embodiment of the present invention;
[0086] Figure 2 is a flowchart of the implementation steps of an embodiment of the present invention;
[0087] Figure 3 A schematic diagram of deconstructing a psychological scale according to an embodiment of the present invention;
[0088] Figure 4 A schematic diagram of generating a bidirectional prompt according to an embodiment of the present invention;
[0089] Figure 5 Schematic diagram of group similarity comparison calculation and prediction according to an embodiment of the present invention. DETAILED DESCRIPTION
[0090] In order to make the purpose, technical scheme and advantages of the present invention clearer, the present invention will be further described in detail below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention. It should be understood that the specific embodiments described here are only used to explain the present invention and are not used to limit the present invention.
[0091] As attached Figure 1 As shown, the mental health assessment method based on prompt learning in this embodiment includes:
[0092] Step 1: Collect original images and text descriptions of mental illness from public datasets and the public network, organize the data and construct the dataset, and divide it into training set, test set, and validation set;
[0093] Step 2: According to the needs of the mental illness category to be evaluated, select the corresponding standardized psychological scale, deconstruct the psychological scale, and construct the macro- and micro-classification relationship of mental illness; reorganize the scale deconstruction items based on prompt learning, and generate two-way prompts including global prompts and local prompts of mental illness based on the evaluation criteria of the reorganized scale;
[0094] Step 3: Input the bidirectional prompts and the text description of the training set into the text encoder, input the images of the training set into the image encoder, freeze the parameters of the text encoder and the image encoder, perform similarity comparison calculations, and then use weighted alignment to obtain the fusion prediction probability, build a network model, train the network model, use the trained model to predict the classification of mental illness, and combine it with the standardized psychological scale to complete the mental health assessment.
[0095] Furthermore, if Figure 2 As shown, the psychological disease dataset described in step 1 is constructed, and the specific process includes:
[0096] Step 1-1, collect images and text descriptions of various mental illnesses. These data come from public mental health datasets such as AEVC2017 (DAIC-WOZ), CMDC; Internet and social media such as VCG Visual China, VEER Gallery, Tuchong Creative; offline interviews, etc.
[0097] Step 1-2: For images that lack text description information, the BLIP-2 model is used to generate text and images to determine appropriate text information;
[0098] Steps 1-3: preprocess the completed data, including removing redundant information, correcting errors, and unifying the format;
[0099] Step 1.4: label the sorted image-text data using labelImg based on specific mental illnesses (such as depression, anxiety, and insomnia) to establish a mapping relationship between the image-text pairs and the severity of mental illnesses;
[0100] Step 1.5, divide the labeled data set into training set, test set and validation set in a ratio of 7:2:1, and make the data distribution of different categories in different data subsets consistent, and then store the segmented data into a data file.
[0101] Using the solution of this embodiment, we constructed a psychological disease dataset through public mental health datasets, the Internet and social media, offline interviews, etc. The dataset contains 8550 sets of labeled image-text pairs. Using real-world datasets, the performance of the method can be fully tested.
[0102] Furthermore, the bidirectional prompt generation in step 2 specifically includes:
[0103] Step 2-1, according to the needs of the psychological disease category to be evaluated, select the corresponding standardized psychological scale, and multiple psychological disease categories can correspond to the same psychological scale; this embodiment is for depression, anxiety, and insomnia.
[0104] A psychological scale is a psychological measurement tool used to assess an individual's psychological characteristics, abilities, emotional states or behavioral patterns. They usually include a series of standardized questions or test items to quantify specific psychological variables through individual answers or performance; psychological scales are widely used in psychological assessment, clinical diagnosis, educational assessment, career planning and research; and some scales may contain multiple mental illnesses. This model focuses on psychological scales for detecting common mental illnesses (such as depression, anxiety, insomnia, etc.), and uses a single scale deconstruction, such as the Shell Depression Inventory (BDI), the Hamilton Anxiety Rating Scale (HAMA), the Athens Insomnia Scale (AIS), etc. These classic scales have been widely used in all walks of life, and this article organizes and summarizes them through search engines, large model questions and answers, etc. Since many mental illnesses (such as obsessive-compulsive disorder, bipolar disorder, PTSD, personality disorders, etc.) are difficult to identify and predict only through pictures and texts or require joint inference, after comprehensive consideration, the model of this embodiment selects depression, anxiety, insomnia and other diseases with relatively clear symptoms, and the selection of scales is also based on this.
[0105] The BERT model is used to deconstruct a specific psychological scale S into three parts: the psychological symptoms contained in the scale Γ = {γ1, γ2, γ3, ..., γ H}、Evaluation items of the scale and the scoring criteria of the scale Where k represents the kth evaluation item. The scale deconstruction formula is:
[0106] Γ,Φ,A=BERT(S)
[0107] Step 2-2, converting the structured scale sub-data (such as a tree structure) into a data type that can be recognized by the encoder;
[0108] Classify disease h The macro dimension α reflected by the evaluation criteria k Mapped to global feature F G :
[0109] F G =f r_global (γ h , α k )
[0110] The micro dimensions included in the evaluation items Combined evaluation criteria Reflected as local feature F L :
[0111]
[0112] Step 2-3, based on the existing paradigm in the field of cue learning, unified context (UC) and class-specific context (CSC) are used, and the original cat method is used to combine UC with the global feature F G Combined into global prompts G :
[0113] t G =f c_global (F G , t UC )
[0114] The sequential structure of cat is fine-tuned to make the head-to-tail connection become embedded according to the specified position, and CSC is combined with the local feature F L Combined into local prompts L :
[0115] t L =f c_local (F L , t CSC )
[0116] Then feed the bidirectional prompt into the text encoder Enc T (·), aligned with other modality data.
[0117] Figure 3 That is the schematic diagram of the deconstruction of the psychological scale in this embodiment. The psychological scale is deconstructed into three-dimensional data using the BERT model in the NLP field. The data between different dimensions are organically combined by manual adjustment to construct macro and micro features, and finally the unstructured psychological scale is converted into structured vector data.
[0118] Figure 4 That is the schematic diagram of the bidirectional prompt generation in this embodiment, which combines the dual features obtained in step 2-2 with the specific context prompt template in prompt learning to generate prompts representing the corresponding features, and inputs the text encoder for alignment analysis with other modal data.
[0119] Furthermore, the similarity comparison calculation and prediction described in step 3 specifically includes the following process:
[0120] Step 3-1: Change the above two-way prompt G With t L , text description D = {d1, d2, d3, ..., d K} Input text encoder Enc T (·), the image X = {x1, x2, x3, ..., x J} Input image encoder Enc I (·), the text encoder uses transformer, the image encoder uses ResNet variants, such as ResNet-50, ResNet-101, etc., freeze the parameters of the image and text encoders, and set the global prompt t G Compare the similarity with image X and make a global probability prediction:
[0121]
[0122] Where x represents the image, C G represents the global coarse-grained classification, <·,·> is used to calculate the cosine similarity, and τ is the adjustment parameter. Similarly, the local hint t L Compare the similarity with the text description D and calculate the local probability:
[0123]
[0124] Among them C L Represents local fine-grained classification;
[0125] Step 3-2: The local probability P L Projected to the global probability p G The same dimensions as:
[0126]
[0127] Then the final fusion probability is obtained by weighted method:
[0128]
[0129] where λ is the hyperparameter.
[0130] Figure 5 This is a schematic diagram of group similarity comparison calculation and prediction in this embodiment. The multimodal psychological disease data obtained in step 1 and the two-way prompts obtained in step 2 are grouped for similarity comparison calculation, and the calculated prediction probabilities are fused to finally obtain the prediction results for specific mental illnesses.
[0131] Based on the same inventive concept, this embodiment also provides a mental health assessment system based on prompt learning, including:
[0132] The preprocessing module is used to collect original images and text descriptions of mental illness from public datasets and public networks, organize the data and construct the dataset, and divide it into training sets, test sets, and validation sets;
[0133] The prompt generation module is used to select the corresponding standardized psychological scale according to the needs of the psychological disease category to be evaluated, deconstruct the psychological scale, and construct the macro- and micro-classification relationship of the psychological disease; reorganize the scale deconstruction items based on prompt learning, and generate two-way prompts including global prompts and local prompts of the psychological disease based on the evaluation criteria of the reorganized scale;
[0134] The evaluation module is used to input the bidirectional prompts and the text description of the training set into the text encoder, input the images of the training set into the image encoder, freeze the parameters of the text encoder and the image encoder, perform similarity comparison calculations, and then use weighted alignment to obtain the fusion prediction probability, build a network model, train the network model, use the trained model to predict the classification of mental illness, and combine it with a standardized psychological scale to complete the mental health evaluation.
[0135] Furthermore, the specific implementation method of the preprocessing module is:
[0136] Collect pictures of mental illness and corresponding text descriptions from public datasets and the public network to generate collected data;
[0137] For images that lack text description information in the collected data, the BLIP-2 model is used to generate images and text, determine appropriate text information, and generate completed data;
[0138] Preprocessing the completed data, wherein the preprocessing includes removing redundant information, correcting errors, and unifying the format;
[0139] Use labelImg to label the preprocessed data based on the categories of mental illness, establish a mapping association between image-text pairs and the severity of mental illness, and construct a complete dataset containing images and text descriptions;
[0140] The complete data set is divided into training set, test set and validation set according to the preset ratio, and the data distribution ratio of different categories in each data set is kept consistent, and the segmented data sets are stored in data files.
[0141] Furthermore, the prompt generation module is specifically implemented as follows:
[0142] According to the needs of the categories of mental illnesses to be evaluated, the corresponding standardized psychological scale is selected. For the standardized psychological scale S, the BERT model is used to deconstruct the standardized psychological scale S into three parts: the categories of mental illnesses contained in the scale Γ = {γ1, γ2, γ3, ..., γ H}、Evaluation items of the scale and the scoring criteria of the scale Among them, γ H is the Hth category of mental illness, is the Mth evaluation item, is the Nth scoring standard in the kth evaluation item; the scale deconstruction formula is:
[0143] Γ,Φ,A=BERT(S)
[0144] Classify disease h The macro dimension α reflected by the evaluation criteria k Mapped to global feature F G :
[0145] F G =f r_global (γ h , α k )
[0146] Among them, f r_global () is the global mapping;
[0147] The micro dimensions included in the evaluation items Portfolio scoring criteria It is reflected as local feature F:
[0148]
[0149] Among them, f r_local () is a local mapping;
[0150] Based on the paradigm in the field of cue learning, we adopt unified context UC and category-specific context CSC to combine UC with the global feature F G Combined into global promptsG :
[0151] t G =f c_global (F G , t UC )
[0152] Among them, t UC To unify the context of text embedding, f c_global () indicates a global combination;
[0153] Combine CSC with local features F L Combined into local prompts L :
[0154] t L =f c_local (F L , t CSC )
[0155] Among them, t CSC is the context-specific text embedding, t c_local () indicates a local combination.
[0156] Furthermore, the specific implementation method of the evaluation module is:
[0157] Set the global prompt G , local prompts L and the text description of the training set D = {d1, d2, d3, ..., d K} Input text encoder Enc T (·), the images of the training set X = {x1, x2, x3, ..., x J} Input image encoder Enc I (·), the text encoder uses transformer, and the image encoder uses ResNet variants, such as ResNet-50, ResNet-101, etc., then freeze the parameters of the image and text encoders based on the pre-trained large visual language model, and convert the global prompt t G Compare the similarity with image X and make a global probability prediction:
[0158]
[0159] Among them, x represents the image, exp() is the exponential function with the natural constant e as the base, and C G represents the global coarse-grained classification, <·,> is used to calculate the cosine similarity, is the global hint of the ith category, τ is the adjustment parameter;
[0160] The local prompt t LCompare the similarity with the text description D and calculate the local probability:
[0161]
[0162] Among them C L represents local fine-grained classification, is the local hint for the i-th category;
[0163] The local probability p L Projected to the global probability p G The same dimensions as:
[0164]
[0165] Among them, f proj () is the projection function;
[0166] The final fusion probability is obtained by weighting to complete the classification prediction of mental illness:
[0167]
[0168] Where λ is the hyperparameter; combined with the standardized psychological scale, the mental health assessment is completed.
[0169] Based on the same inventive concept, this embodiment also provides a computing device, including: one or more processors, one or more memories and one or more programs, wherein the programs are stored in the memories and configured to be executed by the processors, and when the programs are loaded into the processors, the steps of the mental health assessment method based on prompt learning as described in any one of the above items are implemented.
[0170] Based on the same inventive concept, this embodiment also provides a storage medium, which stores a computer program, and the computer program includes program instructions. When the program instructions are executed by a processor, the processor executes the steps of the mental health assessment method based on prompt learning according to any of the above items.
Claims
1. A mental health assessment method based on prompt learning, characterized in that: include: Step 1: Collect original images and text descriptions of mental illness from public datasets and the public network, organize the data and construct the dataset, and divide it into training set, test set, and validation set; Step 2: According to the needs of the mental illness category to be evaluated, select the corresponding standardized psychological scale, deconstruct the psychological scale, and construct the macro- and micro-classification relationship of mental illness; reorganize the scale deconstruction items based on prompt learning, and generate two-way prompts including global prompts and local prompts of mental illness based on the evaluation criteria of the reorganized scale; Step 3: Input the bidirectional prompts and the text description of the training set into the text encoder, input the images of the training set into the image encoder, freeze the parameters of the text encoder and the image encoder, perform similarity comparison calculations, and then use weighted alignment to obtain the fusion prediction probability, build a network model, train the network model, use the trained model to predict the classification of mental illness, and combine it with the standardized psychological scale to complete the mental health assessment.
2. The mental health assessment method based on prompt learning according to claim 1 is characterized in that: The step 1 specifically includes: Step 1.1: Collect pictures of mental illness and corresponding text descriptions from public datasets and the public network to generate collected data; Step 1.2: For images with missing text description information in the collected data, the BLIP-2 model is used to generate images and text, determine appropriate text information, and generate completed data; Step 1.3: preprocessing the completed data, wherein the preprocessing includes removing redundant information, correcting errors and unifying the format; Step 1.4: Use labelImg to label the preprocessed data based on the categories of mental illness, establish a mapping association between image-text pairs and the severity of mental illness, and construct a complete dataset containing images and text descriptions; Step 1.5: Divide the complete data set into training set, test set and validation set according to the preset ratio, and keep the data distribution ratio of different categories in each data set consistent, and save the segmented data set into a data file.
3. The mental health assessment method based on prompt learning according to claim 1 is characterized in that: The step 2 specifically includes: Step 2.1: According to the needs of the mental illness category to be evaluated, select the corresponding standardized psychological scale. For the standardized psychological scale S, use the BERT model to deconstruct the standardized psychological scale S into three parts: the mental illness category Γ included in the scale = {γ1, γ2, γ3, ..., γ H }、Evaluation items of the scale and the scoring criteria of the scale Among them, γ H is the Hth category of mental illness, is the Mth evaluation item, is the Nth scoring standard in the kth evaluation item; the scale deconstruction formula is: Γ,Φ,A=BERT(S) Step 2.2: Assign disease category γ h The macro dimension α reflected by the evaluation criteria k Mapped to global feature F G : F G =f r_global (c h ,a k ) Among them, f r_global () is the global mapping; The micro dimensions included in the evaluation items Portfolio scoring criteria Reflected as local feature F L : Among them, f r_local () is a local mapping; Step 2.3: Based on the paradigm in the cue learning field, we adopt the unified context UC and category-specific context CSC to combine UC with the global feature F G Combined into global prompts G : t G =f c_global (F G ,t UC ) Among them, t UC To unify the context of text embedding, f c_glocal () indicates a global combination; Combine CSC with local features F L Combined into local prompts L : t L =f c_local (F L ,t CSC ) Among them, t CSC is the context-specific text embedding, f c_local () indicates a local combination.
4. The mental health assessment method based on prompt learning according to claim 1 is characterized in that: The step 3 specifically includes: Step 3.1: Set the global prompt G , local prompts L and the text description of the training set D = {d1, d2, d3, ...d K } Input text encoder Enc T (·), the images of the training set X = {x1, x2, x3, ..., x J } Input image encoder Enc I (·), the text encoder uses transformer, the image encoder uses ResNet variant, and then freezes the parameters of the image and text encoders trained based on the pre-trained large visual language model, and converts the global prompt t G Compare the similarity with image X and make a global probability prediction: Among them, x represents the image, exp() is the exponential function with the natural constant e as the base, and C G represents global coarse-grained classification, <·,·> is used to calculate cosine similarity, is the global hint of the ith category, τ is the adjustment parameter; The local prompt t L Compare the similarity with the text description D and calculate the local probability: Among them C L represents local fine-grained classification, is the local hint for the i-th category; Step 3.2: Set the local probability p L Projected to the global probability p G The same dimensions as: Among them, f proj () is the projection function; The final fusion probability is obtained by weighting to complete the classification prediction of mental illness: Where λ is the hyperparameter; combined with the standardized psychological scale, the mental health assessment is completed.
5. A mental health assessment system based on prompt learning, characterized in that: include: The preprocessing module is used to collect original images and text descriptions of mental illness from public datasets and public networks, organize the data and construct the dataset, and divide it into training sets, test sets, and validation sets; The prompt generation module is used to select the corresponding standardized psychological scale according to the needs of the psychological disease category to be evaluated, deconstruct the psychological scale, and construct the macro- and micro-classification relationship of the psychological disease; reorganize the scale deconstruction items based on prompt learning, and generate two-way prompts including global prompts and local prompts of the psychological disease based on the evaluation criteria of the reorganized scale; The evaluation module is used to input the bidirectional prompts and the text description of the training set into the text encoder, input the images of the training set into the image encoder, freeze the parameters of the text encoder and the image encoder, perform similarity comparison calculations, and then use weighted alignment to obtain the fusion prediction probability, build a network model, train the network model, use the trained model to predict the classification of mental illness, and combine it with a standardized psychological scale to complete the mental health evaluation.
6. The mental health assessment system based on prompt learning according to claim 5 is characterized in that: The specific implementation method of the preprocessing module is: Collect pictures and corresponding text descriptions of mental illness from public datasets and the public network to generate collected data; For images that lack text description information in the collected data, the BLIP-2 model is used to generate images and text, determine appropriate text information, and generate completed data; Preprocessing the completed data, wherein the preprocessing includes removing redundant information, correcting errors, and unifying the format; Use labelImg to label the preprocessed data based on the categories of mental illness, establish a mapping association between image-text pairs and the severity of mental illness, and construct a complete dataset containing images and text descriptions; The complete data set is divided into training set, test set and validation set according to the preset ratio, and the data distribution ratio of different categories in each data set is kept consistent, and the segmented data sets are stored in data files.
7. The mental health assessment system based on prompt learning according to claim 5 is characterized in that: The specific implementation method of the prompt generation module is: According to the needs of the categories of mental illnesses to be evaluated, the corresponding standardized psychological scale is selected. For the standardized psychological scale S, the BERT model is used to deconstruct the standardized psychological scale S into three parts: the categories of mental illnesses contained in the scale Γ = {γ1, γ2, γ3, ...γ H }、Evaluation items of the scale and the scoring criteria of the scale Among them, γ H is the Hth category of mental illness, is the Mth evaluation item, is the Nth scoring standard in the kth evaluation item; the scale deconstruction formula is: Γ,Φ,A=BERT(S) Classify disease h The macro dimension α reflected by the evaluation criteria k Mapped to global feature F G : F G =f r_global (c h ,a k ) Among them, f r_global () is the global mapping; The micro dimensions included in the evaluation items Portfolio scoring criteria Reflected as local feature F L : Among them, f r_local () is a local mapping; Based on the paradigm in the field of cue learning, we adopt unified context UC and category-specific context CSC to combine UC with the global feature F G Combined into global prompts G : t G =f c_global (F G ,t UC ) Among them, t UC To unify the context of text embedding, f c_local () indicates a global combination; Combine CSC with local features F L Combined into local prompts L : t L =f c_local (F L ,t CSC ) Among them, t CSC is the context-specific text embedding, f c_local () indicates a local combination.
8. The mental health assessment system based on prompt learning according to claim 5 is characterized in that: The specific implementation method of the evaluation module is: Set the global prompt G , local prompts L and the text description of the training set D = {d1, d2, d3, ..., d K } Input text encoder Enc T (·), the images of the training set X = {x1, x2, x3, ..., x J } Input image encoder Enc I (·), the text encoder uses transformer, the image encoder uses ResNet variant, and then freezes the parameters of the image and text encoders trained based on the pre-trained large visual language model, and converts the global prompt t G Compare the similarity with image X and make a global probability prediction: Where x represents the image, e×p() is an exponential function with the natural constant e as the base, and C G represents global coarse-grained classification, <·,·> is used to calculate cosine similarity, is the global hint of the ith category, τ is the adjustment parameter; The local prompt t L Compare the similarity with the text description D and calculate the local probability: Among them C L represents local fine-grained classification, is the local hint for the i-th category; The local probability p L Projected to the global probability p G The same dimensions as: Among them, p proj () is the projection function; The final fusion probability is obtained by weighting to complete the classification prediction of mental illness: Where λ is the hyperparameter; combined with the standardized psychological scale, the mental health assessment is completed.
9. A computing device, characterized in that include: One or more processors, one or more memories and one or more programs, wherein the programs are stored in the memories and configured to be executed by the processors, and when the programs are loaded into the processors, the steps of the mental health assessment method based on prompt learning according to any one of claims 1 to 4 are implemented.
10. A storage medium, characterized in that: The storage medium stores a computer program, which includes program instructions. When the program instructions are executed by a processor, the processor executes the steps of the mental health assessment method based on prompt learning according to any one of claims 1 to 4.
