Questionnaire-based psychological assessment method and device, electronic equipment and storage medium

Abstract

The embodiment of the invention provides a psychological assessment method and device based on a questionnaire, electronic equipment and a storage medium, and belongs to the technical field of artificial intelligence and medical treatment. The method comprises the following steps: acquiring an initial reply video of replying a general psychological assessment questionnaire by a target object; performing preliminary psychological assessment on the initial reply video to obtain a psychological health category; obtaining a target psychological assessment questionnaire based on the psychological health category; obtaining a target reply video of replying the target psychological assessment questionnaire by the target object; extracting a target expression embedding feature and a target voice embedding feature based on the target reply video; obtaining target questionnaire reply content from the target reply video; performing psychological assessment based on the target expression embedded feature, the target voice embedded feature and the target questionnaire reply content to obtain a target psychological assessment state; wherein the target psychological assessment state is used for representing the psychological health state of the target object. The embodiment of the invention can improve the accuracy of psychological assessment.

Description

Technical Field

[0001] The present application relates to the fields of artificial intelligence and medical technology, and in particular to a questionnaire-based psychological assessment method and device, electronic equipment, and storage medium. Background Art

[0002] In the medical field, psychological assessment is a process of comprehensively analyzing and evaluating a person's psychological state through psychological means. At present, psychological assessment methods mainly rely on psychological assessment questionnaires, which evaluate the psychological state based on the customer's responses to the questionnaire. However, in actual application scenarios, the responses to the questionnaire cannot accurately reflect the user's true psychological state, thus affecting the accuracy of psychological assessment.

[0003] Therefore, how to improve the accuracy of psychological assessment has become a technical problem that needs to be solved urgently. Summary of the invention

[0004] The main purpose of the embodiments of the present application is to propose a questionnaire-based psychological assessment method and device, electronic device and storage medium, aiming to improve the accuracy of recommendations.

[0005] To achieve the above purpose, a first aspect of an embodiment of the present application proposes a psychological assessment method based on a questionnaire, the method comprising:

[0006] Obtaining an initial response video of the target subject responding to a preset universal psychological assessment questionnaire; wherein the universal psychological assessment questionnaire includes open-ended questions;

[0007] Conducting a preliminary psychological assessment on the initial response video to obtain a mental health category;

[0008] Obtaining a target psychological assessment questionnaire based on the mental health category; wherein the target psychological assessment questionnaire includes closed questions;

[0009] Obtaining a target response video of the target subject responding to the target psychological assessment questionnaire;

[0010] Extracting facial expression features based on the target reply video to obtain target facial expression embedding features;

[0011] Extract speech features based on the target reply video to obtain target speech embedding features;

[0012] Obtaining target questionnaire response content from the target response video;

[0013] A psychological assessment is performed based on the target expression embedding features, the target speech embedding features and the target questionnaire answer content to obtain a target psychological assessment state; wherein the target psychological assessment state is used to characterize the mental health state of the target object.

[0014] In some embodiments, the performing of a preliminary psychological assessment on the initial response video to obtain a mental health category includes:

[0015] Extracting facial expression features from the initial reply video to obtain initial facial expression embedding features;

[0016] Performing audio extraction on the initial reply video to obtain initial reply voice data;

[0017] Converting the initial reply voice data into text to obtain original reply text data;

[0018] Extracting semantic features from the original reply text data to obtain initial semantic embedding features;

[0019] Extracting speech features from the initial reply speech data to obtain initial speech embedding features;

[0020] Performing feature concatenation on the initial expression embedding feature, the initial semantic embedding feature and the initial speech embedding feature to obtain an initial multimodal psychological embedding feature;

[0021] The initial multimodal psychological embedding features are preliminarily evaluated to obtain the mental health category.

[0022] In some embodiments, extracting expression features from the initial reply video to obtain initial expression embedding features includes:

[0023] Extracting video frames from the initial response video to obtain an initial video frame image;

[0024] Performing image preprocessing on the initial video frame image to obtain an initial face video image;

[0025] Micro-expressions are extracted from the initial face video image to obtain the initial expression embedding features.

[0026] In some embodiments, the extracting semantic features from the original reply text data to obtain initial semantic embedding features includes:

[0027] Performing text cleaning on the original reply text data to obtain initial reply text data;

[0028] Performing word segmentation processing on the initial reply text data to obtain an initial reply word segmentation sequence;

[0029] Performing text encoding processing on the initial reply word segmentation sequence to obtain initial text encoding features;

[0030] Perform semantic prediction on the initial text encoding features to obtain the initial semantic embedding features.

[0031] In some embodiments, extracting speech features from the initial reply speech data to obtain initial speech embedding features includes:

[0032] Performing spectrum conversion on the initial reply voice data to obtain initial voice spectrum data;

[0033] Performing encoding processing on the initial speech spectrum data to obtain initial speech encoding features;

[0034] Performing attention calculation on the initial speech coding feature to obtain an initial speech attention feature;

[0035] The initial speech attention feature is pooled to obtain the initial speech embedding feature.

[0036] In some embodiments, performing a psychological assessment based on the target expression embedding feature, the target voice embedding feature, and the target questionnaire answer content to obtain a target psychological assessment state includes:

[0037] Performing response evaluation on the target questionnaire response content to obtain a target response evaluation score;

[0038] Extracting semantic features from the target questionnaire response content to obtain target semantic embedding features;

[0039] Performing feature concatenation on the target expression embedding feature, the target semantic embedding feature and the target speech embedding feature to obtain a target multimodal psychological embedding feature;

[0040] A psychological assessment is performed on the target multimodal psychological embedding features and the target response evaluation score to obtain the target psychological assessment state.

[0041] In some embodiments, obtaining a target psychological assessment questionnaire based on the mental health category includes:

[0042] Acquire a plurality of preset candidate closed-ended assessment questions; wherein each of the candidate closed-ended assessment questions contains a psychological state attribute;

[0043] Matching the mental state attributes of a plurality of the candidate closed-ended assessment questions based on the mental health category to obtain a target mental assessment question;

[0044] The target psychological assessment questionnaire is generated based on the target psychological assessment questions.

[0045] To achieve the above-mentioned purpose, a second aspect of the embodiment of the present application proposes a psychological assessment device based on a questionnaire, the device comprising:

[0046] An initial response data acquisition module is used to acquire an initial response video of a target subject responding to a preset general psychological assessment questionnaire; wherein the general psychological assessment questionnaire includes open-ended questions;

[0047] A preliminary psychological assessment module, used for conducting a preliminary psychological assessment on the initial response video to obtain a mental health category;

[0048] An assessment questionnaire generation module, used to obtain a target psychological assessment questionnaire based on the mental health category; wherein the target psychological assessment questionnaire includes closed questions;

[0049] A target response data acquisition module, used to acquire a target response video of the target subject responding to the target psychological assessment questionnaire;

[0050] A target expression feature extraction module is used to extract expression features based on the target reply video to obtain target expression embedding features;

[0051] A target speech feature extraction module, used to extract speech features based on the target reply video to obtain target speech embedding features;

[0052] A target questionnaire answer content acquisition module, used to acquire the target questionnaire answer content from the target answer video;

[0053] The target psychological assessment module is used to perform a psychological assessment based on the target expression embedding features, the target voice embedding features and the target questionnaire answer content to obtain a target psychological assessment state; wherein the target psychological assessment state is used to characterize the mental health state of the target object.

[0054] To achieve the above-mentioned purpose, the third aspect of an embodiment of the present application proposes an electronic device, which includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the method described in the first aspect is implemented.

[0055] To achieve the above-mentioned purpose, the fourth aspect of an embodiment of the present application proposes a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method described in the first aspect is implemented.

[0056] The questionnaire-based psychological assessment method and device, electronic device and storage medium proposed in the present application first obtain an initial response video of a target object responding to a preset general psychological assessment questionnaire; wherein the general psychological assessment questionnaire includes open questions; a preliminary psychological assessment is performed on the initial response video to obtain a mental health category, and a target psychological assessment questionnaire is obtained based on the mental health category; wherein the target psychological assessment questionnaire includes closed questions; thereby the psychological state of the target object is evaluated more deeply; further, a target response video of the target object responding to the target psychological assessment questionnaire is obtained; expression features are extracted based on the target response video to obtain a target expression embedding feature; voice features are extracted based on the target response video to obtain a target voice embedding feature; the target questionnaire response content is obtained from the target response video; psychological assessment is performed based on the target expression embedding feature, the target voice embedding feature and the target questionnaire response content, thereby obtaining a target psychological assessment state that can characterize the mental health state of the target object; wherein the target psychological assessment state is used to characterize the mental health state of the target object. By using two psychological assessment questionnaires for psychological assessment, combined with the multimodal data of the target object when responding to the psychological assessment questionnaire, the psychological state of the target object can be evaluated more comprehensively and accurately, and the accuracy, effectiveness and reliability of the psychological assessment can be improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] Figure 1 is a flowchart of a questionnaire-based psychological assessment method provided in an embodiment of the present application;

[0058] Figure 2 yes Figure 1 Flow chart of step S102 in FIG.

[0059] Figure 3 yes Figure 2 Flow chart of step S201 in FIG.

[0060] Figure 4 yes Figure 2 Flow chart of step S204 in FIG.

[0061] Figure 5 yes Figure 2 Flow chart of step S205 in FIG.

[0062] Figure 6 yes Figure 1 Flow chart of step S103 in FIG.

[0063] Figure 7 yes Figure 1 Flow chart of step S108 in FIG.

[0064] Figure 8 is a schematic diagram of the structure of a questionnaire-based psychological assessment device provided in an embodiment of the present application;

[0065] Fig. 9 It is a schematic diagram of the hardware structure of the electronic device provided in the embodiment of the present application. DETAILED DESCRIPTION

[0066] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0067] It should be noted that, although the functional modules are divided in the device schematic diagram and the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first", "second", etc. in the specification, claims and the above drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0068] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0069] First, some nouns involved in this application are analyzed:

[0070] Artificial intelligence (AI) is a new technical science that studies and develops theories, methods, technologies and application systems for simulating, extending and expanding human intelligence. AI is a branch of computer science that attempts to understand the essence of intelligence and produce a new type of intelligent machine that can respond in a similar way to human intelligence. Research in this field includes robotics, language recognition, image recognition, natural language processing and expert systems. AI can simulate the information process of human consciousness and thinking.

[0071] Natural language processing (NLP): NLP uses computers to process, understand and apply human languages ​​(such as Chinese, English, etc.). NLP is a branch of artificial intelligence and an interdisciplinary subject between computer science and linguistics. It is often referred to as computational linguistics. Natural language processing includes grammatical analysis, semantic analysis, and text understanding. Natural language processing is often used in technical fields such as machine translation, handwritten and printed character recognition, speech recognition and text-to-speech conversion, information intent recognition, information extraction and filtering, text classification and clustering, public opinion analysis and opinion mining. It involves data mining related to language processing, machine learning, knowledge acquisition, knowledge engineering, artificial intelligence research, and linguistic research related to language computing.

[0072] Psychological assessment refers to the general term for comprehensive, systematic and in-depth analysis of individual or group psychological phenomena using a variety of methods such as conversation, observation and testing under the joint guidance of biology, psychology, social science and medical models. The psychological assessment process is designed to help evaluate behavior, cognitive ability, personality traits and characteristics of individuals or groups, and then provide scientific basis and decision-making support for psychological counseling, treatment, education or management.

[0073] At present, psychological assessment methods mainly rely on psychological assessment questionnaires, which evaluate the psychological state of customers based on their responses to the questionnaires. However, in actual application scenarios, the responses to the questionnaires cannot accurately reflect the user's true psychological state, thus affecting the accuracy of psychological assessments.

[0074] Based on this, the embodiments of the present application provide a questionnaire-based psychological assessment method and device, an electronic device, and a storage medium, aiming to improve the accuracy of psychological assessment.

[0075] The questionnaire-based psychological assessment method and device, electronic device and storage medium provided in the embodiments of the present application are specifically illustrated through the following embodiments. First, the questionnaire-based psychological assessment method in the embodiments of the present application is described.

[0076] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Among them, artificial intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0077] AI basic technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics, etc. AI software technologies mainly include computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0078] The questionnaire-based psychological assessment method provided in the embodiment of the present application relates to the field of artificial intelligence and medical technology. The questionnaire-based psychological assessment method provided in the embodiment of the present application can be applied to a terminal, can be applied to a server side, or can be software running in a terminal or a server side. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, etc.; the server side can be configured as an independent physical server, or a server cluster or a distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms; the software can be an application that implements a questionnaire-based psychological assessment method, etc., but is not limited to the above forms.

[0079] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0080] It should be noted that in each specific implementation of the present application, when it comes to the need to perform relevant processing based on data related to user identity or characteristics such as user information, user behavior data, user historical data, and user location information, the user's permission or consent will be obtained first, and the collection, use, and processing of these data will comply with relevant laws, regulations, and standards. In addition, when the embodiment of the present application needs to obtain the user's sensitive personal information, the user's separate permission or consent will be obtained through a pop-up window or by jumping to a confirmation page. After clearly obtaining the user's separate permission or consent, the necessary user-related data for the normal operation of the embodiment of the present application will be obtained.

[0081] Figure 1 is an optional flowchart of the questionnaire-based psychological assessment method provided in the embodiment of the present application. Figure 1 The method may include but is not limited to steps S101 to S108.

[0082] Step S101, obtaining an initial response video of the target subject responding to a preset general psychological assessment questionnaire; wherein the general psychological assessment questionnaire includes open-ended questions;

[0083] Step S102, performing a preliminary psychological assessment on the initial response video to obtain a mental health category;

[0084] Step S103, obtaining a target psychological assessment questionnaire based on the psychological health category; wherein the target psychological assessment questionnaire includes closed questions;

[0085] Step S104, obtaining a target response video of the target subject responding to the target psychological assessment questionnaire;

[0086] Step S105, extracting expression features based on the target reply video to obtain target expression embedding features;

[0087] Step S106, extracting speech features based on the target reply video to obtain target speech embedding features;

[0088] Step S107, obtaining the target questionnaire answer content from the target answer video;

[0089] Step S108, performing a psychological assessment based on the target expression embedding features, the target voice embedding features and the target questionnaire answer content to obtain the target psychological assessment state; wherein the target psychological assessment state is used to characterize the mental health state of the target object.

[0090] Steps S101 to S108 shown in the embodiment of the present application, first, obtain an initial response video of the target object responding to a preset universal psychological assessment questionnaire; wherein the universal psychological assessment questionnaire includes open questions; conduct a preliminary psychological assessment on the initial response video to obtain a mental health category, and obtain a target psychological assessment questionnaire based on the mental health category; wherein the target psychological assessment questionnaire includes closed questions; further, obtain a target response video of the target object responding to the target psychological assessment questionnaire; perform expression feature extraction based on the target response video to obtain a target expression embedding feature; perform voice feature extraction based on the target response video to obtain a target voice embedding feature; obtain the target questionnaire response content from the target response video; perform a psychological assessment based on the target expression embedding feature, the target voice embedding feature and the target questionnaire response content, thereby obtaining a target psychological assessment state that can characterize the mental health state of the target object; wherein the target psychological assessment state is used to characterize the mental health state of the target object. By using two psychological assessment questionnaires for psychological assessment, combined with the multimodal data of the target object when responding to the psychological assessment questionnaire, the psychological state of the target object can be evaluated more comprehensively and accurately, and the accuracy, effectiveness and reliability of the psychological assessment can be improved.

[0091] In step S101 of some embodiments, the preset general psychological assessment questionnaire includes open-ended questions, which are pre-set by business personnel or technical personnel, wherein the number of open-ended questions is a first preset number, such as 3 questions, 5 questions, etc. The first preset number needs to be set in combination with the actual application scenario, but is not limited thereto.

[0092] In one embodiment, in the medical field, the general psychological assessment questionnaire is a question about mental health assessment, which may include 5 open-ended questions, such as:

[0093] Question: Have you been feeling anxious or depressed lately? Please describe how you are feeling.

[0094] Question: What do you think is the biggest source of stress in your life right now? How do you usually deal with it? Please give an example.

[0095] Question: What do you think is your biggest challenge in interpersonal communication? Please give an example.

[0096] Question: How is your sleep quality recently? Have you ever had insomnia or excessive dreaming? Please give an example.

[0097] Question: Do you often have negative thoughts about yourself or self-doubt? In what situations do these thoughts usually occur? Please give an example.

[0098] Through the above questions, we can initially understand the user's mental health categories, such as depression, anxiety, mania, schizophrenia, etc., but not limited to these.

[0099] See also Figure 2 In some embodiments, step S102 may include but is not limited to steps S201 to S207:

[0100] Step S201, extracting expression features from the initial reply video to obtain initial expression embedding features;

[0101] Step S202, extracting audio from the initial reply video to obtain initial reply voice data;

[0102] Step S203, converting the initial reply voice data into text to obtain original reply text data;

[0103] Step S204, extracting semantic features from the original reply text data to obtain initial semantic embedding features;

[0104] Step S205, extracting speech features from the initial reply speech data to obtain initial speech embedding features;

[0105] Step S206, performing feature concatenation on the initial expression embedding feature, the initial semantic embedding feature and the initial speech embedding feature to obtain an initial multimodal psychological embedding feature;

[0106] Step S207, perform a preliminary evaluation on the initial multimodal psychological embedding features to obtain the mental health category.

[0107] In steps S201 to S207 shown in the embodiment of the present application, initial reply voice data is obtained by extracting from the initial reply video, and then the initial reply voice data is converted into text to obtain original reply text data; further, expression feature extraction is performed on the initial reply video to obtain initial expression embedding features; semantic feature extraction is performed on the original reply text data to obtain initial semantic embedding features; voice feature extraction is performed on the initial reply voice data to obtain initial voice embedding features, thereby realizing multimodal feature extraction, and feature splicing is performed on the initial expression embedding features, the initial semantic embedding features and the initial voice embedding features to obtain initial multimodal psychological embedding features; a preliminary assessment is performed on the initial multimodal psychological embedding features to obtain the mental health category, thereby realizing psychological assessment based on multimodal data, effectively making up for the deficiencies or deviations that may exist in single modal information, providing more accurate, detailed and comprehensive psychological assessment results, providing an important reference for the subsequent second psychological assessment, and improving the accuracy of psychological assessment.

[0108] In step S201 of some embodiments, expression features are extracted from the initial reply video using a pre-trained expression feature extraction model to obtain initial expression embedding features;

[0109] Specifically, the expression feature extraction model can be a convolutional neural network model, such as an EfficientNet model, a ResNet model, etc., but is not limited thereto, and the micro-expressions of the target object are extracted from the initial reply video to obtain the initial expression embedding features;

[0110] See also Figure 3 In some embodiments, step S201 may include but is not limited to steps S301 to S304:

[0111] Step S301, extracting video frames from the initial response video to obtain an initial video frame image;

[0112] Step S302, performing image preprocessing on the initial video frame image to obtain an initial face video image;

[0113] Step S303, extracting micro-expressions from the initial face video image to obtain initial expression embedding features.

[0114] In steps S301 to S303 shown in the embodiment of the present application, multiple initial video frame images are obtained by extracting video frames from the initial response video; then image preprocessing is performed on each initial video frame image to obtain an initial facial video image; thereby, the expression feature extraction model is called to extract micro-expressions from each initial facial video image to obtain initial expression embedding features, which can improve the accuracy of facial expression feature extraction, and thus help improve the accuracy of multimodal psychological assessment.

[0115] It can be understood that since the initial reply video is composed of continuous image frames, the number of initial video frame images is multiple. Furthermore, the number of initial facial video images is also multiple. Each initial facial video image corresponds to an initial micro-expression embedding feature. Finally, the initial micro-expression embedding features corresponding to the multiple initial facial video images are processed to obtain the initial expression embedding features corresponding to the initial reply video.

[0116] In step S301 of some embodiments, video frames of the initial response video can be extracted through deep learning models, machine learning models, and video processing tools to obtain initial video frame images, wherein the video processing tools may include but are not limited to OpenCV libraries, FFmpeg command line tools, etc., but are not limited to these.

[0117] In step S302 of some embodiments, image preprocessing may include but is not limited to:

[0118] Adjusting the size of the initial video frame image;

[0119] Adjusting the resolution of the initial video frame image;

[0120] Perform denoising on the initial video frame image;

[0121] Performing standardization processing on the initial video frame image;

[0122] Performing image enhancement processing on the initial video frame image;

[0123] The specific graphics preprocessing operations need to be selected in combination with the actual application scenarios, and the range of size and resolution adjustment needs to be set according to the data requirements of the expression feature extraction model, but is not limited thereto.

[0124] In step S303 of some embodiments, facial features are extracted from each initial face video image through a feature extraction layer (such as a convolutional layer) in an expression feature extraction model to obtain a plurality of initial micro-expression embedding features;

[0125] Furthermore, multiple initial micro-expression embedding features can be fused through attention mechanism, average pooling calculation, etc., and finally the initial expression embedding features can be obtained.

[0126] In some embodiments, after step S302 and before step S303, the questionnaire-based psychological assessment method may also include but is not limited to:

[0127] Perform face positioning on the initial face video image to update the initial face video image.

[0128] That is, the updated initial face video image is the face of the target object, and the background information is removed, thereby reducing the impact of noise and improving the accuracy of expression feature extraction.

[0129] In step S202 of some embodiments, audio extraction can be performed on the initial reply video through audio software or video editing tools to obtain initial reply voice data, wherein the format of the initial reply voice data needs to be selected according to the data requirements of a preset voice feature extraction model, such as FLAC format or WAV format, but not limited to this.

[0130] In step S203 of some embodiments, the initial reply voice data may be converted into text using a preset voice recognition algorithm to obtain original reply text data;

[0131] The speech recognition algorithm may be a Wav2Vec algorithm, a DeepSpeech algorithm, an LSTM model, etc., but is not limited thereto.

[0132] In step S204 of some embodiments, semantic features are extracted from the original reply text data using a pre-trained semantic feature extraction model to obtain initial semantic embedding features; wherein the semantic feature extraction model is used to convert text data into word vector data, i.e., in the form of embedded features.

[0133] Specifically, the semantic feature extraction can be a BERT model, a Word2Vec model, a GloVe model, etc., but is not limited thereto.

[0134] See also Figure 4 In some embodiments, step S204 may include but is not limited to steps S401 to S404:

[0135] Step S401, performing text cleaning on the original reply text data to obtain initial reply text data;

[0136] Step S402, performing word segmentation processing on the initial reply text data to obtain an initial reply word segmentation sequence;

[0137] Step S403, performing text encoding processing on the initial reply word segmentation sequence to obtain initial text encoding features;

[0138] Step S404: perform semantic prediction on the initial text encoding features to obtain initial semantic embedding features.

[0139] Steps S401 to S404 shown in the embodiment of the present application can decompose complex sentences into manageable vocabulary units by performing text cleaning and word segmentation processing on the original reply text data, and perform text encoding processing on each word in the initial reply word segmentation sequence to obtain initial text encoding features containing rich semantic information. Further semantic prediction is performed on the initial text encoding features, which can deeply explore and capture the deep meanings and contextual relationships behind these words, thereby comprehensively and accurately reflecting the psychological state of the target object and providing more accurate and comprehensive semantic feature support for psychological assessment.

[0140] In step S401 of some embodiments, the original response text data can be cleaned by using regular expressions or NLP libraries (such as NLTK or spaCy), where text cleaning includes but is not limited to removing punctuation marks, special symbols, stop words, etc.

[0141] In step S402 of some embodiments, the initial reply text data may be segmented using a segmentation tool, such as Jieba tool, Hanlp tool, etc., to divide the continuous text data into an initial reply segmentation sequence consisting of multiple words or units.

[0142] Therefore, it can be understood that the initial reply segmentation sequence contains a plurality of initial reply segmentation data;

[0143] In some embodiments, the semantic feature extraction model includes an embedding feature extraction layer and a semantic prediction layer, wherein the embedding feature extraction layer is used to convert the text representation into a numerical representation, i.e., an embedding feature; and the semantic prediction layer is used to output the semantic prediction feature;

[0144] Among them, the semantic prediction layer can be a fully connected layer.

[0145] Furthermore, in step S403 of some embodiments, text encoding is performed on each initial reply word segmentation data through the embedding feature extraction layer in the semantic feature extraction model to obtain multiple reply text sub-embedding features as the numerical representation of the initial reply word segmentation data, and then the multiple reply text sub-embedding features are concatenated to obtain the initial text encoding feature.

[0146] In step S404 of some embodiments, semantic prediction is performed on the initial text encoding features through the semantic prediction layer in the semantic feature extraction model to obtain initial semantic embedding features.

[0147] In step S205 of some embodiments, the initial reply voice data is subjected to voice feature extraction by a pre-trained voice feature extraction model to obtain initial voice embedding features; wherein the voice feature extraction model can extract voice features, i.e., voice embedding features, from the voice data. The voice embedding features include, but are not limited to, intonation, tone, volume, speech speed and other information, which need to be set according to the actual application scenario.

[0148] Specifically, the semantic feature extraction can be a HuBERT model, an MFCC model, an LSTM model, etc., but is not limited thereto.

[0149] See also Figure 5 In some embodiments, step S205 may also include but is not limited to steps S501 to S504:

[0150] Step S501, performing spectrum conversion on the initial reply voice data to obtain initial voice spectrum data;

[0151] Step S502, encoding the initial speech spectrum data to obtain initial speech coding features;

[0152] Step S503, performing attention calculation on the initial speech coding feature to obtain the initial speech attention feature;

[0153] Step S504, performing pooling processing on the initial speech attention features to obtain initial speech embedding features.

[0154] Steps S501 to S504 shown in the embodiment of the present application retain the frequency domain characteristics of the speech by converting the initial reply speech data into initial speech spectrum data, and then encode the initial speech spectrum data and convert it into a numerical representation to obtain initial speech coding features; further, the initial speech coding features are paid attention to by the attention mechanism, so that the model can focus on the parts of the speech data that are crucial to emotional expression, that is, those features that reflect changes in intonation and tone; finally, the initial speech attention features are pooled to obtain the initial speech embedding features, which can enhance the ability to capture central information of the speech data and accurately reflect the speech characteristics of the target object in the initial reply speech data, thereby providing more accurate and comprehensive speech feature support for psychological assessment.

[0155] In step S501 of some embodiments, the spectrum conversion of the initial reply speech data may be performed by methods such as Mel filter bank coefficients (Fbank), fast Fourier transform (FFT), short-time Fourier transform (STFT), Mel frequency cepstral coefficients (MFCC), etc. Specifically, it is necessary to make a selection in combination with the actual application scenario, but it is not limited thereto.

[0156] In some embodiments, the initial speech spectrum data is Mel spectrum data of a preset dimension, and the preset dimension needs to be determined according to the data requirements of the speech feature extraction model, but is not limited thereto.

[0157] In some embodiments, the speech feature extraction model includes an encoder and an attention mechanism; wherein the encoder is used to convert spectral data into numerical representation, i.e., embedding features; and the attention mechanism is used to extract contextual information, which helps to improve the accuracy of speech feature extraction.

[0158] In step S502 of some embodiments, the initial speech spectrum data is encoded by an encoder in the speech feature extraction model, and the spectrum data is converted into a numerical representation to obtain initial speech coding features.

[0159] In step S503 of some embodiments, attention calculation is performed on the initial speech coding features through the attention mechanism in the speech feature extraction model. Specifically, step S503 may include but is not limited to the following steps:

[0160] Performing matrix transformation on the initial speech coding features to obtain an original speech transformation matrix; wherein the original speech transformation matrix includes an original query transformation matrix, an original key transformation matrix and an original value transformation matrix;

[0161] Aggregate the original query transformation matrix and the original key transformation matrix to obtain the original attention score;

[0162] Normalize the original attention score to get the original attention weight;

[0163] The original attention weights and the original value transformation matrix are aggregated to obtain the initial speech attention features.

[0164] By introducing the attention mechanism to learn and capture the dependencies between different parts of the initial speech coding features, and assigning different weights to different parts of the initial speech coding features, the learning ability of the speech feature extraction model for the initial response speech data is improved, which helps to accurately extract the speech features of the target object.

[0165] Specifically, by calculating the dot product of the original query transformation matrix and the original key transformation matrix, and applying the softmax function to normalize the calculation according to the column dimension (usually the sequence length dimension), the original attention weights are obtained, which reflect the correlation strength of different parts in the initial speech coding features.

[0166] The original attention weight and the original value transformation matrix are weighted to obtain the initial speech attention feature.

[0167] In step S504 of some embodiments, the initial speech embedding feature may be obtained by performing average pooling processing on the initial speech attention feature.

[0168] Specifically, step S504 may include but is not limited to the following steps:

[0169] The initial speech attention feature is framed based on a preset pooling window length to obtain a plurality of original speech frame features and an original speech frame sequence;

[0170] Calculate the average of each original speech frame feature to obtain multiple original speech average frame features;

[0171] Based on the original speech frame order, the average frame features of multiple original speech are concatenated to obtain the initial speech embedding features.

[0172] It should be noted that the pooling window length is used to compress speech features, which can reduce the size of the feature map, thereby reducing the number of calculated parameters and improving calculation efficiency.

[0173] Specifically, the dimension of the pooling window length needs to be determined based on the data dimension of the initial speech attention feature and the input and output of the speech feature extraction model, but is not limited thereto.

[0174] In step S206 of some embodiments, the initial expression embedding feature, the initial semantic embedding feature and the initial speech embedding feature are concatenated into a complete feature to obtain an initial multimodal psychological embedding feature, that is, initial multimodal psychological embedding feature = [initial expression embedding feature, initial semantic embedding feature, initial speech embedding feature].

[0175] In step S207 of some embodiments, a preliminarily evaluation is performed on the initial multimodal psychological embedding features through a preset general multimodal psychological assessment model, and a label for characterizing a mental health category is output, such as, but not limited to, depression, pessimism, self-doubt, etc.

[0176] Among them, the general multimodal psychological assessment model can be a pre-trained neural network model, such as an emotion recognition model, but is not limited thereto.

[0177] See also Figure 6 In some embodiments, step S103 includes but is not limited to steps S601 to S603:

[0178] Step S601, obtaining a plurality of preset candidate closed-ended evaluation questions, wherein each candidate closed-ended evaluation question contains a psychological state attribute;

[0179] Step S602, matching the psychological state attributes of multiple candidate closed-ended assessment questions based on the psychological health category to obtain a target psychological assessment question;

[0180] Step S603: generating a target psychological assessment questionnaire based on the target psychological assessment questions.

[0181] Steps S601 to S603 shown in the embodiment of the present application match the psychological state attributes of multiple candidate closed-ended assessment questions based on the psychological health categories of the preliminary assessment, screen out questions for the second round of psychological assessment, and derive a target psychological assessment questionnaire. This can improve the pertinence and effectiveness of the psychological assessment, avoid unnecessary redundancy, and thereby more efficiently collect key psychological information, providing a more accurate and reliable basis for subsequent psychological assessments.

[0182] It should be noted that the candidate closed-ended assessment questions are not the same as the open-ended questions included in the general psychological assessment questionnaire, and each candidate closed-ended assessment question contains psychological state attributes. Specifically, the psychological state attributes are pre-set by technical / business personnel.

[0183] In step S602 of some embodiments, by matching the mental health category with the mental state attributes of a plurality of candidate closed-ended assessment questions, matching questions are screened out, which can improve the pertinence of the second round of psychological assessment and help improve the accuracy of the psychological assessment.

[0184] In step S603 of some embodiments, the target psychological assessment questions may be arranged in a random order to generate a target psychological assessment questionnaire.

[0185] In some embodiments, each candidate closed-ended assessment question also contains a psychological assessment weight, wherein the higher the weight, the higher the order in which the question is arranged.

[0186] It is understandable that the target psychological assessment questions are selected from the candidate closed-ended assessment questions, and therefore, the target psychological assessment questions also contain psychological assessment weights. Therefore, the questions can also be arranged according to the psychological assessment weights of the target psychological assessment questions to obtain the target psychological assessment questionnaire.

[0187] For example:

[0188] In one embodiment, in the medical field, a target psychological assessment questionnaire for performing a psychological health assessment includes a plurality of closed questions, specifically including but not limited to the following questions:

[0189] Question: In the past two weeks, have you often felt depressed or upset? Please select:

[0190] A. Every day;

[0191] B. Almost every day;

[0192] C. Sometimes;

[0193] D. Rarely;

[0194] E. Never.

[0195] Question: In the past month, have you often felt nervous or worried about the future? Please select:

[0196] A.Always;

[0197] B. often;

[0198] C. Sometimes;

[0199] D. Rarely;

[0200] E. Never.

[0201] Question: Have you recently had trouble falling asleep, waking up too early, or having trouble falling asleep again after waking up during the night? Please select:

[0202] A. It's like this every night;

[0203] B. Most nights;

[0204] C. Sometimes;

[0205] D rarely;

[0206] E. Never.

[0207] Question: How long did your symptoms last? Please select:

[0208] A. Less than one week;

[0209] B. One week to one month;

[0210] C. One to three months;

[0211] D. Three months to six months;

[0212] E. More than half a year.

[0213] Do these symptoms affect your work, study, social life or other daily activities? Please select:

[0214] A. Serious impact;

[0215] B. has some influence;

[0216] C. Less impact;

[0217] D.No impact.

[0218] In step S105 of some embodiments, expression features are extracted from the target reply video using a pre-trained expression feature extraction model to obtain target expression embedding features.

[0219] Specifically, step S105 may include but is not limited to the following steps:

[0220] Extracting video frames from the target reply video to obtain a target video frame image;

[0221] Performing image preprocessing on the target video frame image to obtain the target face video image;

[0222] Micro-expressions are extracted from the target face video image to obtain the target expression embedding features.

[0223] It should be noted that the specific implementation of the above step S105 is basically the same as the specific implementation of the above steps S301 to S303, and will not be repeated here.

[0224] In some embodiments, after step S105 and before step S106, the questionnaire-based psychological assessment method further includes:

[0225] Perform audio extraction on the target reply video to obtain the target reply voice data;

[0226] In step S106 of some embodiments, speech features of the target reply speech data are extracted using a pre-trained speech feature extraction model to obtain target speech embedding features.

[0227] Specifically, step S016 may include but is not limited to the following steps:

[0228] Performing spectrum conversion on the target reply speech data to obtain target speech spectrum data;

[0229] Encoding the target speech spectrum data to obtain target speech coding features;

[0230] Perform attention calculation on the target speech coding features to obtain the target speech attention features;

[0231] The target speech attention features are pooled to obtain the target speech embedding features.

[0232] It should be noted that the specific implementation of the above step S106 is basically the same as the specific implementation of the above step S202 and step S501 to step S504, and will not be repeated here.

[0233] In step S107 of some embodiments, the target questionnaire answer content is obtained by converting the target answer voice data into text.

[0234] It should be noted that the specific implementation of the above step S107 is basically the same as the specific implementation of the above step S203, which will not be repeated here.

[0235] See also Figure 7 In some embodiments, step S108 may include but is not limited to steps S701 to S704:

[0236] Step S701, evaluating the target questionnaire response content to obtain a target response evaluation score;

[0237] Step S702, extracting semantic features from the target questionnaire response content to obtain target semantic embedding features;

[0238] Step S703, performing feature concatenation on the target expression embedding feature, the target semantic embedding feature and the target speech embedding feature to obtain the target multimodal psychological embedding feature;

[0239] Step S704, performing a psychological evaluation on the target multimodal psychological embedding features and the target response evaluation score to obtain the target psychological evaluation state.

[0240] In steps S701 to S704 shown in the embodiment of the present application, a target answer evaluation score is obtained by evaluating the answer content of the target questionnaire, and semantic features are extracted from the target questionnaire answer content to obtain target semantic embedding features; further, feature splicing is performed on the target expression embedding features, the target semantic embedding features and the target speech embedding features to obtain target multimodal psychological embedding features; finally, a psychological assessment is performed based on the target multimodal psychological embedding features and the target answer evaluation scores to obtain a target psychological assessment state. The psychological assessment can be performed based on multimodal data and questionnaire answer evaluation scores, effectively making up for possible deficiencies or deviations in single modality information, providing more accurate, detailed and comprehensive psychological assessment results, and improving the accuracy of psychological assessment.

[0241] It is understandable that since the target psychological assessment questionnaire includes closed-ended questions, and closed-ended questions have different preset psychological assessment scores for different answers, the questionnaire results are obtained based on the psychological assessment scores, but it is also necessary to combine the multimodal data of the target subjects when answering the target psychological assessment questionnaire, so as to more accurately derive the target psychological assessment status.

[0242] In step S701 of some embodiments, a target response evaluation score is obtained by assigning points to the target subject's answer to each target psychological assessment question and summarizing the obtained points.

[0243] Specifically, different target response evaluation scores may correspond to risk levels of the psychological state, such as high risk, low risk, etc., but are not limited thereto.

[0244] In step S702 of some embodiments, semantic features are extracted from the target questionnaire response content using a pre-trained semantic feature extraction model to obtain target semantic embedding features.

[0245] Specifically, step S702 may include but is not limited to the following steps:

[0246] Perform text cleaning on the target questionnaire responses to obtain target response text data;

[0247] Perform word segmentation processing on the target reply text data to obtain a target reply word segmentation sequence;

[0248] Perform text encoding processing on the target reply word sequence to obtain the target text encoding features;

[0249] Perform semantic prediction on the target text encoding features to obtain the target semantic embedding features.

[0250] It should be noted that the specific implementation of the above step S702 is basically the same as the specific implementation of the above step S401 to step S404, and will not be repeated here.

[0251] In step S703 of some embodiments, the target expression embedding feature, the target semantic embedding feature and the target speech embedding feature are concatenated into a complete feature to obtain a target multimodal psychological embedding feature, that is, target multimodal psychological embedding feature = [target expression embedding feature, target semantic embedding feature, target speech embedding feature].

[0252] In step S704 of some embodiments, a risk assessment is performed on the target multimodal psychological embedding feature through a preset multimodal psychological state assessment model to obtain a psychological state assessment label; wherein the label may be, for example, depression, anxiety, fear, etc., but is not limited thereto.

[0253] Among them, the multimodal psychological state assessment model can be a pre-trained neural network model, such as an emotion recognition model, and the model structure of the multimodal psychological state assessment model and the general multimodal psychological assessment model can be the same or different. If the model structure of the multimodal psychological state assessment model and the general multimodal psychological assessment model is the same, the parameters of the model are different, but not limited to this.

[0254] Furthermore, risk categories, such as high risk, low risk, etc., are determined based on the target response assessment score, thereby generating a target psychological assessment state based on the psychological risk assessment label and the target response assessment score, for example, high risk of depression, low risk of anxiety, etc.

[0255] For example:

[0256] In one embodiment, in the medical field, during a psychological assessment of a target subject in the field of mental health assessment,

[0257] When the target subject answers the general psychological assessment questionnaire, the target subject's initial response video is obtained through a camera or other means, and a preliminary multimodal psychological assessment is conducted on the initial response video, and the mental health category is concluded to be depression;

[0258] Based on the mental health category, a plurality of target psychological assessment questions are screened out from a plurality of preset candidate closed assessment questions to form a target psychological assessment questionnaire.

[0259] When the target subject answers the target psychological assessment questionnaire, the target response video of the target subject is obtained by means of a camera, etc., and a multimodal psychological assessment is performed on the target response video to obtain the target psychological assessment status, which is a moderate depression state.

[0260] Based on this, professional psychologists can issue professional treatment recommendations or prescribe treatment prescriptions based on the target psychological assessment status of the target subject being a moderately depressed state.

[0261] In some embodiments, the questionnaire-based psychological assessment method provided in the embodiments of the present application can also be applied to a variety of scenarios. For example, in the financial field, by conducting psychological assessments on customers, it is possible to accurately understand the customer's psychological state, thereby providing customers with more professional investment advice, investment risk analysis, customized financial services, etc. For example:

[0262] In the financial field, the general psychological assessment questionnaire includes 4 open-ended questions, specifically:

[0263] Question: What is your opinion on the current trend of financial markets? Please explain briefly.

[0264] Question: How much do you think you can bear for investment risk? Please explain why.

[0265] Question: What is your past experience with financial investments? Please share your investment history.

[0266] Question: Please describe your long-term and short-term financial goals. Please briefly explain.

[0267] Through the above questions, we can have a preliminary understanding of the user's investment psychology, such as greedy investment psychology, cautious investment psychology, fearful investment psychology, etc.

[0268] Furthermore, by obtaining an initial response video of the target subject responding to a common psychological assessment questionnaire in the financial field and conducting a preliminary psychological assessment on the initial response video, a mental health category is obtained;

[0269] Obtain a target psychological assessment questionnaire based on the mental health category; the target psychological assessment questionnaire may include but is not limited to the following questions:

[0270] Question: What do you think your risk tolerance is? Please select:

[0271] A. Very high, willing to take high risks in pursuit of high returns;

[0272] B. High, and a certain degree of investment risk can be accepted;

[0273] C. Moderate, tending to adopt a conservative investment strategy;

[0274] D. Lower, hoping to avoid investment risks;

[0275] E. Very low, only willing to accept risk-free or low-risk investments.

[0276] Question: What is your expectation for investment income in the future? Please select:

[0277] A. Very optimistic, expecting significant earnings growth;

[0278] B. Optimistic, believing that the market will maintain steady growth;

[0279] C. Neutral, cautious about market trends;

[0280] D. Pessimistic, worried that the market will fall;

[0281] E. Very pessimistic, expecting a large loss.

[0282] Question: What factors usually influence your investment decisions? Please select (multiple choices are allowed):

[0283] A. Market trend analysis;

[0284] B. Advice from friends or family;

[0285] C. Media reports and expert comments;

[0286] D. intuition or feeling;

[0287] E. Others (please specify).

[0288] Question: Will you change your investment plan due to short-term market fluctuations? Please select:

[0289] A.Always;

[0290] B. often;

[0291] C. Sometimes;

[0292] D. Rarely;

[0293] E. Never.

[0294] Question:Have you felt anxious or uneasy recently because of your investment performance? Please select:

[0295] A. Very anxious and stressed;

[0296] B. A little anxious, but still within control;

[0297] C. Generally, no obvious mood swings;

[0298] D. relatively calm and satisfied with investment performance;

[0299] E. Very calm and have long-term confidence in investment.

[0300] Finally, based on the target response video of the target object answering the target psychological assessment questionnaire in the financial field, a psychological assessment is performed on the target response video to obtain the target psychological assessment status; the target psychological assessment status is used to characterize the mental health status corresponding to the target object's financial investment.

[0301] In one embodiment, during a psychological assessment in the financial sector,

[0302] When the target subject answers the general psychological assessment questionnaire, the initial response video of the target subject is obtained through means such as a camera, and a preliminary multimodal psychological assessment is conducted on the initial response video, and the psychological health category is concluded to be fear investment psychology;

[0303] Based on the mental health category, a plurality of target psychological assessment questions are screened out from a plurality of preset candidate closed assessment questions to form a target psychological assessment questionnaire.

[0304] When the target object answers the target psychological assessment questionnaire, the target object's target response video is obtained by means of a camera, etc., and a multimodal psychological assessment is performed on the target response video to obtain the target's psychological assessment status, which is a mild fear of investment psychology.

[0305] Based on this, business personnel can select appropriate low-risk investment products and recommend them to the target object based on the target object's psychological assessment status of mild fear of investment, or make suggestions for adjusting and improving the current investment direction based on the target's psychological assessment status.

[0306] See also Figure 8 The embodiment of the present application also provides a questionnaire-based psychological assessment device, which can implement the above-mentioned questionnaire-based psychological assessment method, and the device includes:

[0307] The initial response data acquisition module 801 is used to acquire the target subject's initial response video to a preset general psychological assessment questionnaire; wherein the general psychological assessment questionnaire includes open-ended questions;

[0308] A preliminary psychological assessment module 802 is used to perform a preliminary psychological assessment on the initial response video to obtain a mental health category;

[0309] An assessment questionnaire generation module 803 is used to obtain a target psychological assessment questionnaire based on the psychological health category; wherein the target psychological assessment questionnaire includes closed questions;

[0310] The target response data acquisition module 804 is used to acquire the target response video of the target subject responding to the target psychological assessment questionnaire;

[0311] A target expression feature extraction module 805 is used to extract expression features based on the target reply video to obtain target expression embedding features;

[0312] A target speech feature extraction module 806 is used to extract speech features based on the target reply video to obtain target speech embedding features;

[0313] The target questionnaire answer content acquisition module 807 is used to acquire the target questionnaire answer content from the target answer video;

[0314] The target psychological assessment module 808 is used to perform a psychological assessment based on the target expression embedding features, the target voice embedding features and the target questionnaire response content to obtain the target psychological assessment status; wherein the target psychological assessment status is used to characterize the mental health status of the target object.

[0315] The specific implementation of the questionnaire-based psychological assessment device is substantially the same as the specific implementation of the questionnaire-based psychological assessment method described above, and will not be described in detail herein.

[0316] The embodiment of the present application also provides an electronic device, the electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the above-mentioned questionnaire-based psychological assessment method when executing the computer program. The electronic device can be any intelligent terminal including a tablet computer, a car computer, etc.

[0317] See also Fig. 9 , Fig. 9 The hardware structure of an electronic device of another embodiment is illustrated, and the electronic device includes:

[0318] The processor 901 may be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application;

[0319] The memory 902 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 902 can store an operating system and other application programs. When the technical solution provided in the embodiment of this specification is implemented by software or firmware, the relevant program code is stored in the memory 902, and the processor 901 calls and executes the questionnaire-based psychological assessment method of the embodiment of this application;

[0320] Input / output interface 903, used to implement information input and output;

[0321] Communication interface 904, used to realize communication interaction between the device and other devices, which can be realized by wired mode (such as USB, network cable, etc.) or wireless mode (such as mobile network, WIFI, Bluetooth, etc.);

[0322] A bus 905 that transmits information between the various components of the device (e.g., the processor 901, the memory 902, the input / output interface 903, and the communication interface 904);

[0323] The processor 901 , the memory 902 , the input / output interface 903 and the communication interface 904 are connected to each other in communication within the device via a bus 905 .

[0324] An embodiment of the present application also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the above-mentioned questionnaire-based psychological assessment method is implemented.

[0325] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0326] The questionnaire-based psychological assessment method and device, electronic device and storage medium provided in the embodiment of the present application first obtain an initial response video of the target object responding to a preset general psychological assessment questionnaire; wherein the general psychological assessment questionnaire includes open questions; a preliminary psychological assessment is performed on the initial response video to obtain a mental health category, and a target psychological assessment questionnaire is obtained based on the mental health category; wherein the target psychological assessment questionnaire includes closed questions; thereby the psychological state of the target object is evaluated more deeply; further, a target response video of the target object responding to the target psychological assessment questionnaire is obtained; expression features are extracted based on the target response video to obtain a target expression embedding feature; voice features are extracted based on the target response video to obtain a target voice embedding feature; the target questionnaire response content is obtained from the target response video; psychological assessment is performed based on the target expression embedding feature, the target voice embedding feature and the target questionnaire response content, thereby obtaining a target psychological assessment state that can characterize the mental health state of the target object; wherein the target psychological assessment state is used to characterize the mental health state of the target object. By using two psychological assessment questionnaires for psychological assessment, combined with the multimodal data of the target object when responding to the psychological assessment questionnaire, the psychological state of the target object can be evaluated more comprehensively and accurately, and the accuracy, effectiveness and reliability of the psychological assessment can be improved.

[0327] The embodiments described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0328] Those skilled in the art will appreciate that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0329] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0330] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0331] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0332] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0333] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the above units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0334] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0335] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0336] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including multiple instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store programs.

[0337] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but the scope of the rights of the present invention is not limited thereto. Any modification, equivalent substitution and improvement made by a person skilled in the art without departing from the scope and essence of the present invention should be within the scope of the rights of the present invention.

Claims

1. A psychological assessment method based on a questionnaire, characterized in that: The method comprises: Obtaining an initial response video of the target subject responding to a preset universal psychological assessment questionnaire; wherein the universal psychological assessment questionnaire includes open-ended questions; Conducting a preliminary psychological assessment on the initial response video to obtain a mental health category; Obtaining a target psychological assessment questionnaire based on the mental health category; wherein the target psychological assessment questionnaire includes closed questions; Obtaining a target response video of the target subject responding to the target psychological assessment questionnaire; Extracting facial expression features based on the target reply video to obtain target facial expression embedding features; Extract speech features based on the target reply video to obtain target speech embedding features; Obtaining target questionnaire response content from the target response video; A psychological assessment is performed based on the target expression embedding features, the target speech embedding features and the target questionnaire answer content to obtain a target psychological assessment state; wherein the target psychological assessment state is used to characterize the mental health state of the target object.

2. The method according to claim 1, characterized in that The initial response video is subjected to a preliminary psychological assessment to obtain a mental health category, including: Extracting facial expression features from the initial reply video to obtain initial facial expression embedding features; Performing audio extraction on the initial reply video to obtain initial reply voice data; Converting the initial reply voice data into text to obtain original reply text data; Extracting semantic features from the original reply text data to obtain initial semantic embedding features; Extracting speech features from the initial reply speech data to obtain initial speech embedding features; Performing feature concatenation on the initial expression embedding feature, the initial semantic embedding feature and the initial speech embedding feature to obtain an initial multimodal psychological embedding feature; The initial multimodal psychological embedding features are preliminarily evaluated to obtain the mental health category.

3. The method according to claim 2, characterized in that The extracting expression features from the initial reply video to obtain initial expression embedding features includes: Extracting video frames from the initial response video to obtain an initial video frame image; Performing image preprocessing on the initial video frame image to obtain an initial face video image; Micro-expressions are extracted from the initial face video image to obtain the initial expression embedding features.

4. The method according to claim 2, characterized in that: The extracting semantic features of the original reply text data to obtain initial semantic embedding features includes: Performing text cleaning on the original reply text data to obtain initial reply text data; Performing word segmentation processing on the initial reply text data to obtain an initial reply word segmentation sequence; Performing text encoding processing on the initial reply word segmentation sequence to obtain initial text encoding features; Perform semantic prediction on the initial text encoding features to obtain the initial semantic embedding features.

5. The method according to claim 2, characterized in that: The extracting speech features of the initial reply speech data to obtain initial speech embedding features includes: Performing spectrum conversion on the initial reply voice data to obtain initial voice spectrum data; Performing encoding processing on the initial speech spectrum data to obtain initial speech encoding features; Performing attention calculation on the initial speech coding feature to obtain an initial speech attention feature; The initial speech attention feature is pooled to obtain the initial speech embedding feature.

6. The method according to any one of claims 1 to 5, characterized in that: The psychological assessment is performed based on the target expression embedding feature, the target voice embedding feature and the target questionnaire answer content to obtain the target psychological assessment state, including: Performing response evaluation on the target questionnaire response content to obtain a target response evaluation score; Extracting semantic features from the target questionnaire response content to obtain target semantic embedding features; Performing feature concatenation on the target expression embedding feature, the target semantic embedding feature and the target speech embedding feature to obtain a target multimodal psychological embedding feature; A psychological assessment is performed on the target multimodal psychological embedding features and the target response evaluation score to obtain the target psychological assessment state.

7. The method according to any one of claims 1 to 5, characterized in that: The step of obtaining a target psychological assessment questionnaire based on the mental health category includes: Acquire a plurality of preset candidate closed-ended assessment questions; wherein each of the candidate closed-ended assessment questions contains a psychological state attribute; Matching the mental state attributes of a plurality of the candidate closed-ended assessment questions based on the mental health category to obtain a target mental assessment question; The target psychological assessment questionnaire is generated based on the target psychological assessment questions.

8. A psychological assessment device based on a questionnaire, characterized in that: The device comprises: An initial response data acquisition module is used to acquire an initial response video of a target subject responding to a preset general psychological assessment questionnaire; wherein the general psychological assessment questionnaire includes open-ended questions; A preliminary psychological assessment module, used for conducting a preliminary psychological assessment on the initial response video to obtain a mental health category; An assessment questionnaire generation module, used to obtain a target psychological assessment questionnaire based on the mental health category; wherein the target psychological assessment questionnaire includes closed questions; A target response data acquisition module, used to acquire a target response video of the target subject responding to the target psychological assessment questionnaire; A target expression feature extraction module is used to extract expression features based on the target reply video to obtain target expression embedding features; A target speech feature extraction module, used to extract speech features based on the target reply video to obtain target speech embedding features; A target questionnaire answer content acquisition module, used to acquire the target questionnaire answer content from the target answer video; The target psychological assessment module is used to perform a psychological assessment based on the target expression embedding features, the target voice embedding features and the target questionnaire answer content to obtain a target psychological assessment state; wherein the target psychological assessment state is used to characterize the mental health state of the target object.

9. An electronic device, characterized in that: The electronic device comprises a memory and a processor, the memory stores a computer program, and the processor implements the method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images